In Kafka, you can set an upper bound on the bandwidth used to move replicas between brokers, by applying throttling to partition reassign operations in which replicas move. Throttling is the technology that limits the maximum amount of data transferred per second in either data replication or movement operations.
For example, when using bin/kafka-reassign-partitions.sh script provided by Kafka in partition reassignment, you can add the --throttle option to adjust the speed at which data is moved. If you include the --throttle 50000000 option in reassigning operation, this will adjust to only allow partitions to move at a maximum speed of 50MB/s. In this way, you can keep the system stable and avoid overloads that can occur when moving data.
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --execute --reassignment-json-file mkc-cluster.json --throttle 50000000
In mKC, if you're running a partition reassignment that moves partitions, such as Smart Rebalance or Cluster Scaling, you can turn on or off the throttling settings in operations. If you find an issue while monitoring the Partition Job Status screen, such that the partition reassignment is too slow, you can disable throttling or adjust the throttling value to ensure cluster stability and performance.
If you want to set or disable throttling in a partition reassigning operation that is already in progress, please refer to the Setting or Disabling Throttling section of the Partition Jobs article.
How to Safely Use Throttling
While throttling can be a great way in partitions reassignment to keep your system from being overloaded, abuse of throttling can cause problems. Please be careful when using throttling to execute partitions reassignment.
1. Disabling Throttling
Once the partitions reassignment with throttling settings have been completed, you should disable throttling. Continuing to run a cluster without disabling throttling will directly affect the speed at which data is replicated between replicas of a topic, which will result in non-optimal usage of the cluster's resources.
In mKC, when Cluster Scaling or Smart Rebalancing job with set throttling has finished, throttling is automatically disabled.
2. Calculating Throttling
When configuring throttling settings, it is advisable to calculate and set throttling values after checking the network bandwidth where Kafka is installed. Incorrectly setting throttling can lead to several issues.
For example, if throttling is set higher than the network bandwidth without checking in advance, excessive communication between brokers may occur, increasing the load on the brokers and potentially causing them to go down.
On the other hand, if throttling is set too low compared to the message inflow, the partition replication speed may be slower than the data inflow, causing delays in partition movements and indefinitely prolonged data replication.
Throttling Settings
In mKC, Smart Rebalancing and Cluster Scaling are operations in which partition reassignment occurs. When this takes place, you can set or disable throttling. When you set the network bandwidth value, mKC provides a feature that recommends throttling settings accordingly.
Calculating Throttling
If you click the Would you like a throttling recommendation? button to the right of the Throttling Settings, a drawer that provides recommended approximate throttling settings based on the network bandwidth of the cluster will be displayed.
If you enter the network bandwidth on which the Kafka Cluster is installed, throttling recommendation values will be provided. The minimum/maximum throttling recommendations will change whenever the network bandwidth is altered. The buttons are only activated when both minimum and maximum throttling values are above 0, and you can apply the desired throttling values by clicking the Apply lower bound or Apply upper bound buttons.
Please check your network bandwidth entry.
If you enter a network bandwidth that is too low, you may get a throttling value of 0 or some other value. Please enter an accurate network bandwidth so that recommendations of lower/upper throttling values based on the current message throughput can be provided.
When setting throttling, it is recommended to only use 80% of your network bandwidth and leave the remaining 20% as a buffer. mKC also provides throttling recommendations assuming that you only use 80% of your maximum network bandwidth. is a note box. This is a note box. This is a note box. This is a note box. This is a note box. This is a note box. This is a note box. This is a note box. This is a note box. This is a note box. This is a note box. This is a note box.