If metrics have not been set up for the cluster, the Smart Rebalance recommendation job is not possible. Therefore the feature will not be available.
A Kafka Cluster consists of one or more brokers that are connected to each other and delivers messages published by producers to consumers. Additionally, Kafka stores received messages on disk for a certain period of time to prepare for failures and provides an interface that allows consumers to read them again if necessary.
In this process, Kafka Brokers use disk devices, and as you operate a Kafka Cluster, disk usage will inevitably vary across brokers. This imbalance in disk usage indicates that the amount of messages being sent by producers differs, leading to varying levels of network usage among brokers.
Therefore, balancing the disk usage between brokers by moving some topic partitions will ultimately balance the usage of network and other resources as well. mKC provides the Smart Rebalancing feature to easily balance disk usage based on the collected information.
If a partition movement is in progress, accessing the Smart Rebalance screen will automatically redirect to the Partition Jobs screen.
Creating a New Rebalancing Job
Make sure to check the available disk space before rebalancing. There's a risk of disk overflow during the data replication process (moving partitions).
To create a new rebalancing job, go to the Advanced > Smart Rebalance menu.
Step 1: Calculating Recommendations through Disk Balancer
Adjust the Max Recursive Limit to find the optimal total movement. After modifying the count, click New Recommendation button to view the newly calculated recommendations.
| Item | Description |
|---|---|
| Disk Balancer | Graph comparing current and post-rebalancing predicted usage per broker |
| Max recursive limit | Limit of the number of partitions rebalance calculations (recursion) |
| New recommendation | Recalculates partition movements |
| Total Data Movement |
The total amount of data moved as a result of the calculation. Total movement may appear as zero in the following cases.
|
| Configure rebalance | Moves to configuration screen after reviewing recommendations and movement amounts |
Step 2: Confirming and Creating Job Configuration
When you come to the next page, following the calculation from the previous step, you will be able to see the list of topic-partition to be moved. Once you have checked the list of topic-partitions that will be moved based on recommendations and have completed the Throttling Settings, you can start Smart Rebalancing by clicking Smart rebalance.
Throttling Settings
For further details about throttling, refer to the Throttling document.
Step 3: Checking Running Job Status
By clicking the Advanced > Partition Jobs menu on the left, you can check the status for the job that was just created and is currently running. In Partition Jobs, you can cancel the Smart Rebalance Job, or modify or disable the throttle values.
Only one rebalancing job can be running at a time. Therefore, you cannot create a new job if there already is one in progress. Please create a new job only when the previous one has completed.