If no metrics have been set up for the cluster, the Cluster Scaling recommendation cannot be made. Therefore the feature will not be available.
Introduction
Scaling a Kafka Cluster is a task that may frequently occur in production environments due to various reasons such as increased traffic, data growth, or resource constraints. mKC facilitates this process by allowing users to easily perform the necessary steps before or after adjusting the number of cluster brokers.
You can start a new cluster scaling job by selecting the Advanced > Cluster Scaling menu.
If a partition movement is in progress, accessing the Cluster Scaling screen will automatically redirect to the Partition Jobs screen.
Balancing Data after Scale-out
After completing a Scale-out Operation to add new brokers, it is important to ensure that the disk usage of the entire cluster is evenly distributed, including the newly integrated brokers. During this process, you can monitor the data of each broker and efficiently carry out the necessary rebalancing jobs.
Step 1: Selecting Operation Type
First, select Scale-out from Scaling Info.
Step 2: Calculating Recommendations through Disk Balancer
Check if the predicted usage for the new broker was calculated to be evenly distributed. For detailed instructions on using the Disk Balancer, please refer to the Calculating Recommendations through Disk Balancer section (step 1) in the Smart Rebalancing article.
Step 3: Confirming and Creating Job Configuration
After reviewing the results from the previous step, you can see a list of topic-partition movements on the next page. Once you have checked the list of topic-partitions that will be moved based on recommendations and have completed the Throttling Settings, you can start Cluster Scaling by clicking Cluster Scaling.
Throttle Settings
For further details about throttling, refer to the Throttling document.
Step 4: Checking Running Job Status
By clicking the Advanced > Partition Jobs menu on the left, you can check the status for the scaling job that was just created and is currently running.
Transferring Data before Scale-in
Before proceeding with Scale-in Operation which reduces the number of brokers, you have to safely transfer data of the broker to be removed, to other brokers. In this process, you can efficiently perform the necessary rebalancing tasks, to ensure that the data being moved is evenly distributed among the remaining brokers.
Step 1: Selecting Operation Type and Broker to Empty
First, select Scale-in from the scaling options, and choose the broker you wish to empty. If it is not possible to empty the selected broker, the recommendation calculation in the next step may not be available.
The broker cannot be emptied in the following cases:
- When there is not enough free disk space for the selected brokers to be moved
- When a partition exists on which the number of replicas cannot be reduced
Step 2: Calculating Recommendations through Disk Balancer
Check if the predicted usage of the broker to be emptied is calculated to be 0 bytes (excluding the usage of the __blackbox_exporter topic). For detailed instructions on using the Disk Balancer, please refer to the Calculating Recommendations through Disk Balancer section (step 1) in the Smart Rebalancing article.
Step 3: Confirming and Creating Job Configuration
After reviewing the results from the previous step, you can see a list of topic-partition movements on the next page. Once you have checked the list of topic-partitions that will be moved based on recommendations and have completed the Throttling Settings, you can start Cluster Scaling by clicking Cluster Scaling.
Throttle Settings
For further details about throttling, refer to the Throttling article.
Step 4: Checking Running Job Status
By clicking the Advanced > Partition Jobs menu on the left, you can check the status for the scaling job that was just created and is currently running.