Follow below steps to get started with the meshIQ Kafka Console setup.
Prerequisites
Before running this procedure, please verify that the machine on which the installation will be done meets all prerequisites described below.
Linux System
- Operating System: Linux 64 bit OS
- CPU: Minimum 4 virtual CPUs
- Memory: Minimum 8GB RAM
- File System: Minimum 1 GB free space
- Recommended: Install folder /opt
- Recommended: Dedicated Server Instance
Network
- Node Exporter: PORT=9100(TCP)
- JMX Java Agent: PORT=9404(TCP)
- zookeeper jmx: PORT=7000(TCP)
- Web HTTP Server: PORT=8080(TCP)
mKC Setup
These steps assume local installation for all related components.
CLIENT MKC SERVER
Please make sure to install meshIQ Kafka Console (mKC) on a separate server, not on the same server as your Kafka installation. mKC uses its own Kafka installation for internal services.
1. Untar/unzip the mkc_vx.x.x_dpack_vx.x.tar.Z package to the installation path of your choice.
This package contains the directory "mkc" as a root directory.
SHELL COMMAND:
tar -zxvf mkc_vx.x.x_dpack_vx.x.tar.Z
2. To start the mKC services, and for initial setup:
Navigate to "your_installation_path/sbin/" directory. Run the command below and follow the
onscreen instructions.
SHELL COMMAND:
cd <your_installation_path>/sbin
SHELL COMMAND:
./start.sh all
Press the "enter" key to set the install path automatically.
The first time you run the command, you will be asked to specify the untarred/unzipped directory location. After that you will need to re-run the ./start.sh all command.
3. Connect mKC to your existing Kafka installation:
Access the MKC UI from a web browser, use the following address:
User Name: user1 (case sensitive)
Password: pass (case sensitive)
If you try to get to the web user interface immediately after all of the processes have been successfully started, the meshIQ owl logo is displayed for a short time. It may take up to 30 seconds for the login page to appear.
Once you are logged in, select either Explore Kafka or the Kafka link under Quick Access. Then register Kafka and connect to your existing installation. Follow the instructions on the screen.
When you register a cluster, if you have not set up Metrics Settings, mKC displays a banner message at the top of the page to remind you to set up JMX and Node metric ports. In mKC Cluster Settings, scroll down to the Brokers section and click Metrics Settings, located at the upper right of the list of brokers. Select Use JMX Metric and Use Node Metric. Then click Update below the broker list. JMX and Node statistics will be enabled only after Kafka Server setup is complete.
See also the Brokers section of Cluster overview and management.
To connect mKC to Kafka statistics you would need to run mKC Kafka components on Kafka and ZooKeeper servers.
KAFKA SERVER
1. On each Kafka Server:
Untar/unzip the mkc_ServerComponents_dpack_vx.x.tar.Z package to the installation path of your
choice.
This package contains the directory "mkc" as a root directory (the package unzips in /opt/mkc).
SHELL COMMAND:
tar -zxvf mkc_ServerComponents_dpack_vx.x.tar.Z
2. On the Kafka server, to enable mKC to obtain the Kafka metrics, set the KAFKA_OPT environment variable: For example:
export KAFKA_OPTS="-javaagent:/opt/mkc/jmx_agent/jmx_prometheus_javaagent-0.20.0.jar=9404:/opt/mkc/jmx_agent/metric_rules.yml"
If "/opt" is not your install path, then replace /opt in above string with your installation path in both locations.
Restart the Kafka server.
The JMX agent listens on port 9404.
3. Start Node Exporter:
Navigate to "your_installation_path/sbin" directory and run the command below. Follow the onscreen instructions.
SHELL COMMAND:
./start.sh all
Press the "enter" key to set the install path automatically.
The node exporter listens on port 9100.
ZOOKEEPER SERVER
If your Kafka is configured to use ZooKeeper, follow the ZooKeeper Server steps below. But if your Kafka server is configured to use KRaft, you can skip the 3 setup steps below.
1. On each ZooKeeper Server:
Untar/unzip the mkc_ServerComponents_dpack_v1.0.tar.Z package to the installation path of your choice.
This package contains the directory "mkc" as a root directory.
SHELL COMMAND:
tar -zxvf mkc_ServerComponents_dpack_vx.x.tar.Z
2. Ensure zookeeper JMX is enabled.
Edit: YOUR_ZOOKEEPER_INSTALL_PATH/config/zoo.cfg and verify that the variables below are set.
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpPort=7000
metricsProvider.exportJvmInfo=true
3. Start Node Exporter.
Run the command below from "your_installation_path/sbin" directory. Follow the onscreen
instructions.
SHELL COMMAND:
./start.sh all
Press the "enter" key to set the install path automatically.
FIREWALL PORTS CONSIDERATIONS FOR mKC CONNECTIONS
- Make sure that the java jmx agent 9404 port is open from mKC to Kafka.
- Make sure that the node exporter 9100 port is open from mKC to Kafka.
On the ZooKeeper node:
- Make sure that the ZooKeeper JMX port 7000 is open for communication going from mKC to ZooKeeper.
---------------------You have completed mKC implementation.---------------------
ADDITIONAL INFORMATION
sbin files
The files in the sbin folder are listed below.
Running start.sh and stop.sh without any options results in all available options being used.
start.sh all (to start all mkc processes)
stop.sh all (to stop all mkc processes)
show.sh (to get status of all mkc processes)
cleanup.sh (will clean all log files and produce a backup of all the logs)
mkin_env.sh (file contains all necessary environment variable to run mkc)
Enable running startup scripts from anywhere (optional)
To run startup scripts from anywhere, we recommend that you set the MKIN_HOME and PATH environment variables to the mkc and sbin directories in the user profile by following the instructions below.
Copy and paste these two lines toward the end of the user profile file, changing /opt/mkc to the path of the root directory of your mKC installation:
export MKIN_HOME=/opt/mkc
export PATH=$PATH:$MKIN_HOME/sbin
Set the working path manually (optional)
To set the working path manually, edit MKIN_HOME/sbin/mkin_env.sh.
Change the following line, replacing ExistingPath with NewPath:
export MKIN_HOME=ExistingPath