Kafka Health Monitoring Suite

Overview

Apache Kafka is an event-driven data streaming platform that stores and brokers sensitive information with high data throughput and fault tolerance. Broker clusters require health monitoring to better inform partitioning, replication, data persistence and other intricate Kafka maintenance processes. Iris offers a dynamic solution to observe your Kafka platform in real time, compare against historically logged data and ensures your broker does not throttle user experience.

Setting up your environment

Iris provides a containerized Docker environment with a highly configurable Kafka container. Please see the docker-compose.yml if you wish to configure your broker.

1. Install Docker: https://docs.docker.com/get-docker/

2. Fork the repo from the main branch

3. Run the command: npm install

4. Open Docker and check Images tab, it should include the following Docker images to run the container environments. If not, please install the latest images using the docker pull command.

5. Once Docker has been booted up, run the docker-compose.yml file in your terminal using the docker-compose up command.

   • Watch your terminal. If at any point you receive the following error, close your Docker environment in your terminal with CTRL+C and rerun docker-compose up again. Use Docker Desktop, and click on JMX-Kafka's logs to monitor the JMX container logs for any errors you may miss in your terminal.
   

6. Now that the Kafka broker is running and the ports are being read properly, run your Kafka dependent program across the broker to begin streaming your data.

7. In another terminal, run command npm run dev to start the application and begin viewing the broker's health.

Viewing the Metric

1. After running npm run dev, the application will load onto localhost:8080 where you can browse your incoming metrics.

   • On the left pane, there are persistent histograms and piecharts.
   • On the right pane, you can add line charts with different metrics.

2. Click on Metric and Time Frame dropdowns to monitor different Kafka Metrics on different Time Frames.

3. Click on + button in the upper right hand corner to add additional line charts.

4. Click on X button on each line chart container to delete each chart.

5. Click on clock button underneath + in the upper right hand corner to view historical data at a specific time interval. Data is logged on a 15 sec interval to an AWS RDS SQL database.

Changing Ports

1. On load, the default PORT is localhost:9090

2. Click on PORT ACCESSS on the Navigation Bar to switch to a different port

   • Port numbers and passwords are hard coded into a MongoDB database.

   • If you'd like to add additional port number and passwords, use Postman to make a POST request to '/createPort' with keys port and password in the request body.

   • You should receive a positive response of the MongoDB document with the newly created port and hashed password in the response.

   • If you receive a null response, the port already exists in the database.

Other accessible localhost ports

1. To view a list of raw JMX metrics and data : localhost:5556/metrics

2. To view Prometheus : localhost:9090

3. To view a list of available metrics: http://localhost:9090/api/v1/label/__name__/values

4. In order to query different Prometheus end points, follow the syntax : http://localhost:9090/api/v1/query?query={Enter the Metric Name}[{TimeRange}].

• For example: http://localhost:9090/api/v1/query?query=kafka_server_broker_topic_metrics_messagesinpersec_rate[1h]

Using the provided demo

We provide an end to end Kafka producer and consumer to measure the stability of the Iris health monitor. The Producer and Consumer files contain simple message files streamed to the Kafka Broker using AVRO schema and KafkaJS.

• In one terminal, run the command: npm run start:producer. It should show:

  

• In a separate terminal, run the command: npm run start:consumer. It should show:

  

Authors

• Brennan Lee

  • https://www.linkedin.com/in/brennan-lee/
  • https://github.com/blee3395

• Michael Reyes

  • https://github.com/Michaelr499
  • https://www.linkedin.com/in/michael-reyes-b4319216b/

• Walter Li

  • https://github.com/findwalle
  • https://www.linkedin.com/in/li-walter/

• Huu (John) Le

  • https://www.linkedin.com/in/huu-le/
  • https://github.com/JohnLeGit