Metrics with Spring Boot, Prometheus and Grafana

Capturing metrics from your system is critical to understanding its internal behavior and to tune its performance. Without this you are operating in the blind. In this post we will go through how you can gather metrics from a Spring Boot application using Prometheus, Grafana and Micrometer.

The term Observability is widely used nowadays. I do not want to get into a religious debate on what it may be or may not be. I look at this by asking myself — “if i am woken up at midnight with a production issue, can I look at my systems’ collected health related data and ask various questions to help me narrow down to the root cause within a few minutes”. The most common types of data we collect are – metrics, logs and traces. All three are important, but fusing them into one cohesive dataset to allow for faster consumption is the key. I suggest you do some reading to familiarize yourself with what is noise from vendors related to Observability, vs. the real intent. I recommend reading…

This blog specifically deals with collecting Metrics which is one type of timeseries data that can assist with Observability. We will get metrics from a simple Sprint Boot application that publishes a message to Kafka as a result of a REST API call. The app also acts as the consumer since this is a test application. We will focus on collecting JVM Metrics in this blog. In the next one we will implement a custom metric. Intent here is to get you started with the tooling before getting to custom metrics.

  • Micrometer: To expose the metrics from our application. From its website – “Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Think SLF4J, but for metrics.
  • Prometheus: To store metric data. Prometheus is a time series database that stores metric data that is periodically pulled from your running application. You have to “tell” Promethus where to pull the data from and how often to pull.
  • Grafana: To visualize the data. From its website – “Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored.

Steps

    1. Clone the sample Kafka project from https://github.com/thomasma/springkafka
    2. Install the free version of Confluent platform. All tools will be installed (including the CLI) – https://www.confluent.io/download/
    3. Start Kafka using the Confluent CLI
    4. Install and run Docker for Mac – https://docs.docker.com/v17.12/docker-for-mac/install/
    5. Install Grafana using homebrew and start it
    6. Run Prometheus in a docker container (and point it to the actuator/promethus endpoint so it can pull metrics periodically)
    7. Go to the Grafana web dashboard and watch the metrics stream in (for the JVM in our example)

Start Kafka

Start the Confluent platform using command below. This will start zookeeper, kafka, schema-registry, kafka rest proxy, etc.

To stop it (your guessed right) use…

Start Grafana

Start Prometheus

Update the promethus.yml file with paths (and IP) to your local environment before starting it

If all is good you should be able to navigate to http://localhost:9090/graph

The Spring Boot Application

This is a simple Spring Boot application that exposes a REST API to publish a message to a Kafka topic. For simplicity the same application has a consumer that will consume and print the message to STDOUT

To enable a prometheus endpoint edit the POM file to add

Edit application yaml to include

Start the application using

 

Grafana Dashboard

Grafana can be accessed via navigating to http://localhost:3000/ . Navigate to the Dashboards page and click on “Add data source”

Select Prometheus as the time series database on the next screen and configure it such…

Navigate to Manage dashboards and click on Import dashboard. Enter 9568 (you can get the ID by searching in the dashboard repository at https://grafana.com/grafana/dashboards). Click on “Upload .json file” and

Click on Import and you can navigate to the live dashboard (assuming you got all of the other pieces running successfully). You should see…

In Conclusion

You have all the pieces in place to gather metrics from the Spring Boot app. In an upcoming blog we can go into creating custom metrics. I would also recommend searching the Grafana Dashboard repository for other reusable dashboards and install them (look for Kafka and Spring dashboards).