Capturing metrics from your system is critical to understanding its internal behavior and to further improve its performance. Without this, you are operating in the blind. In this post, we will go through how you can gather metrics from a Spring Boot application using Prometheus, Grafana, and Micrometer.
The term Observability is widely used nowadays. I do not want to get into a religious debate on what it may be or may not be. I look at this by asking myself — “if i am woken up at midnight with a production issue, can I look at my systems’ collected health related data and ask various questions to help me narrow down to the root cause within a few minutes”. The most common types of data we collect are – metrics, logs, and traces. All three are essential, but fusing them into one cohesive dataset to allow for faster consumption is the key. I suggest you do some reading to familiarize yourself with what noise from vendors is related to Observability, vs. the real intent. I recommend reading…
- Charity Major’s article (and her references) at Observability – A 3 Year Retrospective
- Ben Sigleman’s article Three Pillars with Zero Answers – Towards a New Scorecard for Observability
This blog deals explicitly with collecting Metrics, which is one type of time series data that can assist with Observability. We will get metrics from a simple Spring Boot application that publishes a message to Kafka in response to a REST API call. The app also acts as the consumer since this is a test application. We will focus on collecting JVM Metrics in this blog. The intent here is to get you started with the tooling before getting to custom metrics.
- Micrometer: To expose the metrics from our application. From its website – “Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Think SLF4J, but for metrics.“
- Prometheus: To store metric data. Prometheus is a time series database that stores metric data that is periodically pulled from your running application. You have to “tell” Prometheus where to pull the data from and how often to pull.
- Grafana: To visualize the data. From its website – “Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored.“
Steps
-
- Clone the sample Kafka project from https://github.com/thomasma/springkafka
- Install the open-source version of Apache Kafka. Refer to https://kafka.apache.org/documentation/#quickstart
- Start Kafka
- Install and run Docker for Mac – https://docs.docker.com/v17.12/docker-for-mac/install/
- Install Grafana using Homebrew and start it.
- Run Prometheus in a Docker container (and point it to the actuator/promethus endpoint so it can pull metrics periodically)
- Go to the Grafana web dashboard and watch the metrics stream in (for the JVM in our example)
Start Kafka
Start the Kafka server using the command below.
|
1 |
bin/kafka-server-start.sh config/server.properties |
To stop it (you guessed right) use…
|
1 |
bin/kafka-server-stop.sh config/server.properties |
Start Grafana
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# to install brew install grafana brew install prometheus # to start brew services start grafana # one-time config - before starting prometheus edit the file /opt/homebrew/etc/prometheus.yml # and add this - job_name: 'spring-actuator' metrics_path: '/actuator/prometheus' scrape_interval: 5s static_configs: - targets: ['localhost:9080'] # now start prometheus brew services start prometheus # to stop brew services stop grafana brew services stop prometheus # if you do not want these services to start every time brew service cleanup grafana brew service cleanup prometheus |
The Spring Boot Application
This is a simple Spring Boot application that exposes a REST API to publish a message to a Kafka topic. For simplicity, the same application has a consumer that will consume and print the message to STDOUT.
To enable a Prometheus endpoint, edit the POM file to add
|
1 2 3 4 |
<dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency> |
Edit the application yaml to include
|
1 2 3 4 5 |
management: endpoints: web: exposure: include: health,info,env,metrics,prometheus |
Start the application using (ensure Grafana and Prometheus have already been started)
|
1 2 3 4 5 6 7 8 9 10 |
mvn clean spring-boot:run # if the app is setup correctly the following should pull metrics compatible with prometheus curl http://localhost:9080/actuator/prometheus # to publish a message call curl -XPOST -H "Content-Type: application/json" -d '{"text": "hello"}' http://localhost:9080/publish # to publish in a loop while true; do curl -XPOST -H "Content-Type: application/json" -d '{"text": "hello"}' http://localhost:9080/publish; sleep 1; done; |
Grafana Dashboard
Grafana can be accessed by navigating to http://localhost:3000/. Navigate to the Dashboards page and click on “Add data source.”
Select Prometheus as the time series database on the next screen and configure it such…
Navigate to Manage dashboards and click on Import dashboard. Enter 9568 (you can get the ID by searching in the dashboard repository at https://grafana.com/grafana/dashboards). Click on “Upload .json file” and
Click on Import and you can navigate to the live dashboard (assuming you got all of the other pieces running successfully). You should see…
In Conclusion
You have all the pieces in place to gather metrics from the Spring Boot app. In an upcoming blog, we can go into creating custom metrics. I would also recommend searching the Grafana Dashboard repository for other reusable dashboards and installing them (look for Kafka and Spring dashboards).