Capturing metrics from your system is critical to understanding its internal behavior and to tune its performance. Without this you are operating in the blind. In this post we will go through how you can gather metrics from a Spring Boot application using Prometheus, Grafana and Micrometer.
The term Observability is widely used nowadays. I do not want to get into a religious debate on what it may be or may not be. I look at this by asking myself — “if i am woken up at midnight with a production issue, can I look at my systems’ collected health related data and ask various questions to help me narrow down to the root cause within a few minutes”. The most common types of data we collect are – metrics, logs and traces. All three are important, but fusing them into one cohesive dataset to allow for faster consumption is the key. I suggest you do some reading to familiarize yourself with what is noise from vendors related to Observability, vs. the real intent. I recommend reading…
- Charity Major’s article (and her references) at Observability – A 3 Year Retrospective
- Ben Sigleman’s article Three Pillars with Zero Answers – Towards a New Scorecard for Observability
This blog specifically deals with collecting Metrics which is one type of timeseries data that can assist with Observability. We will get metrics from a simple Sprint Boot application that publishes a message to Kafka as a result of a REST API call. The app also acts as the consumer since this is a test application. We will focus on collecting JVM Metrics in this blog. In the next one we will implement a custom metric. Intent here is to get you started with the tooling before getting to custom metrics.
- Micrometer: To expose the metrics from our application. From its website – “Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Think SLF4J, but for metrics.“
- Prometheus: To store metric data. Prometheus is a time series database that stores metric data that is periodically pulled from your running application. You have to “tell” Promethus where to pull the data from and how often to pull.
- Grafana: To visualize the data. From its website – “Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored.“
Steps
-
- Clone the sample Kafka project from https://github.com/thomasma/springkafka
- Install the free version of Confluent platform. All tools will be installed (including the CLI) – https://www.confluent.io/download/
- Start Kafka using the Confluent CLI
- Install and run Docker for Mac – https://docs.docker.com/v17.12/docker-for-mac/install/
- Install Grafana using homebrew and start it
- Run Prometheus in a docker container (and point it to the actuator/promethus endpoint so it can pull metrics periodically)
- Go to the Grafana web dashboard and watch the metrics stream in (for the JVM in our example)
Start Kafka
Start the Confluent platform using command below. This will start zookeeper, kafka, schema-registry, kafka rest proxy, etc.
|
1 |
confluent local start |
To stop it (your guessed right) use…
|
1 |
confluent local stop |
Start Grafana
|
1 2 3 4 5 6 7 8 |
# to install brew install grafana # to start brew services start grafana # to stop brew services stop grafana |
Start Prometheus
Update the promethus.yml file with paths (and IP) to your local environment before starting it
|
1 |
docker run -d --name=prometheus -p 9090:9090 -v /Users/mathew/eden_projects/springkafka/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml |
If all is good you should be able to navigate to http://localhost:9090/graph
The Spring Boot Application
This is a simple Spring Boot application that exposes a REST API to publish a message to a Kafka topic. For simplicity the same application has a consumer that will consume and print the message to STDOUT
To enable a prometheus endpoint edit the POM file to add
|
1 |
<dependency><br> <groupId>io.micrometer</groupId><br> <artifactId>micrometer-registry-prometheus</artifactId><br></dependency> |
Edit application yaml to include
|
1 2 3 4 5 |
management: endpoints: web: exposure: include: health,info,env,metrics,prometheus |
Start the application using
|
1 2 3 4 5 6 7 8 9 10 |
mvn clean spring-boot:run # if the app is setup correctly the following should pull metrics compatible with prometheus curl http://localhost:9080/actuator/prometheus # to publish a message call curl -XPOST -H "Content-Type: application/json" -d '{"text": "hello"}' http://localhost:9080/publish # to publish in a loop while true; do curl -XPOST -H "Content-Type: application/json" -d '{"text": "hello"}' http://localhost:9080/publish; sleep 1; done; |
Grafana Dashboard
Grafana can be accessed via navigating to http://localhost:3000/ . Navigate to the Dashboards page and click on “Add data source”
Select Prometheus as the time series database on the next screen and configure it such…
Navigate to Manage dashboards and click on Import dashboard. Enter 9568 (you can get the ID by searching in the dashboard repository at https://grafana.com/grafana/dashboards). Click on “Upload .json file” and
Click on Import and you can navigate to the live dashboard (assuming you got all of the other pieces running successfully). You should see…
In Conclusion
You have all the pieces in place to gather metrics from the Spring Boot app. In an upcoming blog we can go into creating custom metrics. I would also recommend searching the Grafana Dashboard repository for other reusable dashboards and install them (look for Kafka and Spring dashboards).