Skip to content
Home ยป How To Collect Kubernetes Cluster Metrics With Prometheus

How To Collect Kubernetes Cluster Metrics With Prometheus

Deploying a monitoring solution for your Kubernetes ecosystem is crucial. Achieving the strongest level of observability is a main goal for all cloud professionals.

Out of the box, Kubernetes does not offer a monitoring solution to provide a better look into your systems and help identify issues in your cluster. Luckily, we have Prometheus.

Prometheus is an open-source systems monitoring and alerting toolkit that is part of the Cloud Native Computing Foundation as the second hosted project, after Kubernetes! Simply put, Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

Words cannot express how much I love the Prometheus project and its community. I have been using Prometheus both professionally and for personal projects almost every single day for the past 3 years!

What are Prometheus Metrics?

In simple terms, Prometheus metrics are numerical measurements. The term time series refers to the recording of changes over time. What users want to measure differs drastically from application to application. For a web server, it could be request times; for a database, it could be the number of active connections or active queries; for a Kubernetes cluster, it could be the number of evicted pods in a namespace, and so on. Metrics are extremely important when trying to understand why your application is working in a certain way.

Kubernetes is made up of so many components, how can I possibly monitor all of them?

This is why Prometheus works extremely well when monitoring Kubernetes clusters. Kubernetes natively exposes metrics on an endpoint on all of its components for Prometheus to easily scrape! Life can’t get better than this.

Just in case my excitement scared you there, this means that Kubernetes is already displaying all of its system metrics for each of its components without any additional configurations. To make things sweeter, Prometheus is already looking for these endpoints when deployed into a Kubernetes cluster out of the box. A fully operational and reliable monitoring solution can be achieved with very little effort. Let’s jump right into it then, shall we?

What is the Prometheus Operator?

We will be deploying the Prometheus Operator to our Kubernetes cluster to monitor our cluster. More specifically, we will be deploying the kube-prometheus-stack Helm chart, with Helm of course, built and managed by the amazing Prometheus community!

I know that may seem like a lot at once but hold on! I promise you that the heavy lifting is already done for us to get Prometheus deployed to our Kubernetes cluster.

The Prometheus Operator helps manage Prometheus and all of its glory atop a Kubernetes cluster. Without the Prometheus Operator, there are actually a lot of Prometheus components that we would have to manually manage. Now, we can manage everything by editing one configuration file with Helm. Helm, if you don’t already know, helps you manage Kubernetes applications with Charts. You can think of Helm as a templating tool that builds all necessary Kubernetes files for an application from a single configuration file for you to deploy with ease.

As I previously mentioned, we will be leveraging the kube-prometheus-stack Helm chart. This Helm chart will provide us an easy to operate end-to-end Kubernetes cluster monitoring solution with Prometheus using the Prometheus Operator. Some experienced folk may say that this is overkill to get a simple Prometheus server running in our Kubernetes Cluster, and I agree. The kube-prometheus-stack is extremely powerful and provides a lot more than just the Prometheus server, but it’s just as easy as deploying any other Helm chart so why the heck not?! (Well actually it’s because it’s heavier than deploying just the Prometheus server, but we will be turning off a lot of components so it’s not that bad!)

How to deploy Prometheus to our Kubernetes Cluster.

If you don’t already have a Kubernetes cluster deployed on your local machine, you can easily do so by following my simple guide here! We will be using Kind to deploy our Kubernetes cluster for Prometheus to monitor.

With our Kubernetes cluster deployed with Kind, let’s get Prometheus up and running to monitor our cluster.

If you don’t already have Helm installed on your computer, it’s recommended you do so now:

Via Homebrew on MacOS:

brew install helm


Via script on Linux:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh


After Helm is installed, add the Prometheus community Helm repository and ensure the repo is updated.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update


Before we proceed to install the kube-prometheus-stack, we must create a unique values.yaml file. This configuration file will be used to instruct Helm on what exactly we want deployed with what config. Without this unique values.yaml file, the default Helm chart will be deployed. We really don’t want that right now considering how massive this Helm chart really is.

The following is a good example of a values.yaml file you can use. This will disable some unnecessary components at this time. In future posts, we will be using both Grafana and Thanos so stay tuned!

alertmanager:
  enabled: false

grafana:
  enabled: false
  
thanosRuler:
  enabled: false
  


Now with this values.yaml file in our current working directory, we can run the following Helm command to deploy Prometheus via the kube-prometheus-stack. The RELEASE_NAME is the name of your Helm deployment so name it whatever you like. Before we deploy Prometheus, however, let’s quickly create a new namespace in our Kubernetes cluster to host our Prometheus stack.

kubectl create namespace monitoring
helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack -f values.yaml -n monitoring 


After your run your Helm command, you should see a similar output:

Check out what you now have running inside of your monitoring namespace! If you list your Kubernetes pods you’ll find that you have Prometheus successfully deployed to your cluster! Awesome!

All of the Prometheus pods here play an important role in monitoring your Kubernetes cluster, but we are going to focus on one pod in particular, the prometheus-prom-kube-prometheus-stack-prometheus-0 pod. Depending on your Helm release name, the pod name may be slightly different. This pod is the Prometheus server! As you read this, your Prometheus server is collecting metrics from your Kubernetes cluster every 30 seconds(this is the scrape interval by default).

How to view Kubernetes metrics with Prometheus.

Kubernetes metrics are now being collected with Prometheus deployed to your Kubernetes cluster, but how can we view these metrics to actually make use of them? The Prometheus server scrapes the Kubernetes endpoints for their metrics and retains them for a short period of time. If this sounds like a problem to you then don’t you worry, because it is. Data retention is not exactly native to Prometheus and other applications exist for this very purpose but I will cover that in a future post.

To view these metrics, we can simply port forward the Prometheus server application for us to access through our local machine.

kubectl port-forward prometheus-prom-kube-prometheus-stack-prometheus-0 9090

If we now open a web browser and go to localhost:9090, we can access the Prometheus server.

Cool, we’re in! From here we can query all of our available metrics. If we simply type up in the query search, it gives us the up metric for each of the application deployed in our Kubernetes cluster.

How about something a bit more interesting? Here’s a query to get the current memory usage in bytes for each of the pods in the monitoring namespace.

sum by (pod) (rate(container_memory_working_set_bytes{namespace="monitoring"}[1m]))

Wrapping up

In this post we looked at how to collect Kubernetes cluster metrics with Prometheus. Deploying the Prometheus Operator with the kube-prometheus-stack Helm chart into your Kubernetes cluster is just scratching the surface. The world of monitoring is extremely vast. Tools like Prometheus are both powerful and helpful, and you should leverage them to better strengthen observability into your cloud infrastructure.

Thank you so much for reading and I really hope you learned something new! If you have any questions or suggestions, please add them in the comments section below.

Leave a Reply

Your email address will not be published. Required fields are marked *