This is our second post related to Kubernetes observability tutorial series, We will find out how to monitor all aspects of your applications running in Kubernetes, including the following:
- Ingesting and analysing logs
- performance collection and health metrics
- Monitoring application performance with Elastic APM
We will use Elastic Observability for analysis of container metrices in Kibana by the use of Metrics app and out-of-the-box dashboards.
Collection of metrices from Kubernetes
Same as the targets of the logs of Kubernetes , collection of metrics from Kubernetes can be a challenge for a few reasons:
- Running of kubernets on different host which must be monitored by some metrices such as CPU, memory, disk utilisation, and disk and network I/O.
- Own set of metrics are also produced by kubernets containers which are mini VMs
- As both applications can run on application server and database so they have their own reporting methods.
Monitoring of Kubernets deployment becomes complex when many technologies are used by the organizations for handling metrices. We can use Elastic Observability which combine the logs and APM data for analysis and visibility purpose.
Collection of K8s metrics with Metricbeat
Metricbeat is similar to Filebeat and is the only component we can use for collection of various metrics from the pods running in our Kubernetes cluster, as well as Kubernetes’ own cluster metrics. Its modules gives a quick and easy way for picking up metrics from various sources and shipping them to Elasticsearch as ECS-compatible events, which are then corelated to logs, uptime and APM data. Metricbeat is deployed on Kubernetes in following two ways:
- A single pod can be used for collection of Kubernetes metrics. It uses kube-state-metrics for collection of cluster-level metrics.
- Metricbeat can be deployed as a single instance by Daemonset. It is used for collecting metrices from pods which are deployed on some host.Metricbeat communicates with kubelet APIs for getting the components which are running on that host.It uses different method such as auto-detection for further asking the components for collection of technology-specific metrics.
Before you get started: The following tutorial has a Kubernetes environment setup. We’ve created a supplementary blog that walks you through the process of setting up a single-node Minikube environment with a demo application for running the rest of the activities.
Collection of host, Docker, and Kubernetes metrics
Every DaemonSet instance is used to collect host, Docker and Kubernetes metrics, are defined in following way in the YAML
config $HOME/k8s-o11y-workshop/metricbeat/metricbeat.yml
:
System (host) metric configuration
system.yml: |-
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
- core
- diskio
# - socket
processes: ['.*']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
metric configuration of Docker
docker.yml: |-
- module: docker
metricsets:
- "container"
- "cpu"
- "diskio"
- "event"
- "healthcheck"
- "info"
# - "image"
- "memory"
- "network"
hosts: ["unix:///var/run/docker.sock"]
period: 10s
enabled: true
metrics configuration of Kubernetes
it has metrics collection from pods deployed onto the host after communication with kubelet API:
kubernetes.yml: |-
- module: kubernetes
metricsets:
- node
- system
- pod
- container
- volume
period: 10s
host: ${NODE_NAME}
hosts: ["localhost:10255"]
- module: kubernetes
metricsets:
- proxy
period: 10s
host: ${NODE_NAME}
hosts: ["localhost:10249"]
check out the Metricbeat documentation for more information about Metricbeat modules and data behind the metricsets,
Collection of Kubernetes state metrics and events
A single instance is used for collection of kubernets metrices which are deployed to be joined with metrices API for monitering the changes in state of objects. This defines the config:
$HOME/k8s-o11y-workshop/Metricbeat/Metricbeat.yml
:
kubernetes.yml: |-
- module: kubernetes
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_pod
- state_container
# Uncomment this to get k8s events:
- event
period: 10s
host: ${NODE_NAME}
hosts: ["kube-state-metrics:8080"]
Autodiscovery of Metricbeat by the use of pod annotations
The deployment of Metricbeat DaemonSet can autodetect the components running in the pods and application of the modules of Metricbeat for collection of metrices which are specific to technology. Pod annotations can be used for enabling the autodiscovery , it indicates modules for specific configurations.This section of Metricbeat config enables Kubernetes-based autodiscovery.
$HOME/k8s-o11y-workshop/metricbeat/metricbeat.yml
:
metricbeat.autodiscover:
providers:
- type: kubernetes
host: ${NODE_NAME}
hints.enabled: true
There are two components used for autodiscovery:
- NGINX definition $HOME/k8s-o11y-workshop/nginx/nginx.yml
template:
metadata:
labels:
app: nginx
annotations:
co.elastic.metrics/module: nginx
co.elastic.metrics/hosts: '${data.host}:${data.port}'
- MySQL definition
$HOME/k8s-o11y-workshop/mysql/mysql.yml
template:
metadata:
labels:
app: mysql
annotations:
co.elastic.metrics/module: mysql
co.elastic.metrics/hosts: 'root:[email protected](${data.host}:${data.port})/'
See the Metricbeat documentation for more information .
Collection of application metrics, Prometheus-style
The Spring Boot petclinic application shows a range of application-specific metrics which are exposed in a form that can be scraped by Prometheus. You can navigate to the application http endpoint at http://<public-ip>:30080/metrics/Prometheus to check how the metrics are reported. we will be using Metricbeat for collection of these metrics and using Elastic components for all our observability needs.
Here is an example of what our application reports:
This configuration hints in petclinic YAML deployment config to tell Metricbeat to collect these metrics using the Prometheus module.
$HOME/k8s-o11y-workshop/petclinic/petclinic.yml
:
template:
metadata:
labels:
app: petclinic
annotations:
co.elastic.metrics/module: prometheus
co.elastic.metrics/hosts: '${data.host}:${data.port}'
co.elastic.metrics/metrics_path: '/metrics/prometheus'
co.elastic.metrics/period: 1m
Generally, Metricbeat can add up or replace the Prometheus server . If it is already deployed and using Prometheus server, Metricbeat can be used to send the metrics out of the server with the help of Prometheus Federation API, and provides visibility across multiple Prometheus servers, Kubernetes namespaces and clusters, enables correlation of Prometheus metrics with Logs, APM and uptime events. If a simplified monitoring architecture is needed then we can use Metricbeat for collecting Prometheus metrics and send them straight into Elasticsearch.
Enrichment of Metadata
The following processors help all the events collected by Metricbeat to be enriched by.
$HOME/k8s-o11y-workshop/metricbeat/metricbeat.yml
:
processors:
- add_cloud_metadata:
- add_host_metadata:
- add_kubernetes_metadata:
- add_docker_metadata:
This helps in correlating the metrics with the hosts, Kubernetes pods, Docker containers, and cloud-provider infrastructure metadata and correlating with other pieces of observability puzzle, for example application performance monitoring data and logs.
Metrics in Kibana
The configuration of Metricbeat shows the following views in the Metrics app. . Kibana helps to search things with its search bar.It helps in filtering and zomming things when we are looking for some things. We have used only one host, see below:
Host infrastructure metrics
Docker infra and metrics (table view)
Kubernetes infra and metrics
Metrics explorer
Out-of-the-box Kibana dashboards
Metricbeat has many of pre-built Kibana dashboards which we can easily be added to your cluster with a single command. We can use these dashboards as it is, or as a point to start custom dashboards tailored in the way we want . See the below dashboards:
Host
System
Docker
NGINX
MySQL
CONCLUSION
In this article, we saw a collecting application and Kubernetes metrics with Metricbeat. We can start monitoring our systems and infrastructure today. Sign up for a free trial of Elasticsearch Service on Elastic Cloud, or download the Elastic Stack and host it yourself.
Once you have it all and running, check the availability of your hosts with uptime monitoring, and check the applications running on your hosts with Elastic APM. You will have an observable system, completely integrated with your new metrics cluster. If you run into any difficulty or have questions, jump over to our comments section— we’re here to help.