Observability
Prometheus
A monitoring system with a time series database: it scrapes metrics in a pull model and offers the PromQL query language.
What it is
Prometheus was created at SoundCloud in 2012. It later became the second CNCF project after Kubernetes and reached graduated status (2018). It is the metrics standard in the Kubernetes world.
The pull model: Prometheus polls the /metrics HTTP endpoints of your services at a set interval and stores values as time series with labels. Short-lived jobs can use the Pushgateway.
Data is read with PromQL, and alert rules are built on top of it. Notifications are sent by a separate Alertmanager. For graphs you usually plug in Grafana.
Use it when
- Monitoring Kubernetes and microservices with dynamic target discovery.
- You need time series metrics: latency, error rate, load.
- You want alerts based on a flexible query language.
- You want an open source solution with no vendor lock-in.
Avoid it when
- You need logs or traces: Loki, Elasticsearch and Jaeger are for those.
- You need long-term storage across many clusters without extra components (Thanos, Mimir).
- You need 100% exact event accounting such as billing: metrics are aggregated and sampled.
Key concepts
- Metric types
- Counter (only grows), Gauge (goes up and down), Histogram and Summary (distributions such as latency).
- Labels
- Key-value pairs that slice a metric (method, path, status). Too many unique values cause a cardinality problem.
- Scrape
- Polling a
/metricsendpoint on a schedule, with targets defined in config or discovered automatically. - PromQL
- A time series query language:
rate(),sum by(),histogram_quantile(). - Alertmanager
- Groups, deduplicates and routes alerts to Slack, email or PagerDuty.
Example
Interview questions
1. How does the Prometheus pull model differ from push?
In pull, Prometheus polls services itself, so it is easy to see that a target is down and you do not configure each service with a collector address. Push suits short-lived jobs, which is what the Pushgateway is for.
2. What metric types exist?
A Counter only grows (request count), a Gauge can move both ways (memory in use), Histogram and Summary describe the distribution of values like latency. For counters you usually take
rate().3. What is label cardinality and why is it a problem?
It is the number of unique combinations of label values. A label with unbounded values, such as user ID, creates millions of series and exhausts memory and disk.
4. Why Alertmanager if Prometheus already has alerts?
Prometheus only evaluates rules and generates alerts. Alertmanager groups them, removes duplicates, stays silent during maintenance and sends them to the right channels.
Interview questions from the bank
- What is observability and how do logs, metrics and traces differ?JuniorDevOps
- How does RED differ from USE? What are SLI, SLO and error budget, and which alerts wake a person at night?MiddleDevOps
- What metric types does Prometheus have and why is a histogram almost always used for HTTP latency?MiddleDevOps
- A team has 200 alerts a day, everything is red, and nobody reacted to a real incident for 40 minutes. What do you do?MiddleDevOps
- After a release Prometheus started crashing with OOM, and a label with values like
user_idappeared in the metrics. What happened and what do you do?SeniorDevOps - SLO 99.9% over 30 days. The alert
error_ratio > 0.001 for 5mwakes people at night over every blip. How do you redo it?SeniorDevOps
Comparisons with this technology
Related technologies
- GrafanaA visualization platform: dashboards and alerts on top of Prometheus, Loki, SQL databases and dozens of other sources.
- KubernetesKubernetes is a container orchestration system that deploys, scales and heals applications across a cluster of servers.
- Grafana LokiA log storage system that indexes only labels, like Prometheus, and therefore costs less than full text solutions.
- OpenTelemetryA single open standard and SDK set for traces, metrics and logs, independent of the vendor.
- DockerDocker packages an application with its dependencies into a container that runs the same on a laptop, a server and the cloud.
- DatadogA commercial SaaS observability platform: metrics, logs, traces, APM and security in one place.
Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).