Observability
Prometheus
A monitoring system with a time series database: it scrapes metrics in a pull model and offers the PromQL query language.
What it is
Prometheus was created at SoundCloud in 2012. It later became the second CNCF project after Kubernetes and reached graduated status (2018). It is the metrics standard in the Kubernetes world.
The pull model: Prometheus polls the /metrics HTTP endpoints of your services at a set interval and stores values as time series with labels. Short-lived jobs can use the Pushgateway.
Data is read with PromQL, and alert rules are built on top of it. Notifications are sent by a separate Alertmanager. For graphs you usually plug in Grafana.
Use it when
- Monitoring Kubernetes and microservices with dynamic target discovery.
- You need time series metrics: latency, error rate, load.
- You want alerts based on a flexible query language.
- You want an open source solution with no vendor lock-in.
Avoid it when
- You need logs or traces: Loki, Elasticsearch and Jaeger are for those.
- You need long-term storage across many clusters without extra components (Thanos, Mimir).
- You need 100% exact event accounting such as billing: metrics are aggregated and sampled.
Key concepts
- Metric types
- Counter (only grows), Gauge (goes up and down), Histogram and Summary (distributions such as latency).
- Labels
- Key-value pairs that slice a metric (method, path, status). Too many unique values cause a cardinality problem.
- Scrape
- Polling a
/metricsendpoint on a schedule, with targets defined in config or discovered automatically. - PromQL
- A time series query language:
rate(),sum by(),histogram_quantile(). - Alertmanager
- Groups, deduplicates and routes alerts to Slack, email or PagerDuty.
Example
Interview questions
1. How does the Prometheus pull model differ from push?
In pull, Prometheus polls services itself, so it is easy to see that a target is down and you do not configure each service with a collector address. Push suits short-lived jobs, which is what the Pushgateway is for.
2. What metric types exist?
A Counter only grows (request count), a Gauge can move both ways (memory in use), Histogram and Summary describe the distribution of values like latency. For counters you usually take
rate().3. What is label cardinality and why is it a problem?
It is the number of unique combinations of label values. A label with unbounded values, such as user ID, creates millions of series and exhausts memory and disk.
4. Why Alertmanager if Prometheus already has alerts?
Prometheus only evaluates rules and generates alerts. Alertmanager groups them, removes duplicates, stays silent during maintenance and sends them to the right channels.
Interview questions from the bank
- Users report that checkout sometimes takes 10 seconds. You have logs, Prometheus metrics and OpenTelemetry traces. Where do you start, and what does each of these signals tell you?JuniorDevOps
- A new checkout API launches in a month, and before launch you have to set up dashboards and on call. Which metrics do you collect for the service, which for resources, what SLO do you propose, and what exactly wakes a person at night?MiddleDevOps
- On the dashboard, API p99 latency is computed as the average of each pod's p99 reported by a Summary. The graph is flat, yet users complain about slowness. What is wrong, and what should you use instead?MiddleDevOps
- A team has 200 alerts a day, everything is red, and nobody reacted to a real incident for 40 minutes. What do you do?MiddleDevOps
- After a release Prometheus started crashing with OOM, and a label with values like
user_idappeared in the metrics. What happened and what do you do?SeniorDevOps - SLO 99.9% over 30 days. The alert
error_ratio > 0.001 for 5mwakes people at night over every blip. How do you redo it?SeniorDevOps
Comparisons with this technology
Related technologies
- GrafanaA visualization platform: dashboards and alerts on top of Prometheus, Loki, SQL databases and dozens of other sources.
- KubernetesKubernetes is a container orchestration system that deploys, scales and heals applications across a cluster of servers.
- Grafana LokiA log storage system that indexes only labels, like Prometheus, and therefore costs less than full text solutions.
- OpenTelemetryA single open standard and SDK set for traces, metrics and logs, independent of the vendor.
- DockerDocker packages an application with its dependencies into a container that runs the same on a laptop, a server and the cloud.
- DatadogA commercial SaaS observability platform: metrics, logs, traces, APM and security in one place.
Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).