Observability
Grafana Loki
A log storage system that indexes only labels, like Prometheus, and therefore costs less than full text solutions.
What it is
Loki was created by Grafana Labs in 2018 with the idea of a Prometheus for logs. It indexes not the whole text but only labels (service, namespace, level), and stores the log lines themselves as compressed chunks in cheap storage such as S3.
Logs reach Loki through an agent: Grafana Alloy, Promtail or the OpenTelemetry Collector. Queries are written in LogQL, similar to PromQL: first select a stream by labels, then filter by text and parse.
The approach is much cheaper and simpler to run than Elasticsearch, but full text search across the whole volume is slower. Loki integrates well with Grafana: from a metric graph you can jump to logs from the same moment.
Use it when
- You already use Prometheus and Grafana and want logs in the same stack.
- High log volume with a limited storage budget.
- Kubernetes, where you need to find pod logs by labels quickly.
- Searching by labels followed by text filtering is enough.
Avoid it when
- You need fast full text search and complex field analytics: Elasticsearch is better.
- Labels have huge cardinality (user ID as a label).
- The AGPL license does not fit your scenario.
Key concepts
- Label index
- The index is built only from stream labels, so it is small and writes are cheap.
- Stream
- A set of logs with the same combination of labels.
- LogQL
- The query language: stream selector
{app="api"}, filter|= "error", metrics from logsrate(). - Agent
- Grafana Alloy, Promtail or another collector that reads files and ships logs to Loki with labels.
- Object storage
- Log chunks are stored in S3, GCS or the local filesystem.
Example
Interview questions
1. Why is Loki cheaper than Elasticsearch?
It indexes only labels rather than the whole log text, so the index is small and writes and storage cost less. The price is slower full text search.
2. Which labels should you use in Loki?
Labels with few values: service, environment, namespace, level. High cardinality (user ID, URL) should stay in the log text, not in labels.
3. How do you get metrics from logs in Loki?
LogQL has metric queries such as
rate({app="api"} |= "error" [5m]). They turn a stream of lines into a numeric series for graphs and alerts.
Interview questions from the bank
- What is observability and how do logs, metrics and traces differ?JuniorDevOps
- How do you set up logging so that one
requestIdortrace_idfinds all lines of a request across all services?MiddleDevOps - After a release Prometheus started crashing with OOM, and a label with values like
user_idappeared in the metrics. What happened and what do you do?SeniorDevOps - The security team found tokens, passwords and password reset links in the logs. How do you fix this systematically?MiddleWeb security
Related technologies
- GrafanaA visualization platform: dashboards and alerts on top of Prometheus, Loki, SQL databases and dozens of other sources.
- PrometheusA monitoring system with a time series database: it scrapes metrics in a pull model and offers the PromQL query language.
- ELK StackThe Elasticsearch, Logstash and Kibana set for collecting, full text searching and visualizing logs and events.
- OpenTelemetryA single open standard and SDK set for traces, metrics and logs, independent of the vendor.
- KubernetesKubernetes is a container orchestration system that deploys, scales and heals applications across a cluster of servers.
Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).