Observability
ELK Stack
The Elasticsearch, Logstash and Kibana set for collecting, full text searching and visualizing logs and events.
What it is
ELK is three Elastic products: Elasticsearch (search engine and store), Logstash (a data processing pipeline) and Kibana (a UI for search and dashboards). Elasticsearch appeared in 2010, and the modern stack is called the Elastic Stack because lightweight Beats agents and Elastic Agent were added.
A typical flow: apps write logs, an agent (Filebeat) ships them to Logstash or straight to Elasticsearch, the data is indexed, and in Kibana you search, filter and visualize it.
The main strength is fast full text search and field analytics. The price: resource demands and cluster administration. Elasticsearch licensing changed: since 2021 Elastic License and SSPL, and since 2024 an AGPL-3.0 option was added again.
Use it when
- You need fast full text search over logs and complex field analytics.
- Centralized logging for a large infrastructure.
- Security analytics (SIEM) or event search.
- You already have Elasticsearch administration experience.
Avoid it when
- A limited server budget: the stack uses a lot of memory and disk, Loki will be cheaper.
- There is no team to manage the cluster, shards and upgrades.
- You only need metric visualization: Prometheus and Grafana are enough.
Key concepts
- Index / Shard
- An index is split into shards distributed across nodes for scaling and fault tolerance.
- Inverted index
- A structure that maps terms to documents and enables fast full text search.
- Logstash pipeline
- Input, filter, output: reading, parsing (grok) and shipping data.
- Kibana
- Discover for search, Lens for visualizations, dashboards and alerts.
- ILM
- Index Lifecycle Management: automatic movement of indexes between hot, warm, cold and deletion.
Example
Interview questions
1. What does ELK include and what does each part do?
Elasticsearch stores, indexes and searches data, Logstash collects and transforms data, Kibana gives the UI for search and dashboards. Lightweight Beats agents often replace Logstash at the edge.
2. What is a shard and why is it needed?
A shard is a part of an index stored on one node. Shards spread data and load across the cluster, and shard replicas provide fault tolerance.
3. How does ELK differ from Loki?
ELK indexes the whole text, so search is fast and flexible but uses more resources. Loki indexes only labels and stores logs cheaply, but full text search is slower.
4. How do you control data growth?
Use ILM: move old indexes to cheaper nodes and delete them by age, and also limit fields and index templates.
Interview questions from the bank
Related technologies
- ElasticsearchElasticsearch is a distributed search and analytics engine built on Lucene with a REST API and JSON documents.
- OpenSearchOpenSearch is an open source distributed search and analytics engine, a fork of Elasticsearch 7.10 under Apache-2.0.
- Grafana LokiA log storage system that indexes only labels, like Prometheus, and therefore costs less than full text solutions.
- GrafanaA visualization platform: dashboards and alerts on top of Prometheus, Loki, SQL databases and dozens of other sources.
- Apache KafkaKafka is a distributed event log for high-throughput data streaming with durable storage.
- OpenTelemetryA single open standard and SDK set for traces, metrics and logs, independent of the vendor.
Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).