Caching and search
Elasticsearch
Elasticsearch is a distributed search and analytics engine built on Lucene with a REST API and JSON documents.
What it is
Elasticsearch stores JSON documents and builds an inverted index on top of Apache Lucene. Because of that, full-text search, filters and aggregations run in milliseconds even over billions of records.
Data is split into shards and replicated across cluster nodes. It is often used for site search, log analysis (the ELK stack), metrics and security, and more recently for vector search too.
About the license. In 2021 Elastic moved from Apache-2.0 to SSPL and the Elastic License, which led to the OpenSearch fork by AWS. In 2024 Elastic added AGPLv3 as another option for the Elasticsearch code.
Use it when
- You need full-text search with relevance, facets and highlighting.
- You analyze large volumes of logs and events.
- You need near real-time aggregations and analytics.
- You build hybrid search: text together with vectors.
Avoid it when
- It would be the primary transactional database: Elasticsearch does not replace a relational DB.
- You need simple search over a small catalog: Meilisearch or Typesense are easier to set up.
- You cannot afford to operate a cluster: it is memory hungry.
Key concepts
- Inverted index
- A structure that stores, for each term, the list of documents where it appears.
- Shard / replica
- A shard is a part of an index for horizontal scaling, a replica is its copy for fault tolerance.
- Analyzer
- A chain of tokenization and filters (lowercase, stemming) that prepares text for indexing.
- Mapping
- The index schema: field types and how each field is analyzed.
- Aggregation
- Computations over results: grouping, histograms, metrics.
Example
Cheat sheet
| Command or code | What it does |
|---|---|
GET /_cluster/health | Cluster health (green, yellow, red) |
PUT /products | Create an index |
POST /products/_doc { "name": "Phone" } | Add a document |
GET /products/_search { "query": { "match": { "name": "phone" } } } | Full text search |
{ "query": { "term": { "status": "active" } } } | Exact match on a keyword field |
{ "bool": { "must": [...], "filter": [...] } } | Combine clauses: filter does not affect relevance |
GET /products/_mapping | View the index mapping |
{ "aggs": { "by_brand": { "terms": { "field": "brand" } } } } | Aggregation: group by a field |
GET /_cat/indices?v | Indexes with sizes |
DELETE /products | Delete an index |
How to learn it
- Understand that it is an inverted index search engine, not a primary transactional database.
- Run Elasticsearch and Kibana in Docker and open Dev Tools to send queries.
- Create an index, add a few documents through
_docand find them with amatchquery. - Learn mappings:
textversuskeyword, analyzers, tokenization, and why changing a mapping later is hard. - Master the Query DSL:
boolwithmust,filter,should, ranges, highlighting, pagination. - Tune relevance: field boosts, synonyms, fuzzy search, autocomplete.
- Try aggregations (terms, date_histogram) for facets and simple analytics.
- Learn shards, replicas, ILM and wire indexing from your app through the official client.
Interview questions
1. What is an inverted index?
It maps terms to the documents that contain them, like the index at the back of a book. A word search does not scan every document, it takes a ready list.
2. Why shards and replicas?
Shards spread an index across nodes to scale volume and queries. Replicas copy shards for fault tolerance and speed up reads.
3. Why not use Elasticsearch as the primary database?
It is near real-time rather than immediately consistent, has no full transactions and can lose data when misconfigured. Keep the source of truth in a database and index a copy in Elasticsearch.
4. How does Elasticsearch differ from OpenSearch?
OpenSearch is a fork of Elasticsearch 7.10 by AWS under Apache-2.0, now run by a community under the Linux Foundation. The APIs are similar but drift apart over time.
Interview questions from the bank
- Compare three ways to store a suggestion index: the completion suggester in Elasticsearch, a field with an edge_ngram analyzer, and a prefix index in Redis (a sorted set with the top 10 per prefix). Describe the data model of each, the endpoint API, and what you would choose for 2 million products and 10 million popular queries.SeniorSystem design
- After creating a listing, the user is redirected to the My listings page, and the new listing is not there: the Elasticsearch projection lags by 1 to 3 seconds. Users press again and create duplicates. What are the fix options, and which would you choose?SeniorSystem design
Comparisons with this technology
Related technologies
- OpenSearchOpenSearch is an open source distributed search and analytics engine, a fork of Elasticsearch 7.10 under Apache-2.0.
- MeilisearchMeilisearch is a fast search engine written in Rust with search-as-you-type, typo tolerance and simple setup.
- TypesenseTypesense is a fast C++ search engine with typo tolerance, facets and search-as-you-type that keeps its index in memory.
- ELK StackThe Elasticsearch, Logstash and Kibana set for collecting, full text searching and visualizing logs and events.
- PostgreSQLA powerful open source relational database with extensions, JSONB, full-text search and strong SQL support.
Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).