Learning ElasticSearch

1. Fundamentals

1.1 Origin: Lucene

Lucene: Apache’s open-source Java full-text search engine library.
Advantages: easy to extend, high performance (pure Java, embeddable).
Drawbacks: complex to use — you must handle low-level details like index creation and query parsing, with no distributed support.
Elasticsearch is built on Lucene, providing a distributed, easy-to-use RESTful API.

1.2 The tech stack (ELK)

Elasticsearch: the storage, search, and analytics engine.
Logstash: a server-side data processing pipeline that collects and transforms data before sending it to ES.
Kibana: a visualization platform for building charts and dashboards and managing ES.
Beats: lightweight data shippers that send to Logstash or ES.

1.3 Installing with Docker

Elasticsearch (single node):

docker run -d --name es \
  -e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
  -e "discovery.type=single-node" \
  -v es-data:/usr/share/elasticsearch/data \
  -v es-plugins:/usr/share/elasticsearch/plugins \
  --privileged \
  --network testNet \
  -p 9200:9200 -p 9300:9300 \
  elasticsearch:7.12.1

9200: the HTTP API port
9300: the internal node communication port
discovery.type=single-node: single-node mode (for testing)

Kibana:

docker run -d --name kibana \
  -e ELASTICSEARCH_HOSTS=http://es:9200 \
  --network=testNet \
  -p 5601:5601 \
  kibana:7.12.1

1.4 The Inverted Index

Forward index: document ID → document content. Good for exact lookup by ID, but fuzzy search requires scanning all documents — inefficient.
Inverted index:
- Document: a piece of JSON data stored in ES.
- Term: the smallest unit of a document after tokenization.
- Structure: term → list of document IDs (with position, frequency, etc.).
- Advantage: quickly locates all documents containing a given term, enabling efficient full-text search.

1.5 The Analyzer

A component that splits text into terms, made up of three parts:

Character Filter: preprocessing (e.g. stripping HTML tags).
Tokenizer: tokenizes by rules.
Token Filter: further processes terms (lowercasing, removing stop words).

The IK analyzer: the most commonly used Chinese tokenization plugin.

Install: download the matching-version zip into the plugins/ik directory and restart ES.
Two modes:
- ik_smart: coarsest-grained splitting.
- ik_max_word: finest-grained splitting.
Custom dictionaries: edit IKAnalyzer.cfg.xml and add ext.dic (new words) and stopword.dic (stop words).

1.6 Basic concepts

Index: a collection of documents of the same type, like a “table” in a database.
Document: the basic storage unit in ES, serialized to JSON.
Mapping: defines the types, analyzers, etc. of a document’s fields, like a “Schema” in a database.

Common Mapping field types:

Strings: text (tokenized), keyword (exact match, not tokenized).
Numbers: long, integer, short, byte, double, float.
Date: date.
Boolean: boolean.
Geo: geo_point (lat/lon), geo_shape (regions).
Object: object (nested fields).
Array: not defined specially; ES supports it automatically.

2. Index Operations (RESTful API)

All operations send JSON-formatted data over the HTTP API.

2.1 Create an index

PUT /index-name
{
  "settings": {
    "number_of_shards": 3,    // number of shards
    "number_of_replicas": 2   // number of replicas
  },
  "mappings": {
    "properties": {
      "field-name": {
        "type": "text",          // field type
        "analyzer": "ik_max_word" // analyzer
      }
    }
  }
}

2.2 Read an index

GET /index-name
GET /index-name/_mapping
GET /_cat/indices?v   # view all indexes

2.3 Update an index

Full document update: a PUT request with the full JSON data, equivalent to deleting the old document and creating a new one.
Partial document update: a POST request using the _update API.

POST /index-name/_update/document-ID
{
  "doc": {
    "field": "new value"
  }
}

Mapping update limitation: once a field type is created, most types can’t be changed (e.g. text to keyword), but you can add new fields. To change a type, you must rebuild the index.

2.4 Delete an index

DELETE /index-name

2.5 Bulk operations

Complete multiple document inserts/deletes/updates in a single request.

POST /_bulk
{"index":{"_index":"idx","_id":"1"}}
{"title":"Doc 1"}
{"delete":{"_index":"idx","_id":"2"}}
{"update":{"_index":"idx","_id":"3"}}
{"doc":{"title":"Updated"}}

Practices for designing a Mapping from a database table:

MySQL’s varchar → use text for full-text search, keyword for exact query/sorting.
Numeric types map to integer/long/float.
Use date uniformly for dates; a format can be specified.
Use geo_point for lat/lon queries.
Set index: false on fields that don’t need searching to save space.

3. Java REST Client

The official recommendation is the Java High Level REST Client (7.x) and the Elasticsearch Java API Client (recommended for 8.x+). Here we use the 7.x high-level client as an example.

Add the dependency:

<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-high-level-client</artifactId>
    <version>7.12.1</version>
</dependency>

Initialize the client:

RestHighLevelClient client = new RestHighLevelClient(
    RestClient.builder(new HttpHost("localhost", 9200, "http"))
);

3.1 Index operations

// Create an index
CreateIndexRequest request = new CreateIndexRequest("my_index");
request.settings(Settings.builder()
    .put("index.number_of_shards", 3)
    .put("index.number_of_replicas", 2));
request.mapping("{\"properties\":{...}}", XContentType.JSON);
CreateIndexResponse response = client.indices().create(request, RequestOptions.DEFAULT);

// Delete an index
DeleteIndexRequest deleteRequest = new DeleteIndexRequest("my_index");
client.indices().delete(deleteRequest, RequestOptions.DEFAULT);

// Existence check
GetIndexRequest getRequest = new GetIndexRequest("my_index");
boolean exists = client.indices().exists(getRequest, RequestOptions.DEFAULT);

3.2 Document operations

// Add a document
IndexRequest indexReq = new IndexRequest("my_index").id("1");
User user = new User("John Smith", 25);
indexReq.source(JSON.toJSONString(user), XContentType.JSON);
IndexResponse indexResp = client.index(indexReq, RequestOptions.DEFAULT);

// Partial update
UpdateRequest updateReq = new UpdateRequest("my_index", "1")
    .doc("age", 26);
client.update(updateReq, RequestOptions.DEFAULT);

// Bulk operations
BulkRequest bulk = new BulkRequest();
bulk.add(new IndexRequest("my_index").id("1").source(...));
bulk.add(new DeleteRequest("my_index", "2"));
client.bulk(bulk, RequestOptions.DEFAULT);

3.3 Query operations

Build the query conditions and execute the search.

SearchRequest searchRequest = new SearchRequest("my_index");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders.matchQuery("title", "elasticsearch"));
searchRequest.source(sourceBuilder);

SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
SearchHit[] hits = response.getHits().getHits();
for (SearchHit hit : hits) {
    String json = hit.getSourceAsString();
    User user = JSON.parseObject(json, User.class);
}

4. DSL Query Syntax

DSL (Domain Specific Language) builds queries based on JSON and is ES’s most powerful search method.

4.1 Query categories overview

Leaf queries: query specific values on specific fields.
- Full Text: matches after tokenization, with relevance scoring.
- Term-Level: no tokenization, direct exact matching of terms.
- Geo: coordinate queries.
Compound queries: combine multiple leaf or compound queries, modifying scoring, filtering, etc.
Special queries: e.g. script, exists.

4.2 Full-text queries

Apply to text fields; they tokenize and compute a relevance score _score.

match: fuzzy match, OR relationship after tokenization.

GET /index/_search
{
  "query": {
    "match": {
      "content": "Elasticsearch intro"
    }
  }
}

match_phrase: phrase match, requiring the tokens to be in the same order and contiguous.

{ "query": { "match_phrase": { "content": "Elasticsearch intro" } } }

multi_match: multi-field match.

{ "query": { "multi_match": { "query": "intro", "fields": ["title", "content"] } } }

4.3 Exact queries

Apply to keyword, numbers, dates, booleans; they don’t compute a relevance score.

term: exact-value match.

{ "query": { "term": { "status": "active" } } }

range: range query.

{
  "query": {
    "range": {
      "price": { "gte": 100, "lte": 500 }
    }
  }
}

terms: multi-value exact match.

{ "query": { "terms": { "category": ["Technology", "Education"] } } }

4.4 Geo queries

The field type must be geo_point.

// Rectangular range
{
  "query": {
    "geo_bounding_box": {
      "location": {
        "top_left": { "lat": 40, "lon": 116 },
        "bottom_right": { "lat": 39, "lon": 117 }
      }
    }
  }
}
// Distance range (center + radius)
{
  "query": {
    "geo_distance": {
      "distance": "10km",
      "location": { "lat": 39.9, "lon": 116.4 }
    }
  }
}

4.5 Compound Queries

bool query: combines multiple conditions, including must (AND), should (OR), must_not (NOT), and filter (filter, no scoring).

{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "phone" } }
      ],
      "filter": [
        { "range": { "price": { "gte": 1000, "lte": 3000 } } },
        { "term": { "brand": "Huawei" } }
      ],
      "must_not": [
        { "term": { "status": "delisted" } }
      ],
      "should": [
        { "term": { "feature": "5G" } },
        { "term": { "feature": "fast-charge" } }
      ],
      "minimum_should_match": 1
    }
  }
}

filter doesn’t participate in scoring and performs better; prefer it for filter conditions.

4.6 Sorting and pagination

Sorting:

{
  "query": { "match_all": {} },
  "sort": [
    { "price": "desc" },
    { "_score": "desc" }
  ]
}

Sorting on a text field requires enabling fielddata or using the keyword sub-field.

Pagination:

{
  "query": { "match_all": {} },
  "from": 0,  // starting offset
  "size": 10  // documents per page
}

The deep-pagination problem:

As from grows, ES has to fetch from+size documents from each shard and sort them at the coordinating node; memory and latency rise sharply.
Solutions:
- search_after: uses the sort value of the previous page’s last document to query the next page in real time, with no from overhead, but doesn’t support jumping to arbitrary pages.
- scroll: generates a data snapshot, good for traversing all data (e.g. exports), but not real-time.
- Limit pagination depth (e.g. max_result_window: 10000).

4.7 Highlighting

{
  "query": { "match": { "content": "elasticsearch" } },
  "highlight": {
    "fields": {
      "content": {
        "pre_tags": ["<em>"],
        "post_tags": ["</em>"]
      }
    }
  }
}

The response returns an extra highlight field containing the highlighted snippets.

4.8 Aggregations

Aggregations extract statistics, groupings, and other analytical info from data.

Aggregation categories:

Bucket aggregations: grouping, e.g. terms, range, date_histogram.
Metric aggregations: compute statistics, e.g. avg, sum, max, min, stats.
Pipeline aggregations: compute again on the results of other aggregations.

DSL aggregation example:

{
  "size": 0,  // don't return documents, only the aggregation result
  "aggs": {
    "brand_group": {
      "terms": { "field": "brand" },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } }
      }
    }
  }
}

Meaning: group by brand and compute each group’s average price.

Implementing aggregations with the Java client:

SearchRequest searchRequest = new SearchRequest("items");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders.matchAllQuery());

// Build the aggregation
TermsAggregationBuilder aggregation = AggregationBuilders
    .terms("brand_group")
    .field("brand.keyword")
    .subAggregation(AggregationBuilders.avg("avg_price").field("price"));
sourceBuilder.aggregation(aggregation);
searchRequest.source(sourceBuilder);

SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
Aggregations aggs = response.getAggregations();
Terms brandTerms = aggs.get("brand_group");
for (Terms.Bucket bucket : brandTerms.getBuckets()) {
    String brand = bucket.getKeyAsString();
    Avg avgPrice = bucket.getAggregations().get("avg_price");
    System.out.println(brand + ":" + avgPrice.getValue());
}

5. Supplement on ES Core Principles

5.1 Shards and replicas

Shard: an index is horizontally split into multiple shards, each an independent Lucene index.
- Advantages: distributed storage, parallel search, horizontal scaling of capacity and throughput.
Replica: a copy of each primary shard, providing high availability and sharing query load.
- The shard count can’t be changed after index creation (it can be adjusted via the _split/_shrink API).
- The replica count can be adjusted dynamically.

5.2 Write and search flows

Write: the document is routed to a primary shard; after the primary writes it, it forwards to the replicas, and confirmation returns only when all are done.
Search: the coordinating node forwards the request to the relevant shards (primary or replica); each shard returns results, and the coordinating node merges and sorts them.

5.3 Optimization tips

Avoid too many fields and deep aggregations.
Use filter instead of must to reduce scoring computation.
Design the Mapping sensibly; use keyword instead of text for exact search.
Separate hot and cold data, combined with Index Lifecycle Management (ILM).

1. Fundamentals#

1.1 Origin: Lucene#

1.2 The tech stack (ELK)#

1.3 Installing with Docker#

1.4 The Inverted Index#

1.5 The Analyzer#

1.6 Basic concepts#

2. Index Operations (RESTful API)#

2.1 Create an index#

2.2 Read an index#

2.3 Update an index#

2.4 Delete an index#

2.5 Bulk operations#

3. Java REST Client#

3.1 Index operations#

3.2 Document operations#

3.3 Query operations#

4. DSL Query Syntax#

4.1 Query categories overview#

4.2 Full-text queries#

4.3 Exact queries#

4.4 Geo queries#

4.5 Compound Queries#

4.6 Sorting and pagination#

4.7 Highlighting#

4.8 Aggregations#

5. Supplement on ES Core Principles#

5.1 Shards and replicas#

5.2 Write and search flows#

5.3 Optimization tips#