inanzzz | Building Elasticsearch queries and important considerations

Examples below are based on elasticsearch version 1.7 so depending on your version, you might or might not be able run certain queries. If your query fails, you can validate with query below which will tell you what's wrong with it.

curl -XGET '127.0.0.1:9200/product_test/_validate/query?explain=true&pretty=1' -d '
{
  your query goes here
}'

Queries and Filters

Elasticsearch has a single set of components called queries which can have endless combinations. Components can be used in "filtering" context and "query" context as explained below. For more information, visit Queries and Filters.

Filtering context queries are known as "non-scoring" or "filtering" query. The query simply asks the question: "Does this document match?".

Querying context queries are known as "scoring" query. The query simply asks the question: "Does this document match? if so, then how well does it match?". This concept of relevance is well suited to full-text search. A scoring query calculates how relevant each document is to the query and manipulate it's _score (relevance).

Performance

Filtering (non-scoring) queries: They just try to find if the document is a match or not so they tend to be faster. They can be cached in memory for faster access too.

Scoring queries: They check if the document is a match or not then calculate the relevance so they tend to be slower. Also, query results are not cacheable.

In general, a filtering (non-scoring) query will outperform a scoring query.

General rule: Use "scoring queries" for full-text or relevance dependant queries otherwise use "filtering (non-scoring) queries" for everything else.

Most Important Queries

There are many different queries for us to use but some of them used more often than others. List below includes the most important queries. For more information, visit Most Important Queries.

match_all

This query returns all documents.

# Give me all documents.
curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "match_all": {}
  }
}'

match

This is used 99% of times if you want a full-text or exact value search.

# Give me documents where "Product1" keyword found in "name" mapping.
curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "match": {
      "name": "Product1" # Full-text (analyzed) field
    }
  }
}'

# Give me documents where "id" mapping equals to "1".
curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "match": {
      "id": 1 # Exact-value (not_analyzed) field
    }
  }
}'

Tip: For exact-value searches, you should use filter instead of query because filter is cached.

multi_match

This is same as match but used to query more than one field. It behaves like OR condition.

# Give me documents as long as "Product1" keyword is found either in "name" or "shops.name" mappings.
curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "multi_match": {
      "query": "Product1",
      "fields": [
        "name",      # Full-text (analyzed) field
        "shops.name" # Exact-value (not_analyzed) field
      ]
    }
  }
}'

range

It is used for finding numbers or dates which satisfies the given condition. The conditions are:

gt: Greater than
gte: Greater than or equal to
lt: Less than
lte: Less than or equal to

# Give me documents where "id" mapping equals to "4/5/6".
curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "range": {
      "id": {
        "gte": 4,
        "lt": 7
      }
    }
  }
}'

term

This is used to search by exact values such as numbers, dates, booleans or not_analyzed exact-value string fields so this is not used for full-text searches.

# Give me documents where "shops.name" mapping equals to "Shop7".
curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "term": {
      "shops.name": "Shop7" # Exact-value (not_analyzed) field
    }
  }
}'

# Give me only searchable documents.
curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "term": {
      "is_searchable": true # Exact-value (not_analyzed) field
    }
  }
}'

terms

This is same as term but used to query more than one keywords in a single field. It behaves like OR condition.

# Give me documents as long as "Shop4" or "Shop7" keywords are found "shops.name" mapping.
curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "terms": {
      "shops.name": ["Shop4", "Shop7"] # Exact-value (not_analyzed) field
    }
  }
}'

# Give me documents where IDs equal to "4" or "7" in "id" mapping.
curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "terms": {
      "id": [4, 7] # Exact-value (not_analyzed) field
    }
  }
}'

missing

This is used to find documents if the "mapping" doesn't contain any data. It is same as field = '' or field IS NULL.

# Give me documents where "name" mapping has NULL or no data in it.
curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "filtered": {
      "filter": {
        "missing": {
          "field": "name"
        }
      }
    }
  }
}'

exists

This is used to find documents if the "mapping" does contain data. It is same as field <> '' or field IS NOT NULL.

# Give me documents where "name" mapping has data in it.
curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "filtered": {
      "filter": {
        "exists": {
          "field": "name"
        }
      }
    }
  }
}'

Combining queries

To combine multiple queries into a single query, you should use bool query which accepts the following parameters. For more information, visit Combining queries.

must: Clauses that must match for the document to be included.

must_not: Clauses that must not match for the document to be included.

should: If the clauses match, they increase the _score otherwise, they have no effect. Main aim is to manipulate document relevance so it is a "scoring" query.

filter: If the clauses match, document is included otherwise excluded. This has no affect on _score nor relevance so it is a "non-scoring" query.

Example query below must satisfy 2 conditions in order to include or exclude documents. On top of that there are 2 rules which will value certain documents more than the others. Read the comments below.

curl -XPOST '127.0.0.1:9200/product_test/_search?pretty' -d '
{
  "query": {
    "bool": {
      "must": {
        "match": {
          "name": "Product"  # Make sure document "name" contains "Product" keyword. Include.
        }
      },
      "must_not": {
        "match": {
          "name": "Product1" # Make sure document "name" does not contain "Product1" keyword. Exclude.
        }
      },
      "should": [
        {
          "match": {
            "id": 7          # If document "id" equals to "1" then score it higher so it appears higher. Scoring.
          }
        },
        {
          "range": {
            "year": {
              "gt": "2014"   # If document "year" greater than "2014" then score them higher so they appear higher. Scoring.
            }
          }
        }
      ]
    }
  }
}'

Note: If there are no must clauses, at least one should clause has to match. However, if there is at least one must clause, no should clauses are required to match.

Adding a filtering query

Opposed to logic above, if you don't want to increase the value of documents based on their "year" then you will just filter them instead. By moving range into filter clause, it becomes "non-scoring" query so as a result performance increases too.