We know that after dynamic mapping is started, the fields in an index may automatically increase as the number of fields in the imported document increases. In some cases, “mapping explosion” will occur, that is, the number of fields exceeds our tolerance range, and another disadvantage is that the import speed will slow down as the number of fields increases. This is because more fields need to be segmented. More fields segmented means more disk space. This is not allowed in many cases. In this case, we can create runtime Fields on the fly. The advantage of this approach is that we can define fields in the Mapping that we like, and we can set up Runtime fields for less important fields, although the speed at which they are searched is affected. To balance search effectiveness and flexibility, you usually search and filter index fields, such as timestamps. Elasticsearch automatically uses these index fields first when running the query, reducing response time. You can then use runtime Fields to limit the number of fields required by Elasticsearch to calculate its value. Using index fields with Runtime Fields gives you flexibility in querying the data you index and how you define other fields.

In today’s exercise, we will go through an example to show how to create Runtime Fields dynamically.

 

example

If dynamic Field Mapping is enabled when the Dynamic parameter is set to Runtime, the new fields are automatically added to the index mapping as Runtime fields. We use the following command to create an index template:

PUT _index_template/my_dynamic_index
{
  "index_patterns": [
    "my_dynamic_index-*"
  ],
  "template": {
    "mappings": {
      "dynamic": "runtime",
      "properties": {
        "timestamp": {
          "type": "date",
          "format": "yyyy-MM-dd"
        },
        "response_code": {
          "type": "integer"
        }
      }
    }
  }
}
Copy the code

Above we define an index template of index pattern named my_dynamic_index-*. Mapping that any index starting with my_dynamic_index- will have. Above we define two fields: timestamp and response_code. We’re also starting dynamic as runtime. This means that any document whose field name is not TIMESTAMP and response_code will be defined as Runtime fields. They are not segmented when importing documents, but are automatically used when searching by generating Runtime fields.

Next, create an index my_dynamic_index-1:

POST my_dynamic_index-1/_bulk
{"index":{}}
{"timestamp": "2021-01-01", "response_code": 200, "new_data": "data-1", "another_data": "another1"}
{"index":{}}
{"timestamp": "2021-01-01", "response_code": 200, "new_data": "data-1", "another_data": "another2"}
{"index":{}}
{"timestamp": "2021-01-01", "response_code": 200, "new_data": "data-2", "another_data": "another3"}
{"index":{}}
{"timestamp": "2021-01-01", "response_code": 200, "new_data": "data-2", "another_data": "another4"}
Copy the code

Above we see timestamp and response_codeo, but we also see new_data and another_data. Since these two fields are not defined in the Mapping and Dynamic is set to Runtime, these two fields will be set to Runtime Fields. After executing the above command, we can view the index mapping using the following command:

GET my_dynamic_index-1/_mapping
Copy the code

The command above shows:

{
  "my_dynamic_index-1" : {
    "mappings" : {
      "dynamic" : "runtime",
      "runtime" : {
        "another_data" : {
          "type" : "keyword"
        },
        "new_data" : {
          "type" : "keyword"
        }
      },
      "properties" : {
        "response_code" : {
          "type" : "integer"
        },
        "timestamp" : {
          "type" : "date",
          "format" : "yyyy-MM-dd"
        }
      }
    }
  }
}
Copy the code

From the above we can see that new_data and another_data are both set to Runtime fields, and their type is keyword.

We can query in the following way:

GET my_dynamic_index-1/_search
{
  "query": {
    "match": {
      "new_data": "data-2"
    }
  }
}
Copy the code

The response to the above command is:

{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : {" total ": {" value" : 2, the "base" : "eq"}, "max_score" : 1.0, "hits" : [{" _index ": "My_dynamic_index - 1", "_type" : "_doc", "_id" : "xVhmmXcBQeGXF - ccp1Km", "_score" : 1.0, "_source" : {" timestamp ": "2021-01-01", "response_code" : 200, "new_data" : "data-2", "another_data" : "another3" } }, { "_index" : "My_dynamic_index - 1", "_type" : "_doc", "_id" : "xlhmmXcBQeGXF - ccp1Km", "_score" : 1.0, "_source" : {" timestamp ": "2021-01-01", "response_code" : 200, "new_data" : "data-2", "another_data" : "another4" } } ] } }Copy the code

From the above results, I can see that there are two documents being searched. The presentation looks no different from our normal search, just like a normal field, but it must be noted that it is not as efficient as the fields defined in mapping for a large number of documents. If, in future use, we find that the new_data field needs to be split during import to improve efficiency, you can change the index template to:

PUT _index_template/my_dynamic_index
{
  "index_patterns": [
    "my_dynamic_index-*"
  ],
  "template": {
    "mappings": {
      "dynamic": "runtime",
      "properties": {
        "timestamp": {
          "type": "date",
          "format": "yyyy-MM-dd"
        },
        "response_code": {
          "type": "integer"
        },
        "new_data": {
          "type": "keyword"
        }
      }
    }
  }
}
Copy the code