Lab 7.2: Data streams

Objective:

In this lab, you will define an index template that creates a new data stream.

EXAM PREP: Start by creating a new index template that is a clone of the existing my-metrics-template but has the following settings:
- the name of the template is my-metrics-ds-template
- the index pattern is my_metrics-*-*
- creates a data stream
- all the other settings of my-metrics-template remain the same
Solution
- In Index Management click the Index Templates tab. Locate your my-metrics-template and use the three dots under Actions to select Clone
- Change the name of template to "my-metrics-ds-template"
- The index pattern should be the data type name, in this case my_metrics-*-*
- The Create data stream switch should be on
- No other changes are needed, click Next until you get to the Review page
- Click Create template
You can also complete this task by running the following command in Console:
```
PUT _index_template/my-metrics-ds-template
  {
    "priority": 500,
    "index_patterns": [
      "my_metrics-*-*"
    ],
    "data_stream": {},
    "composed_of": [
      "time-series-mappings",
      "time-series-settings"
    ]
  }
```

In Console, run the following command to create the data stream:

POST my_metrics-service.status-dev/_doc
{
  "@timestamp": "2021-07-04",
  "status": "UP",
  "message": "Service is running."
}

The response should be something like:

{
  "_index" : ".ds-my_metrics-service.status-dev-2021.07.04-000001",
  "_type" : "_doc",
  "_id" : "C9BS2XkBJ3Q40ZYI1zLs",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 3,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Let's change the data stream by adding a new field. Open the Index Management page and click Component Templates tab.
Edit the time-series-mappings by adding a field named data_stream.type, which is a constant_keyword. Save your changes to the template.
Solution
- In Index Management click the Component Templates tab. Locate your time-series-mappings and use the three dots under Actions to select Edit
- Skip to Mappings and Add field with a name of data_stream.type and type of Constant keyword.
- No other changes are needed, click Next until you get to the Review page
- Click Save component template
You can also solve this task by running the following command in Console:
```
PUT _component_template/time-series-mappings
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "text"
        },
        "status": {
          "type": "keyword"
        },
        "data_stream.type": {
          "type": "constant_keyword"
        }
      }
    }
  }
}
```
Return to Console and view the mappings for all the backing indices:
```
GET my_metrics-service.status-dev/_mapping
```
Notice that the mapping of the initial backup index has not been updated.

Manually roll over the data stream.

Solution

POST my_metrics-service.status-dev/_rollover/

Index the following document into the data stream:

POST my_metrics-service.status-dev/_doc
{
  "@timestamp": "2021-07-05",
  "status": "UP",
  "message": "Service is running.",
  "data_stream.type": "my_metrics"
}

This document will be indexed in a different index than the first document.

View the mappings for all the backing indices again:
```
GET my_metrics-service.status-dev/_mapping
```
Notice there is now a value for data_stream.type. This is the static value for any future documents indexed to this backing index. Future backing indices will have a blank value, which will also be set by the first document indexed to that backing index. This is not the behaviour we want, as all backing indices for this data stream should have the same value already set. Update the time-series-mappings component template once again to add my_metrics as the default value for the data_stream.type field.
Solution
- In Index Management click the Component Templates tab. Locate your time-series-mappings and use the three dots under Actions to select Edit
- Skip to Mappings and click the pencil icon next to data_stream.type.
- Toggle the Set value switch, enter a value of my_metrics and Update.
- No other changes are needed, click Next until you get to the Review page
- Click Create template
You can also solve this task by running the following command in Console:
```
PUT _component_template/time-series-mappings
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "text"
        },
        "status": {
          "type": "keyword"
        },
        "data_stream.type": {
          "type": "constant_keyword",
          "value": "my_metrics"
        }
      }
    }
  }
}
```

Manually roll over the data stream.

Solution

POST my_metrics-service.status-dev/_rollover/

Index the following document into the data stream:

POST my_metrics-service.status-dev/_doc
{
  "@timestamp": "2021-07-06",
  "status": "UP",
  "message": "Service is running."
}

Notice that there is no data_stream.type field.

Perform a simple search request on the data stream:
```
GET my_metrics-service.status-dev/_search
```
You should get three documents.

Search for the documents where data_stream.type is equal to my_metrics :

GET my_metrics-service.status-dev/_search
{
  "query": {
    "match": {
      "data_stream.type" : "my_metrics"
    }
  }
}

You should get two documents

OPTIONAL: Convert the my-metrics alias to a data stream.

Solution

The key is to run POST _data_stream/_migrate/my-metrics. However, this initially produces an error, stating:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "no matching index template found for data stream [my-metrics]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "no matching index template found for data stream [my-metrics]"
  },
  "status" : 400

To solve this, clone the my-metrics-template index template and set the index pattern to my-metrics. Ensure the new template sets the data stream flag and has a high priority (say, 900 or so) to avoid conflicts. An example solution might look like this:

PUT _index_template/my-metrics-template-copy
{
  "priority": 900,
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "my-metrics-policy"
        }
      }
    }
  },
  "index_patterns": ["my-metrics"],
  "data_stream": {},
  "composed_of": [
    "time-series-mappings",
    "time-series-settings"
  ]
}

Summary:

In this lab, you learned how to create and modify data streams.