Lab 7.3: Index lifecycle management

Objective:

In this lab, you will create an index lifecycle policy and apply it to a data stream.

EXAM PREP: In Kibana, go to Stack Management -> Index Lifecycle Policies. Define a new policy that satisfies the following requirements:
- the name of the policy is my-metrics-policy
- the index is in the hot phase for 2 minutes
- when the index rolls over, it immediately moves to the warm phase, and the number of replicas is set to 1
- after 2 minutes in the warm phase, the index moves to the cold phase and the number of replicas is set to 0
- 5 minutes after rolling over, the index is deleted
Solution
- Click Create policy to start a new policy
- Name the policy my-metrics-policy
- In the Hot phase, expand the Advanced settings and set the rollover maximum age to 2 minutes. You will need to turn off Use recommended defaults and also delete the value for maximum size. Leave the remaining defaults.
- Activate the Warm phase and set the number of replicas to 1. Leave the Move data into phase when set to 0, and the rest of the values can remain the defaults.
- Activate the Cold phase and set the Move data into phase when to 2 minutes. Additionally, set the number of replicas to 0. Change the Keep data in this phase forever to Delete after this phase. The remaining defaults should be left alone.
- In the Delete phase change the Move data into phase when to 5 minutes.
- Click Save policy
The policy should look like the following:
```
PUT _ilm/policy/my-metrics-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "2m"
          },
          "set_priority": {
            "priority": 100
          }
        },
        "min_age": "0ms"
      },
      "warm": {
        "min_age": "0d",
        "actions": {
          "set_priority": {
            "priority": 50
          },
          "allocate": {
            "number_of_replicas": 1
          }
        }
      },
      "cold": {
        "min_age": "2m",
        "actions": {
          "set_priority": {
            "priority": 0
          },
          "allocate": {
            "number_of_replicas": 0
          }
        }
      },
      "delete": {
        "min_age": "5m",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
```
In Console, run the following command, which sets the poll interval for lifecycle policies to 30 seconds:
```
PUT _cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval": "30s"
  }
}
```
Note: this is far too low for production. The default of 10 minutes is generally acceptable.

Apply your new policy to the my-metrics-ds-template template.

Solution

In Index Management, open the my-metrics-ds-template template, click on Manage and select Edit.

At the Index settings, add the following:

{
  "index.lifecycle.name": "my-metrics-policy"
}

Review and save the template.

The solution for Console is:

PUT _index_template/my-metrics-ds-template
{
  "priority": 500,
  "template": {
    "settings": {
      "index.lifecycle.name": "my-metrics-policy"
    }
  },
  "data_stream": {},
  "index_patterns": [
    "my_metrics-*-*"
  ],
  "composed_of": [
    "time-series-mappings",
    "time-series-settings"
  ]
}

Back in Console, delete the my_metrics-service.status-dev data stream from the previous lab:
```
DELETE _data_stream/my_metrics-service.status-dev
```

Index the following document to re-create the data stream:

POST my_metrics-service.status-dev/_doc
{
  "@timestamp": "2021-07-04",
  "status": "UP",
  "message": "Service is running."
}

Display the settings of your data stream to check if you correctly set up ILM.
```
GET my_metrics-service.status-dev/_settings
```
You should see that the my-metrics-policy policy manages the backing indices. The first index should be in the hot phase.
Run the following command to see how the indices are progressing:
```
GET my_metrics-service.status-dev/_ilm/explain
```

Over the next 5 minutes, you should see indices being created, moving through the phases, and being deleted. Ensure your policy is working as expected.

Summary:

In this lab, you saw how to create an index lifecycle policy to manage the indices for time-series data.