Graylog rewriting "index template" in Elasticsearch

Graylog Central (peer support)
8

I was unsuccessful on this issue, so I had to appeal for an “alternative solution” with some “wires” here and there:

  • I created a step in the “curator” that forces the migration of any index and shard with less than 1 day to the “hot” nodes.
  • And I created a shell script that applies the settings I need in the newest created index.

It is not a seamless solution, but it solves my problem. I will keep an eye out for new Graylog updates in the hope that this feature I need will appear in new versions.

Here’s my solution, hope it helps someone else with a similar problem:

“Wire” script:

cat wire.sh
#!/bin/bash

CURRENT_INDEX=$(curl -s -XGET "0.0.0.0:9200/_cat/indices?pretty" -H 'Content-Type: application/json' | grep 'graylog_' | awk '{print $3}' | sort -t _ -k 2 -rn | head -1)

curl -s --output /dev/null -XPUT "0.0.0.0:9200/${CURRENT_INDEX}/_settings?pretty" -H 'Content-Type: application/json' --data '
{
  "index.routing.allocation.include.data": "hot",
  "index.routing.allocation.include.box_type": "warm,cold",
  "index.routing.allocation.require.box_type": "warm,cold",
}'

“Wire” Cronjob (running via cron after a daily rotation which allocates new indexes randomly across my ES Cluster):

cat /etc/cron.d/wire
MAiLTO=""
SHELL=/bin/bash
1 1 * * *       root bash wire.sh

Elasticsearch Curator:

actions:
  1:
    action: allocation
    description: "Apply shard allocation filtering rules to newest indexes"
    options:
      key: box_type
      value: hot
      allocation_type: require
      wait_for_completion: True
      max_wait: 3600
      timeout_override:
      continue_if_exception: False
      disable_action: False
      allow_ilm_indices: True
    filters:
      - filtertype: pattern
        kind: prefix
        value: graylog_
      - filtertype: age
        source: creation_date
        direction: younger
        unit: days
        unit_count: 1