Graylog rewriting "index template" in Elasticsearch

Graylog Central (peer support)
1

Hi Everyone!

I’m new using Graylog, as well as use Elastichsearch in cluster and MongoDB, it’s been a long learning curve to me but I’m enjoying everything.

My environment is:
Graylog 4, 2 nodes running in cluster
Elasticsearch 7, 6 nodes (2 hot, 2 warm, 2 cold)
MongoDB (4.4.4) running in Graylogs master node
elasticsearch-curator (5.8.3) running in the Elastichsearch master nodes

I was trying to configure Elasticsearch ILM in my end, but I saw here (improved elasticsearch index management (ILM) · Issue #9828 · Graylog2/graylog2-server · GitHub) that it can’t be done.

So I followed the recomendantion to use “Elasticsearch Curator” to do the dirty job and set configuration in Graylog Web Interface to rotate index based on time, rotating everyday.

This is working as expected, except for one step.

To make curator migrate data between hot, warm and cold nodes I had to add some custom setting in my index, as follow:

"routing": {
  "allocation": {
    "include": {
      "_tier_preference": "data_content",
      "data": "hot",
      "box_type": "warm,cold"
    },
    "require": {
      "box_type": "warm,cold"
    }
  }
}

"index.routing.allocation.include.data": "hot" # To force new indexes to be created in Hot Nodes
"index.routing.allocation.include.box_type": "warm,cold" # To be used by curator to know where to alocate data.
"index.routing.allocation.require.box_type": "warm,cold" # To be used by curator to know where to alocate data.

Finally, my problem is:

To make this “curator step” working I need to apply this custom setting to every new index created automaticly by Graylog.

But I wasn’t able to persist this settings in the “index template”.

Thing I tried to do was:

  • Add this settings to “graylog-internal”
  • Create a new template only with this settings making match in the same index pattern “graylog_*” to make both settings to merge applying my custom setting
  • Clone “graylog-internal” into a new template, add my custom settings in this clone template and delete “graylog-internal”

But everytime Graylog rotates the current index and create a new one, all I get is the template (whatever it is) been rewited by graylog returning to the old settings:

     "index": {
       "analysis": {
         "analyzer": {
           "analyzer_keyword": {
             "filter": "lowercase",
             "tokenizer": "keyword"
           }
         }
       }
     }
   }

And then a new index created based in those old settings.

Is there a way to append my custom settings somewhere in Graylog to force it to create a new index the right way and avoid it to rewrite my template?

Sorry about my verbose post, I just followed the first rule “Supply as much information as possible” listed here: Questions and You: A guide to getting an answer

1 Like
2

@reimlima
If I understand this correct, you are trying to rotate you indices ( i.e. Default Index, graylog-internal, etc…)? If so you can do this on the front end, System/Indices and select Edit on your Index you want.

image
image1289×445 22.4 KB

Upper Right hand corner to can manually rotate your indices.

image

I havent used Elasticsearch ILM,nor Elasticsearch Curator but sound interesting thou.
Hope this helps.

3

Hi @gsmith , thanks for your reply!

No man, sorry, you got it wrong.

Long story short what I’m trying to do is add some custom settings in “graylog-internal” index template in order to make it attend my needs.

But everytime I tried to do that, Graylog seems to ignore it and roll it back to the default setting.

I’m looking for a way to persist those settings somewhere to make Graylog take it into account everytime a new index be created.

4

@reimlima
My apologies, Unfortunately I have not modified templates yet. I have seen some post/s in here where people have done something simular with index templates. Maybe someone here can jump in.
Sorry I cant be more help.

5

@gsmith Not a Problem at all,

Maybe i have to apologize for not make myself clear at first time.

But thanks anyway :+1:

6

Reading through this the second time… it looks like you tried a Custom Index Mapping but it’s not entirely clear… This is the way you would enforce a field type, I am not sure if it would work with he extra settings. Elastic has it’s own work with templates. I have only worked with Graylog’s custom Index Mapping docs, not the Elastic Template API…

7

@tmacgbay thank you for you reply.

Yes, I’m trying to apply the same logic from “Custom Index Mapping” but in settings instead.

At first I was focusing in change this in Elasticsearch without success, now I’m trying to find a way to make it via Graylog, changing things somewhere, maybe in MongoDB or I don’t know.

My tought was create a template with this additional settings that matches the same index pattern as the “graylog-internal” hoping both merged like it says in “Custom Index Mapping” but it didn’t work.

8

I was unsuccessful on this issue, so I had to appeal for an “alternative solution” with some “wires” here and there:

  • I created a step in the “curator” that forces the migration of any index and shard with less than 1 day to the “hot” nodes.
  • And I created a shell script that applies the settings I need in the newest created index.

It is not a seamless solution, but it solves my problem. I will keep an eye out for new Graylog updates in the hope that this feature I need will appear in new versions.

Here’s my solution, hope it helps someone else with a similar problem:

“Wire” script:

cat wire.sh
#!/bin/bash

CURRENT_INDEX=$(curl -s -XGET "0.0.0.0:9200/_cat/indices?pretty" -H 'Content-Type: application/json' | grep 'graylog_' | awk '{print $3}' | sort -t _ -k 2 -rn | head -1)

curl -s --output /dev/null -XPUT "0.0.0.0:9200/${CURRENT_INDEX}/_settings?pretty" -H 'Content-Type: application/json' --data '
{
  "index.routing.allocation.include.data": "hot",
  "index.routing.allocation.include.box_type": "warm,cold",
  "index.routing.allocation.require.box_type": "warm,cold",
}'

“Wire” Cronjob (running via cron after a daily rotation which allocates new indexes randomly across my ES Cluster):

cat /etc/cron.d/wire
MAiLTO=""
SHELL=/bin/bash
1 1 * * *       root bash wire.sh

Elasticsearch Curator:

actions:
  1:
    action: allocation
    description: "Apply shard allocation filtering rules to newest indexes"
    options:
      key: box_type
      value: hot
      allocation_type: require
      wait_for_completion: True
      max_wait: 3600
      timeout_override:
      continue_if_exception: False
      disable_action: False
      allow_ilm_indices: True
    filters:
      - filtertype: pattern
        kind: prefix
        value: graylog_
      - filtertype: age
        source: creation_date
        direction: younger
        unit: days
        unit_count: 1
9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.