Graylog and index.refresh_interval

#1

Hello everybody,

Graylog version : 2.2.3
ES version : 2.4.2

I’m asking how do you configure the (elastic) index.refresh_interval in your graylog environment ?

I’ve read that it could be possible to do this by :

  • configuring elasticsearch.yml (but some has reported that this doesn’t work)
  • index setting (but it seems that it not applied on every new index)
  • updating graylog template (but I don’t want to break my graylog install ! ;-))

So what is the best/more effective way to achieve this ? I 'm looking for a solution where the setting is applied on every new index

Also, do you know any way to get actual index.refresh_interval setting ?
I tried with curl localhost:9200/graylog_*?pretty=true but no trace of index.refresh_interval

Thanks for your advice !

Process and output buffer is 100% utilized
(Jochen) #2

Create your own index template: http://docs.graylog.org/en/2.3/pages/configuration/elasticsearch.html#custom-index-mappings

#3

OK @jochen . So If my understanding is good, I would have to create the following graylog-custom-mapping.json :

{
  "template": "graylog_*",
  "settings": {
    "index.refresh_interval" : "30s"
  }
}

And then -> curl -X PUT -d @'graylog-custom-mapping.json' 'http://localhost:9200/_template/graylog-custom-mapping?pretty' ?

Thanks for your help !

(Jochen) #4

Close, but no cigar.

#5

Yes thank you @jochen it works ! I forced an index rotation and now I can see the refresh interval set to 30 sec:

curl localhost:9200/graylog_193?pretty=true

{
  "graylog_193" : {
    "aliases" : {
      "graylog_deflector" : { }
    },
    "mappings" : {
      "message" : {
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string"
            },
            "match" : "gl2_*"
          }
        }, {
          "store_generic" : {
            "mapping" : {
              "index" : "not_analyzed"
            },
            "match" : "*"
          }
        } ],
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          "AccessList" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "type" : {
            "type" : "string",
            "index" : "not_analyzed"
          },

[…]

          "user" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "refresh_interval" : "30s",
        "number_of_shards" : "3",
        "creation_date" : "1508149701125",
        "analysis" : {
          "analyzer" : {
            "analyzer_keyword" : {
              "filter" : "lowercase",
              "tokenizer" : "keyword"
            }
          }
        },
        "number_of_replicas" : "1",
        "uuid" : "Iav6nc8aTgq_EOiM6xrGcg",
        "version" : {
          "created" : "2040299"
        }
      }
    },
    "warmers" : { }
  }
}

Just to confirm, if I apply the template on graylog_*, this will only apply on actual and not future index automatically ?

(Jochen) #6

Yes, correct. Index templates are only applied when creating new indices.

This being said, make sure to include all index prefixes you have in other index sets and not just graylog_*, if you want to apply the settings to all newly created indices.

(system) closed #7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.