ES high GC overhead issue

Graylog Central (peer support)
8

Yes. i also search out this. but it seems increase the timer only keep node from removed from the cluster, the long delay in response should lead to cumulative backlog of requests among the cluster.

i have use nginx to loadbalance logs from gelf agents into each graylog nodes. when i start the input, they are evenly distributed into all 3 graylog nodes, and after serveral hours, i check the graylog nodes, i found nginx have stopped to send log to one of those 3 nodes and all inputs get to left 2 graylog nodes.

I also noted that ES node have high frequency(almost 1ms) warn log output:

[2018-04-11T17:06:31,639][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,647][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,647][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,648][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,648][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,649][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,649][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,649][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,650][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,650][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,650][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,651][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,651][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,651][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,652][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,652][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-11T17:06:31,653][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]

this should be the graylog default template’s dynamic template using legacy setting “not_analyzed” instead of “false”:

"dynamic_templates" : [
          { "internal_fields" : { "match" : "gl2_*", "mapping" : { "type" : "keyword" } } },
          { "store_generic"   : { "match" : "*",     "mapping" : { "index" : "not_analyzed" } } }
        ]