i have observed the ES GC problem shortly pause graylog output to it under a low load(1/3 of target load). in that time one ES node has high CPU usage while other two ES nodes has low CPU usages.
although i have set stale_master_timeout to 10000 ms, but system report cluster sync issue:
on es1:
[2018-04-12T23:20:10,404][WARN ][o.e.d.z.PublishClusterStateAction] [es1.mylogs.com] received a cluster state from a different master than the current one, rejecting (received {es3.mylogs.com}{PM-3Q9lUSn6mhvE9wWairQ}{TLpY_hzJSHmj43IWAE2T9g}{es3.mylogs.com}{192.168.1.23:9300}, current {es1.mylogs.com}{2mQ1AOofQxyxpkQW5zPK9g}{QAa0eZSwScaba_rkCg3fIg}{es1.mylogs.com}{192.168.1.21:9300})
[2018-04-12T23:20:14,020][WARN ][o.e.d.z.ZenDiscovery ] [es1.mylogs.com] discovered [{es3.mylogs.com}{PM-3Q9lUSn6mhvE9wWairQ}{TLpY_hzJSHmj43IWAE2T9g}{es3.mylogs.com}{192.168.1.23:9300}] which is also master but with an older cluster_state, telling [{es3.mylogs.com}{PM-3Q9lUSn6mhvE9wWairQ}{TLpY_hzJSHmj43IWAE2T9g}{es3.mylogs.com}{192.168.1.23:9300}] to rejoin the cluster ([node fd ping])
on es2:
[2018-04-12T23:29:36,307][WARN ][o.e.t.TransportService ] [es2.mylogs.com] Received response for a request that has timed out, sent [42086ms] ago, timed out [12086ms] ago, action [internal:discovery/zen/fd/master_ping], node [{es1.mylogs.com}{2mQ1AOofQxyxpkQW5zPK9g}{QAa0eZSwScaba_rkCg3fIg}{es1.mylogs.com}{192.168.1.21:9300}], id [216911]
on es3:
[2018-04-12T23:29:35,902][WARN ][o.e.t.TransportService ] [es3.mylogs.com] Received response for a request that has timed out, sent [41662ms] ago, timed out [11662ms] ago, action [internal:discovery/zen/fd/master_ping], node [{es1.mylogs.com}{2mQ1AOofQxyxpkQW5zPK9g}{QAa0eZSwScaba_rkCg3fIg}{es1.mylogs.com}{192.168.1.21:9300}], id [218597]
[2018-04-12T23:30:15,266][WARN ][o.e.t.TransportService ] [es3.mylogs.com] Received response for a request that has timed out, sent [36318ms] ago, timed out [6317ms] ago, action [internal:discovery/zen/fd/master_ping], node [{es1.mylogs.com}{2mQ1AOofQxyxpkQW5zPK9g}{QAa0eZSwScaba_rkCg3fIg}{es1.mylogs.com}{192.168.1.21:9300}], id [218668]
Currently i am using openJDK. does oracle JVM work better on GC than openJDK?