Graylog and ElasticSearch Troubleshooting

[JoeG]

When I see these log entries, how do I know to which ElasticSearch node I was connecting:
Jan 5 02:26:05 graylog1.foo.com graylog: 2019-01-05T02:26:05.525Z WARN [Messages] Failed to index message: index=<graylog_1244> id= error=<{“type”:“es_rejected_execution_exception”,“reason”:“rejected execution of org.elasticsearch.transport.TransportService$4@b3819dc on EsThreadPoolExecutor[bulk, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@433cde54[Running, pool size = 8, active threads = 8, queued tasks = 5369, completed tasks = 207715]]”}>
Jan 5 02:26:05 graylog1.foo.com graylog: 2019-01-05T02:26:05.525Z WARN [Messages] Failed to index message: index=<gl_windows_68> id= error=<{“type”:“es_rejected_execution_exception”,“reason”:“rejected execution of org.elasticsearch.transport.TransportService$4@409455ef on EsThreadPoolExecutor[bulk, queue capacity = 500, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@27b4718a[Running, pool size = 8, active threads = 8, queued tasks = 1564, completed tasks = 188895]]”}>

It appears that one of my nodes has a different value and it would be nice if the logs depicted which server to which it connected (unless it does already and I just don’t know enough about the products).

Also, should the Graylog key elasticsearch_hosts be configured to communicate with the ElasticSearch master nodes, data nodes, or all ES nodes?

[jan (Jan Doberstein)]

What ES Server Graylog connects to is not mentioned in the log message. It would be a feature request to have that included.

Personally I would recommend to have all ES hosts configured in Graylog. Just to avoid the situation that all configured are not reachable and ES looks down for Graylog.

[JoeG]

Thank you for clearing that up.