Graylog 4 elasticsearch autodiscovery

Graylog Central (peer support)
1

Hi, i am struggling to configure automatic discovery for elasticsearch nodes.
On the host there are two docker containers: one of graylog and the second of elasticsearch.
I configure the following in graylog.conf

elasticsearch_hosts = http://***:***@127.0.0.1:9200
elasticsearch_discovery_enabled = True
elasticsearch_discovery_default_user = ****
elasticsearch_discovery_default_password = ****
elasticsearch_discovery_default_scheme = http
elasticsearch_discovery_filter = es-datanode*
elasticsearch_discovery_frequency = 10s
elasticsearch_version = 6
elasticsearch_connect_timeout = 10s
elasticsearch_socket_timeout = 60s
elasticsearch_max_total_connections = 300

After restarting graylog container i observe the following in the logfile

2021-05-16T14:09:15.693Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=172.31.201.11:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 21]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1358426, setName='rs01', canonicalAddress=172.31.201.11:27017, hosts=[172.31.200.175:27017], passives=[172.31.201.11:27017, 172.31.201.137:27017], arbiters=[], primary='172.31.200.175:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, lastWriteDate=Sun May 16 14:09:15 UTC 2021, lastUpdateTimeNanos=2957503880646318}
2021-05-16T14:09:15.759Z INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:40454}] to 172.31.200.175:27017
2021-05-16T14:09:15.911Z INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <16384> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-05-16T14:09:16.057Z INFO  [ElasticsearchVersionProvider] Elasticsearch version set to 6.0.0 - disabling version probe.
2021-05-16T14:09:16.094Z INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2021-05-16T14:09:16.094Z INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2021-05-16T14:09:16.145Z INFO  [JestClientFactory] Using custom ObjectMapper instance
2021-05-16T14:09:16.146Z INFO  [JestClientFactory] **Node Discovery disabled..**.
2021-05-16T14:09:16.146Z INFO  [JestClientFactory] Idle connection reaping disabled...
2021-05-16T14:09:16.146Z INFO  [JestClientFactory] Authentication cache set for preemptive authentication

Meaning it does not use node autodiscovery defined by the filter, and using only local elasticsearch container to ingest logs/send search requests

Can you please point me where could be the problem ?

2

Hello,

I’m a little confused,

Then

So is there two Elasticsearch? One is local with Graylog and the other elasticsearch is in another container?
If this is correct make sure the correct ports are opened for Elasticsearch so they can communicate.

What do you get when executeing the following?

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

https://docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html#automatic-node-discovery
NOTE: Automatic node discovery does not work when using the Amazon Elasticsearch Service because Amazon blocks certain Elasticsearch API endpoints.

Hope that helps.

3

I see. Sorry for being unclear:
On each and every graylog node in the cluster, that runs as docker container on dedicated host, there is also coordinating only node of elasticsearch, also running as a container. Elasticsearch data nodes run on dedicated ec2 instances (overall 80 dedicated elasticsearch data nodes, 3 dedicated master nodes and several more coordinator nodes for kibana). Elasticsearch status is totally ok and everything is working fine: local coorinate node of elasticsearch sending index/search requests to elasticsearch.
The only issue that all traffic is going through this local elasticsearch node, and automatic discovery to distribute traffic between other coordinating nodes in cluster is not working: all requests flow through the local elasticsearch node

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.