ERROR [Cluster] Couldn't read cluster health for indices [graylog_*]

yoyohand · September 12, 2018, 3:51pm

Hi everybody,

Sorry for my english
I try to add new elasticsearch node on my graylog, but I don’t work
I followed this procedure for add one or two elasticsearch server http://docs.graylog.org/en/2.4/pages/configuration/graylog_ctl.html#multi-vm-setup , but after restart my graylog server I have a error

In this file (/var/log/graylog/server/current), I have this error message:

2018-09-12_15:36:15.01776 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*]
2018-09-12_15:36:15.01821 
2018-09-12_15:36:15.01898 {"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null} (n/a)
2018-09-12_15:36:15.01973 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2018-09-12_15:37:15.07353 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2018-09-12_15:37:45.04232 INFO  [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2018-09-12_15:38:15.13366 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2018-09-12_15:38:45.08471 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2018-09-12_15:39:15.09761 ERROR [ConfigurationManagementPeriodical] Error while running migration <V20161130141500_DefaultStreamRecalcIndexRanges{2016-11-30T14:15:00Z}>
2018-09-12_15:39:15.09813 org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector
2018-09-12_15:39:15.09865 
2018-09-12_15:39:15.09907 {"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null}
2018-09-12_15:39:15.09972 	at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:95) ~[graylog.jar:?]
2018-09-12_15:39:15.10043 	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
2018-09-12_15:39:15.10100 	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
2018-09-12_15:39:15.10135 	at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:326) ~[graylog.jar:?]
2018-09-12_15:39:15.10196 	at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
2018-09-12_15:39:15.10230 	at org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges.upgrade(V20161130141500_DefaultStreamRecalcIndexRanges.java:108) ~[graylog.jar:?]
2018-09-12_15:39:15.10284 	at org.graylog2.periodical.ConfigurationManagementPeriodical.doRun(ConfigurationManagementPeriodical.java:43) [graylog.jar:?]
2018-09-12_15:39:15.10322 	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
2018-09-12_15:39:15.10373 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
2018-09-12_15:39:15.15100 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*]

Can you help me?
Thank’s you,

derPhlipsi · September 12, 2018, 4:17pm

Heyo @yoyohand,

I’ll give you two pointers that should provide you with a fix.
https://github.com/elastic/elasticsearch/issues/26488#issuecomment-326894161

If these two comments do not solve your issue, post your Elasticsearch logs. (Remember to use tripple-backticks to format log files properly.)

Greetings,
Philipp

yoyohand · September 12, 2018, 4:20pm

Hi DerPhlipsi,

Sorry for Backticks this is my first message on this community
Thank’s for your reply, I will try to solve my problem with your links, I’ll let you know

Greetings,

derPhlipsi · September 12, 2018, 4:21pm

No worries mate, it’s all good

I hope the links help

Greetings,
Philipp

yoyohand · September 13, 2018, 12:10pm

Hello,

I answer a little late, I work on several projects in parallel
I’m slowly moving on to the problem, i tried to apply the procedure in the links, now i get a new error message
My first VM is configured for run Graylog, web and MongoDB (no Elasticsearch). After reconfigure, I don’t have error message in elasticsearch log
My 2nd VM is configured for run only Elasticsearch. After reconfigure, I don’t have error message in elasticsearch log.
My last VM is configured for run only Elasticsearch. After reconfigure, I have error message

2018-09-13_11:57:30.68729 [INFO ][o.e.d.z.ZenDiscovery ] [6FrLEVT] failed to send join request to master [{6FrLEVT}{6FrLEVT5RN2l9XMrZ84BBw}{XGGTzKmBTo2KVjjLWiHTlg}{192.168.12.32}{192.168.12.32:9300}], reason [RemoteTransportException[[6FrLEVT][192.168.12.32:9300][internal:discovery/zen/join]]; nested: IllegalArgumentException[can’t add node {6FrLEVT}{6FrLEVT5RN2l9XMrZ84BBw}{H-3alPnmSbSBHjQSKlBBtQ}{192.168.12.33}{192.168.12.33:9300}, found existing node {6FrLEVT}{6FrLEVT5RN2l9XMrZ84BBw}{XGGTzKmBTo2KVjjLWiHTlg}{192.168.12.32}{192.168.12.32:9300} with the same id but is a different node instance]; ]

The error message is clear, I am looking for the solution on google at the same time I answer this topic
Hoping to quickly fix this problem, thank you

jan · September 13, 2018, 12:26pm

Did you clone your VM`s ?

If the above answered yes, shutdown elasticsearch, delete /var/lib/elasticsearch and start elasticsearch again. Everything should run now smoothly.

Note - all data that might be present (yet) is then lost.

yoyohand · September 13, 2018, 12:58pm

Yes I have cloned my VM
Is not problem for data, I am in the testing phase of the tool / deployment because I find it great

I delete /var/lib/elasticsearch on all my VMs?

I try to test this solution in the afternoon and I come back!

jan · September 13, 2018, 2:50pm

I delete /var/lib/elasticsearch on all my VMs?
on all VM`s running elasticsearch, yes

yoyohand · September 13, 2018, 3:24pm

I don’t have this folder /var/lib/elasticsearch

Maybe you talked about this folder

/var/opt/graylog/data/elasticsearch ?

jan · September 13, 2018, 3:26pm

How did you installed Graylog?

Did you run the OVA/Omnibus package?

Please do yourself a favour and make a clean installation.

yoyohand · September 13, 2018, 3:35pm

Yes I run OVA/Omnibus

I have configured my first VM and after I have cloned this VM (x2) for add nodes elasticsearch

After cloned my VM I use this article http://docs.graylog.org/en/2.4/pages/configuration/graylog_ctl.html#multi-vm-setup but my cluster with 1 Graylog server node and 2 another for Elasticsearch didn’t work.

Maybe the clone is not recommended and I will have to leave with 3 VM clean?

jan · September 13, 2018, 4:08pm

you should run graylog-ctl cleanse followed by graylog-ctl reconfigure

That might help you. Cloning the VM might give you some problemes - like you see.

yoyohand · September 13, 2018, 4:11pm

Ok, I will not be able to test this week now more time but I will test next week and I will keep you informed! Thanks a lot for your help,
Yoann

yoyohand · September 17, 2018, 9:00am

Hello,

After run graylog-ctl cleanse followed by graylog-ctl reconfigure and retry configure multi-vm, it’s work!
I think you were right, it is the fact of having cloned my VM without reset graylog that my problems have appeared! I can continue my tests, thanks for the help again

system · October 1, 2018, 9:00am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elastic connection issues according to the server.log Graylog Central (peer support)	5	778	November 5, 2020
Graylog is not searching Graylog Central (peer support)	13	1730	March 3, 2021
Graylog Could not retrieve Elasticsearch cluster health Graylog Central (peer support) elastic	2	1149	August 2, 2022
Multiple Errors Between Graylog & Elasticsearch Graylog Central (peer support)	6	1317	January 26, 2022
Post Upgrade (Graylog 3.0.2 / Elasticsearch 6.7.2) Index problems Graylog Central (peer support)	5	1126	June 3, 2019

ERROR [Cluster] Couldn't read cluster health for indices [graylog_*]

Related Topics