ERROR [Cluster] Couldn't read cluster health for indices [graylog_*]


(Yoann) #1

Hi everybody,

Sorry for my english :confused:
I try to add new elasticsearch node on my graylog, but I don’t work :frowning:
I followed this procedure for add one or two elasticsearch server http://docs.graylog.org/en/2.4/pages/configuration/graylog_ctl.html#multi-vm-setup , but after restart my graylog server I have a error

In this file (/var/log/graylog/server/current), I have this error message:

2018-09-12_15:36:15.01776 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*]
2018-09-12_15:36:15.01821 
2018-09-12_15:36:15.01898 {"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null} (n/a)
2018-09-12_15:36:15.01973 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2018-09-12_15:37:15.07353 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2018-09-12_15:37:45.04232 INFO  [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2018-09-12_15:38:15.13366 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2018-09-12_15:38:45.08471 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #1).
2018-09-12_15:39:15.09761 ERROR [ConfigurationManagementPeriodical] Error while running migration <V20161130141500_DefaultStreamRecalcIndexRanges{2016-11-30T14:15:00Z}>
2018-09-12_15:39:15.09813 org.graylog2.indexer.ElasticsearchException: Couldn't collect indices for alias graylog_deflector
2018-09-12_15:39:15.09865 
2018-09-12_15:39:15.09907 {"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null}
2018-09-12_15:39:15.09972 	at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:95) ~[graylog.jar:?]
2018-09-12_15:39:15.10043 	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
2018-09-12_15:39:15.10100 	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
2018-09-12_15:39:15.10135 	at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:326) ~[graylog.jar:?]
2018-09-12_15:39:15.10196 	at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
2018-09-12_15:39:15.10230 	at org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges.upgrade(V20161130141500_DefaultStreamRecalcIndexRanges.java:108) ~[graylog.jar:?]
2018-09-12_15:39:15.10284 	at org.graylog2.periodical.ConfigurationManagementPeriodical.doRun(ConfigurationManagementPeriodical.java:43) [graylog.jar:?]
2018-09-12_15:39:15.10322 	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
2018-09-12_15:39:15.10373 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
2018-09-12_15:39:15.15100 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*]

Can you help me?
Thank’s you,


(Philipp Ruland) #2

Heyo @yoyohand,

I’ll give you two pointers that should provide you with a fix.


If these two comments do not solve your issue, post your Elasticsearch logs. (Remember to use tripple-backticks to format log files properly.)

Greetings,
Philipp


(Yoann) #3

Hi DerPhlipsi,

Sorry for Backticks this is my first message on this community :slight_smile:
Thank’s for your reply, I will try to solve my problem with your links, I’ll let you know :wink:

Greetings,


(Philipp Ruland) #4

No worries mate, it’s all good :slight_smile:

I hope the links help :slight_smile:

Greetings,
Philipp


(Yoann) #5

Hello,

I answer a little late, I work on several projects in parallel :slight_smile:
I’m slowly moving on to the problem, i tried to apply the procedure in the links, now i get a new error message
My first VM is configured for run Graylog, web and MongoDB (no Elasticsearch). After reconfigure, I don’t have error message in elasticsearch log
My 2nd VM is configured for run only Elasticsearch. After reconfigure, I don’t have error message in elasticsearch log.
My last VM is configured for run only Elasticsearch. After reconfigure, I have error message

2018-09-13_11:57:30.68729 [INFO ][o.e.d.z.ZenDiscovery ] [6FrLEVT] failed to send join request to master [{6FrLEVT}{6FrLEVT5RN2l9XMrZ84BBw}{XGGTzKmBTo2KVjjLWiHTlg}{192.168.12.32}{192.168.12.32:9300}], reason [RemoteTransportException[[6FrLEVT][192.168.12.32:9300][internal:discovery/zen/join]]; nested: IllegalArgumentException[can’t add node {6FrLEVT}{6FrLEVT5RN2l9XMrZ84BBw}{H-3alPnmSbSBHjQSKlBBtQ}{192.168.12.33}{192.168.12.33:9300}, found existing node {6FrLEVT}{6FrLEVT5RN2l9XMrZ84BBw}{XGGTzKmBTo2KVjjLWiHTlg}{192.168.12.32}{192.168.12.32:9300} with the same id but is a different node instance]; ]

The error message is clear, I am looking for the solution on google at the same time I answer this topic
Hoping to quickly fix this problem, thank you


(Jan Doberstein) #6

Did you clone your VM`s ?

If the above answered yes, shutdown elasticsearch, delete /var/lib/elasticsearch and start elasticsearch again. Everything should run now smoothly.

Note - all data that might be present (yet) is then lost.


(Yoann) #7

Yes I have cloned my VM
Is not problem for data, I am in the testing phase of the tool / deployment because I find it great :slight_smile:

I delete /var/lib/elasticsearch on all my VMs?

I try to test this solution in the afternoon and I come back!


(Jan Doberstein) #8

I delete /var/lib/elasticsearch on all my VMs?
on all VM`s running elasticsearch, yes


(Yoann) #9

I don’t have this folder /var/lib/elasticsearch

Maybe you talked about this folder

/var/opt/graylog/data/elasticsearch ?


(Jan Doberstein) #10

How did you installed Graylog?

Did you run the OVA/Omnibus package?

Please do yourself a favour and make a clean installation.


(Yoann) #11

Yes I run OVA/Omnibus

I have configured my first VM and after I have cloned this VM (x2) for add nodes elasticsearch

After cloned my VM I use this article http://docs.graylog.org/en/2.4/pages/configuration/graylog_ctl.html#multi-vm-setup but my cluster with 1 Graylog server node and 2 another for Elasticsearch didn’t work.

Maybe the clone is not recommended and I will have to leave with 3 VM clean?


(Jan Doberstein) #12

you should run graylog-ctl cleanse followed by graylog-ctl reconfigure

That might help you. Cloning the VM might give you some problemes - like you see.


(Yoann) #13

Ok, I will not be able to test this week now more time but I will test next week and I will keep you informed! Thanks a lot for your help,
Yoann


(Yoann) #14

Hello,

After run graylog-ctl cleanse followed by graylog-ctl reconfigure and retry configure multi-vm, it’s work!
I think you were right, it is the fact of having cloned my VM without reset graylog that my problems have appeared! I can continue my tests, thanks for the help again :slight_smile:


(system) #15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.