Disregard /notalink/ in some of the links. I had to remove stuff interpreted as hyperlinks because I am a new user. 1. Describe your incident:
I am trying to test clustering with Graylog Open. I read in another post that basically you just need to install a node and point it to the same database. I ran the install for Graylog Server and Graylog Datanode and I provided the current database for the new node. The new node (node 2) shows up in the UI for my first node (node 1), but it is listed as Unavailable. Node 2 does not seem to have a UI that responds to requests. When I make a request to OpenSearch at node2:9200, I get a response that says “OpenSearch Security not initialized.”
I’ve verified that the services are running and I’ve also verified that I can reach the database from node 2.
How can I further troubleshoot this issue? Is there some specific documentation somewhere that I have overlooked?
Thank you!
2. Describe your environment:
OS Information: CentOS 9 Stream, Installed services via RPM process from the documentation.
Package Version: Version: 6.1.3+73526ba, codename Noir
JVM: PID 8742, Eclipse Adoptium 17.0.13 on Linux 5.14.0-533.el9.x86_64
Service logs, configurations, and environment variables:
Node 2 datanode.log file says this over and over
" Wait for cluster to be available …"
3. What steps have you already taken to try and solve the problem?
I’ve read documentation. I also restarted services and tried starting the node from the UI on Node 1. I also verified that database connectivity looked correct.
4. How can the community help?
Please point me in some direction. I am good with documentation or logs to check, etc., but I am not sure where else to look at this point.
I can connect to the DB on node1 from node2 with the mongosh utility. I haven’t tried to write to it with the utility but it seems to be working.
I see these errors in the server.log file on node1.
2024-12-04T14:35:15.711-06:00 ERROR [VersionProbe] Unable to retrieve version from indexer node maco9dotglgsb2.cr.example.com:9200: unknown error - an excep
tion occurred while deserializing error response: {}
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'OpenSearch': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false'
)
at [Source: (okio.Buffer$inputStream$1); line: 1, column: 12]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2572) ~[graylog.jar:?]
at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2598) ~[graylog.jar:?]
at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2606) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:765) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3659) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2747) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:867) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:753) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectReader._initForReading(ObjectReader.java:357) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2115) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1501) ~[graylog.jar:?]
at retrofit2.converter.jackson.JacksonResponseBodyConverter.convert(JacksonResponseBodyConverter.java:33) ~[graylog.jar:?]
at retrofit2.converter.jackson.JacksonResponseBodyConverter.convert(JacksonResponseBodyConverter.java:23) ~[graylog.jar:?]
at org.graylog2.storage.versionprobe.VersionProbe.lambda$probeSingleHost$3(VersionProbe.java:149) ~[graylog.jar:?]
at org.graylog2.storage.versionprobe.VersionProbe.rootResponse(VersionProbe.java:208) ~[graylog.jar:?]
at org.graylog2.storage.versionprobe.VersionProbe.probeSingleHost(VersionProbe.java:159) ~[graylog.jar:?]
at org.graylog2.storage.versionprobe.VersionProbe.lambda$probeAllHosts$2(VersionProbe.java:125) ~[graylog.jar:?]
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
I see these errors in the server.log on node 2.
2024-12-04T14:38:28.168-06:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: No route to host. - No route to host.
2024-12-04T14:38:28.168-06:00 INFO [VersionProbe] Indexer is not available. Retry #2165
2024-12-04T14:38:33.169-06:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: No route to host. - No route to host.
2024-12-04T14:38:33.169-06:00 INFO [VersionProbe] Indexer is not available. Retry #2166
2024-12-04T14:38:38.170-06:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: No route to host. - No route to host.
2024-12-04T14:38:38.170-06:00 INFO [VersionProbe] Indexer is not available. Retry #2167
2024-12-04T14:38:43.171-06:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: No route to host. - No route to host.
2024-12-04T14:38:43.171-06:00 INFO [VersionProbe] Indexer is not available. Retry #2168
2024-12-04T14:38:48.173-06:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: No route to host. - No route to host.
2024-12-04T14:38:48.173-06:00 INFO [VersionProbe] Indexer is not available. Retry #2169
2024-12-04T14:38:53.174-06:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: No route to host. - No route to host.
2024-12-04T14:38:53.174-06:00 INFO [VersionProbe] Indexer is not available. Retry #2170
2024-12-04T14:38:58.175-06:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: No route to host. - No route to host.
2024-12-04T14:38:58.175-06:00 INFO [VersionProbe] Indexer is not available. Retry #2171
2024-12-04T14:39:03.176-06:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: No route to host. - No route to host.
2024-12-04T14:39:03.176-06:00 INFO [VersionProbe] Indexer is not available. Retry #2172
2024-12-04T14:39:08.177-06:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: No route to host. - No route to host.
2024-12-04T14:39:08.177-06:00 INFO [VersionProbe] Indexer is not available. Retry #2173
Update:
I think I just found an error in my firewall config.
OK, progress. I had defined a zone for mongodb access and it was restricting traffic on other ports. That’s working now. I can get to the UI on node 2. I am still seeing an error in the datanode.log file on node 2 that reads
No route to host: maco9dotglgsb2.cr.example.com/10.12.24.140:9300
Update: This was also a firewall issue. Thanks for the help!
