Hello,
I’m confused on the following statements.
I realize that the settings before the Upgrade was working, and this was a major version upgrade I think because you have two conflicting statements as shown above. Some settings and configuration have changed between versions.
I did get a chance to look over both log files, and I see where Graylog had to fix a couple things, I also see the Graylog Nodes have ID within the logs. I’m assuming there is nothing in the elasticsearch or MongoDb log files that would pertain to this? The only thing I can think of is the configurations.
Since there was an install of two major versions GL & ES (stated from the first post) and some setting have change, I’m wondering if this could be the problem/issue. So, I had to dig through my personal documentation on this to see what I did and compared to what you have.
I also noticed your using HTTPS/FQDN, so I have some suggestions for this.
Since there has been a lot of posts, I just want to sum it up.
You have three nodes
Node-01 GL, ES, Mongo (Master node)
Node-02 GL, ES, Mongo (Not Master node)
Node-03 ES, Mongo
- Some suggestion on your Graylog Configurations.
In the past I just copied one Graylog node configuration file over to the second Graylog node. On the second node node-02 adjusted the BIND IP Address and If there was TCP/TLS then adjusted the IP Address. That was it.
- Graylog Configuration file suggestions.
Make sure both Graylog are the same on node-01 & node-02 with a few exceptions.
Your HTTPS connection on Node -01
http_bind_address = node1.xx.xx:9000 <--Good
If your Using TLS/TCP you should have something like this.
http_publish_uri = https:// node1.xx.xx:9000/ <---Use HTTPS /w FQDN.
You might want to configure as shown above, if not Graylog will use you BIND address which is http:// .
The URL for this should be https://node1.xx.xx:9000 and if the certs are incorrect, you may have other issues.
These setting shown below are for Graylog, not only do you have to configure Elasticsearch YAML file but also must configure Graylog configuration file for all your Elasticsearch/MongoDb nodes.
elasticsearch_hosts = http://node1:9200 <--Yours
elasticsearch_hosts = http://10.200.6.95:9200, http://10.200.6.96:9200, http://10.200.6.97:9200 <-- Perhaps try this suggestion
mongodb_uri = mongodb://localhost/graylog,node2.xx.xx:27017/graylog?replicaSet=rs01 <--You Have this
mongodb_uri = mongodb://10.200.6.92:27017,10.200.6.93:27017,10.200.6.94:27017/graylog?replicaSet=replica01 <--Perhaps try this suggestion
Elasticsearch configuration Check/Suggestion
This setting discovery.zen.ping.unicast.hosts in ES is no longer in version 7.10. As shown below.
discovery.seed_hosts: ["10.0.1.101", "10.0.1.102", "10.0.1.103"] <--Your ES config file should look something like this.
This is a good read if you haven’t seen this already.
Notes that I have using FQDN in my Graylog environment. So, sum it up, in your /etc/host file this helps that your system knows who these other devices are.
Servers are Add to Local DNS/and have PTR Records, that is if you haven’t done it already. Since I see your using TCL/TLS/SSL the pointer records PTR (Also known as Reverse lookups) is a must.
This will prevent issues later when one node needs to contact another node specially when using Certificates.
Perhaps edit /etc/hosts file like this on all three nodes-01,02,03.
10.10.10.04 Node-001.domain.com
10.10.10.05 Node-002.domain.com
10.10.10.06 Node-003.domain.com
If you second node does show up on the Web UI, I have a feeling that Graylog doesn’t really know there is a second node, MongoDb does hold all the metadata.
What I’m looking for is something to tell us what’s going on, with nothing in the logs, kind hard to tell why Graylog is not displaying your second node, My apologies I haven had this issue yet so I’m just trying to troubleshoot the issue.
EDIT: I did some more research on this. I know some of these post may not be your version, but the issue is the same, pretty much what I explained above.
EDIT2: Out of curiosity what do you see what you execute this on you Graylog server?
curl -X GET 'http://ES-Host:9200/_cat/nodes?v'
Hope that helps