1. Describe your incident:
I built a new graylog server and it worked. Moved it to a new network and assigned a different IP. Changed the mongo.conf, server.conf and datanode.conf to have the new IP address. Now I get the following error in the datanode log files:
Opensearch REST api of process 2740 unavailable. Cause: Host name 'current ip address’ does not match the certificate subject provided by the peer (CN=COMPUTERNAME)
I am no longer able to access the GUI and I assume this is because the services are not starting correctly.
2. Describe your environment:
As this is an initial setup, there is no production data yet.
OS Information: RHEL 9
Package Version: 6.1
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
I was trying to figure out how to reset Graylog back to the initial state. I am thinking the generated certificates that are created during the first GUI login are likely the problem. However my attempts to purge the mongodb and other similar endeavors failed.
I’ve read through these forums and found similar but not quite the same problem, none of which were able to help me. Possible I missed something here for sure as I am new to this product.
4. How can the community help?
If anyone has an idea as to how to reset the graylog server so I can get back to the initial config GUI, I feel like that might help. Or if there is a way via command line to reset the offending certificates that would be good also.
I am open to trying to generate a new certificate, but previously those were done via the web interface GUI, and I am not sure I can replicate everything on my own via command line.
Within the Graylog DB in mongo, I think you could get away with just dropping the data node collection. Dropping the the whole graylog DB would cause all configuration to be nuked.
The Data Node itself will also hang on to configuration and so deleting configuration under /var/lib/graylog-datanode/opensearch/config would also help get back to a preflight state to build new certs.
I dropped the graylog datanodes collection and deleted the config directory. This seems to have fixed the problem!!
Here are the steps I took to get this done. Your mileage may vary…
mongosh --host *ipaddress*
use graylog
show collections
db.datanodes.drop()
show collections
exit
cd /var/lib/graylog-datanode/opensearch
mv ./config/ ./archive/
mkdir config
chown graylog-datanode:graylog-datanode ./config -R
reboot
Probably could have done this without the reboot, if you wanted to restart services, I admit I took the easy way out.
Got my initial login info from /var/log/graylog-server.log and it worked.
Interestingly after going through the CA initialization again, I was able to log in with the administrator password I set originally (from the first setup about a week ago). So that is one thing that didn’t reset in this process.
Thanks for the help @Wine_Merchant , I appreciate it. I feel like I saw this same question worded about 3 different ways on these forums, with no real answers, so hopefully this will help the next person.
For any future reference - the critical and needed step is to remove the datanode/config/keystore.jks file. This keystore is holding the - now outdated - certificate. If you remove and restart the datanode service, it should be regenerated with a new, correct certificate and correct hostname.
