After the reboot step the webinterface took a long time to ‘reboot’ (about 10 minutes). When it was rebooting neither the search was responding and the system/index gave an immediate error Fetching global index stats failed: cannot GET http://10.63.1.217:9000/api/system/indices/index_sets/stats (500).
After digging in the log of the server component i saw repeated error message: Could not connect to 10.63.1.217:9200 which is the elasticsearch component.
Using netstat -l i dont see any process listening on 9200 or 9300 (the default ports used by elasticsearch).
Trying to start elasticsearch in /opt/graylog/elasticsearch gives me the following error: Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME.
The colleague who originally installed graylog and maintained it was unable to transfer knowledge/documenation before he left.
Well, it’s time to bone up on your Linux experience!
First order of business is to figure out where Java is installed, whether it’s still there and why Elastic seems to be unable to find it. Looks like something got broken either during the upgrade, or long before that.
Did you reboot the box once before actually starting the upgrade? I.E. did you do a sanity check of the box before you started changing it?
Also, do you guys have backups? In case you need to do a rollback.
Of course, the easiest thing to do is make sure that ES can find Java again.
Normally i would have added the java path to the PATH variable but i have no idea in where the jvm binaries are located in the OVA image.
It is installed somewhere else graylog (server) coudn’t start either.
Before upgraded we’ve rebooted multiple times to ‘fix’ messages not being processed.
As for backups, we have them, but the colleague who left maintained that aswell so i want to resort to that later.
The Graylog Virtual Machine Appliance was designed only as a showcase of Graylog and its cluster mode. This appliance is intended for proof of concept, testing, lab or other such applications. Please, deploy this appliance in a network that is isolated from the internet. In most cases, Graylog does not recommend using this appliance in a production environment.
Throw it away and deploy a new one. (maybe you can restore the data to the new OVA.)
To install a new standalone server on your favorite distro not more than 1 hour. After you can do the restore, what also could be a good time to make your graylog (disaster) recovery plan.
You need the mongodb’s database, and elasticsearch data. (without running ES, maybe you can use the elastic data dir)
