1. Describe your incident:
Following single node installation instructions, I setup an input for syslog udp and it will run for various amounts of times (minutes, hours, even a couple days) and then it’s breaking…
When I log in to the server all indexes are gone, the input I setup is gone and I get Internal Server Errors when trying to look at the Alerts or Streams page. Other than those errors, I’m not seeing anything in server.log and datanode.log that line up with it breaking.
2. Describe your environment:
-
OS Information: Ubuntu 22.04 LTS on ESXI vm
-
Package Version: 6.2
-
Service logs, configurations, and environment variables: VM has 8 cpus, 64gb ram, 1TB storage.
3. What steps have you already taken to try and solve the problem?
In order to get it working again, I restart server service and it acts like a new install and takes me to the setup page for the datanode certs again. I also have to go into mongod and drop the index_field_types as it won’t start b/c of dup key errors.
Rinse/repeat. I’ve got over a week trying to get this stable/running. Twice I’ve started from a fresh VM. The second time the server ran for less than 20 minutes before bombing…
4. How can the community help?
Any help would be appreciated in figuring out what is breaking. Can provide logs, etc. I’m just not seeing anything glaring in them. Only errors occur after it breaks and I try to log in to GUI..