My server is running in ESXi, the host lost communication with graylog over the weekend. After a bit of troubleshooting why I am not seeing any new logs I see I have graylog_deflector for my current shard. what I think i need to do is run the below but I am not sure how to do that, can someone advise on running the below so I can move graylog_deflector files to graylog_105. I am running graylog on Ubuntu 18.4 and I followed the graylog setup on this site.
right now search is not returning anything from the last few days. And I have an err message that says Deflector exists as an index and is not an alias. (triggered 16 hours ago)
The deflector is meant to be an alias but exists as an index. Multiple failures of infrastructure can lead to this. Your messages are still indexed but searches and all maintenance tasks will fail or produce incorrect results. It is strongly recommend that you act as soon as possible.
A few days ago the ESXi host graylog runs on went nuts and lost connection to several servers. I had to run fsck on /ubuntu–vg-root to get the OS to boot. I thought all was good but Monday I checked and search was not working.
Please enter content (application/x-www-form-urlencoded) to be POSTed:
And my index alias (i think thats what I am looking at) are graylog_### . The one that is failed says this graylog_104 Index does not contain any messages. (764.0B / 0 messages)
The bad index
graylog_deflector Time range of index is unknown, because index range is not available. Please recalculate index ranges manually. (12.8GB / 8,616,142 messages)
What is your elasticsearch cluster status?
If it’s red, don’t do anything on GL web intraface, it won’t work.
First fix elastic, and make the status green.
curl http://localhost:9200/_cat/indices returns several indicies. green open graylog_deflector yoPb2xQvRP-DmkTCTgwFIw has data, "green open graylog_104 WG9qG27cScCoCbpZjFmNNg " does not have data, can’t i somehow copy data from yoPb2xQvRP-DmkTCTgwFIw to WG9qG27cScCoCbpZjFmNNg and reindex? Or how do I alias yoPb2xQvRP-DmkTCTgwFIw to graylog_105. Graylog_105 is not a current index alias.