Missing messages after restart

I’m running Graylog 2.2.3. Last week I restarted both graylog and elasticsearch because something went wrong in elasticsearch and we had a huge and growing backlog of unprocessed messages.

Now we’ve noticed that on the graylog only very recent messages are shown. On the elasticsearch node I seem to now have two indices:

data/graylog/nodes/0 which has all the older indices like graylog_163 through graylog_246, and
data/graylog/nodes/1 which has the new messages in indices graylog_0 through graylog_9.

Is there any way to tell graylog to show messages from data/graylog/nodes/0 too? Or could I just stop elasticsearch, move the files from one dir to the other and restart it?

Reading the docs it seems like maybe I could create an elasticsearch alias but I don’t then see how to tell graylog how to use the alias.

Thanks for any help!

What did you do exactly and what’s in the logs of your Elasticsearch and Graylog nodes directly before and after you “repaired” them?

First I restarted graylog. It’s running in kubernetes and so it’s possible that a new instance of graylog was up an running before the old instance was terminated. I’m guessing that’s why graylog thought it should be a new node instead of the old node.

I also restarted elasticsearch.

Here’s a gist with graylog logs before/after it’s restart and the elasticsearch log from after it’s restart. Unfortunately I don’t have the es log from before I restarted it.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.