Missing messages after restart

(Monty Zukowski) #1

I’m running Graylog 2.2.3. Last week I restarted both graylog and elasticsearch because something went wrong in elasticsearch and we had a huge and growing backlog of unprocessed messages.

Now we’ve noticed that on the graylog only very recent messages are shown. On the elasticsearch node I seem to now have two indices:

data/graylog/nodes/0 which has all the older indices like graylog_163 through graylog_246, and
data/graylog/nodes/1 which has the new messages in indices graylog_0 through graylog_9.

Is there any way to tell graylog to show messages from data/graylog/nodes/0 too? Or could I just stop elasticsearch, move the files from one dir to the other and restart it?

Reading the docs it seems like maybe I could create an elasticsearch alias but I don’t then see how to tell graylog how to use the alias.

Thanks for any help!

(Jochen) #2

What did you do exactly and what’s in the logs of your Elasticsearch and Graylog nodes directly before and after you “repaired” them?

(Monty Zukowski) #3

First I restarted graylog. It’s running in kubernetes and so it’s possible that a new instance of graylog was up an running before the old instance was terminated. I’m guessing that’s why graylog thought it should be a new node instead of the old node.

I also restarted elasticsearch.

Here’s a gist with graylog logs before/after it’s restart and the elasticsearch log from after it’s restart. Unfortunately I don’t have the es log from before I restarted it.

(system) closed #4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.