Ran out of space, now new logs are not being processed

Graylog Central (peer support)
1

We ran out of diskspace on our Graylog server and ElasticSearch went into read-only mode. We moved the ElasticSearch indicies to a larger location, put the DB back into writeable mode, and restarted MongoDB. We thought everything was working well and can search old logs, but it appears that new logs are not being processed and we are now getting an error about garbage collection taking too long.

Not really sure where to go from here, as we don’t know the data flow, or how to check ElasticSearch is receiving the records. We do see Graylog telling us the messages/min for each of the collectors and can see the messages arriving on tcpdump, however.

2

He @snovak

one step after another. Did you restart Graylog after you have made Elasticsearch read-write again?

Can Graylog connect to Elasticsearch? Does System > Indices show you the indices of your setup? Did you perform index rotation - just to be sure?

3

Yes, restarted Graylog, the indices show up fine in the configuration. I’m not sure how to do a rotation of them, but it appears that it is set to 20 indices with delete as the rotation method.

4

you go into: System > Indices > INDEX NAME and choose “maintenance” and select on that pop-down rotate active write index

5

That seems to have fixed it. Thx.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.