We ran out of diskspace on our Graylog server and ElasticSearch went into read-only mode. We moved the ElasticSearch indicies to a larger location, put the DB back into writeable mode, and restarted MongoDB. We thought everything was working well and can search old logs, but it appears that new logs are not being processed and we are now getting an error about garbage collection taking too long.
Not really sure where to go from here, as we don’t know the data flow, or how to check ElasticSearch is receiving the records. We do see Graylog telling us the messages/min for each of the collectors and can see the messages arriving on tcpdump, however.