I’ve noticed that since doing a yum upgrade from the last 2.2 release to 2.3.0, doing searches that go back “some time” crash. I haven’t done in-depth analysis, but 7 days works, 14 days works, 30 days - nope. “All messages” nope.
eg if I’m looking at one of our Inputs, and click the “show received messages” button, that basically tries to show all messages ever received by that Input channel. If I do that on an Input channel I’ve just created, it crashes (even though it’s only got ~1 day of data). If I choose 7 days, it works. The crash is instantaneous too - it doesn’t look like it is doing much before it crashes. I ran a “tail -f” on the graylog/server.log and there was no output generated when this crash occurs. Same for ES
The crash looks like this in the browser
Could not execute search
There was an error executing your search. Please check your Graylog server logs for more information.
How many indices does 30 days cover, hmm - how would I figure that out? But I created a new Input channel and pushed that data into a separate index set (ie it’s not the default). I have that set to 20 indices (18Gb) and it still crashes on a 30-day search. The default index set is currently at >7Tb, 830 indices, 10^9 documents - but I can’t tell you if that’s >30 days because it crashes
