Long time-series searches broken since upgrading to 2.3.0


(Jason Haar) #1

Hi there

I’ve noticed that since doing a yum upgrade from the last 2.2 release to 2.3.0, doing searches that go back “some time” crash. I haven’t done in-depth analysis, but 7 days works, 14 days works, 30 days - nope. “All messages” nope.

eg if I’m looking at one of our Inputs, and click the “show received messages” button, that basically tries to show all messages ever received by that Input channel. If I do that on an Input channel I’ve just created, it crashes (even though it’s only got ~1 day of data). If I choose 7 days, it works. The crash is instantaneous too - it doesn’t look like it is doing much before it crashes. I ran a “tail -f” on the graylog/server.log and there was no output generated when this crash occurs. Same for ES

The crash looks like this in the browser

Could not execute search
There was an error executing your search. Please check your Graylog server logs for more information.

Error Message:
Unable to perform search query.
Details:
Search status code:
500
Search response:
cannot GET https://srv.name/api/search/universal/relative?query=*&range=2592000&limit=150&sort=timestamp%3Adesc (500)


(Jochen) #2

How many indices would querying “30 days” include?


(Scampuza) #3

This very same situation is happening for me… I havent found any solution yet,


#4

you could try setting http.max_initial_line_length to something large (I use 150kb) on all ES noddes, or wait the release of 2.3.1.


(Jason Haar) #5

How many indices does 30 days cover, hmm - how would I figure that out? But I created a new Input channel and pushed that data into a separate index set (ie it’s not the default). I have that set to 20 indices (18Gb) and it still crashes on a 30-day search. The default index set is currently at >7Tb, 830 indices, 10^9 documents - but I can’t tell you if that’s >30 days because it crashes :slight_smile:

Jason


(Jochen) #6

This sounds related to the following issue which was fixed in Graylog 2.3.1:
https://github.com/Graylog2/graylog2-server/issues/4054

Also “crash” might be the wrong word since Graylog keeps running, doesn’t it?


(Jason Haar) #7

FYI upgrading to graylog-2.3.1 fixed this. Thanks!


(system) #8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.