Hi,
I’m trying to retrieve a lot of logs from Graylog using an automated script in javascript.
Because of the huge ammount of logs, I need to make many requests with different offset as Graylog returns are limited.
Currently I am testing the script by requesting 1000 logs by 1000 logs
> /api/search/universal/relative
with
range: 0
decorate:false
limit:1000
offset:0
After each request I set offset = offset + limit.
This works well until the offset is 10.000. Once offset=10000 → I get an error 500:
{“type”:“ApiError”,“message”:“Unable to perform search query”}
I read the error was caused by the max_result_window from elasticsearch which is by default set to 10.000.
However, I don’t understand why this limit impacts me because my searches results are limited to 1000 logs for each request !
I mean requesting the logs between 10.000 and 11.000 should return 1000 logs anyway so why would I be concerned the max_result_window limit ?
Does it comes from the way is working the endpoint: /api/search/universal/relative ?
If the only solution is to increase the max_result_window limit, what coudl be the consequences in terms of resources consuming ? This remediation would be also limited because increasing the limit to 50.000 would not allow me to request the 1000 logs between 50.000 and 51.000.
