Relative searches with API

Hi,

I’m trying to retrieve a lot of logs from Graylog using an automated script in javascript.

Because of the huge ammount of logs, I need to make many requests with different offset as Graylog returns are limited.

Currently I am testing the script by requesting 1000 logs by 1000 logs

>  /api/search/universal/relative

with
range: 0
decorate:false
limit:1000
offset:0

After each request I set offset = offset + limit.

This works well until the offset is 10.000. Once offset=10000 → I get an error 500:

{“type”:“ApiError”,“message”:“Unable to perform search query”}

I read the error was caused by the max_result_window from elasticsearch which is by default set to 10.000.

However, I don’t understand why this limit impacts me because my searches results are limited to 1000 logs for each request !

I mean requesting the logs between 10.000 and 11.000 should return 1000 logs anyway so why would I be concerned the max_result_window limit ?

Does it comes from the way is working the endpoint: /api/search/universal/relative ?

If the only solution is to increase the max_result_window limit, what coudl be the consequences in terms of resources consuming ? This remediation would be also limited because increasing the limit to 50.000 would not allow me to request the 1000 logs between 50.000 and 51.000.

1 Like

Hello @weird-oecophylla

I did a test from the info you gave\ through GL’s API. I did not receive an error 500.

Example:

AND with limit.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.