OTX Rate Limit causes Process Buffer then Journal to Fill

Graylog Central (peer support)
1

Dear All,

Perhaps some one can help me here.

I have some PA firewalls which I need to do threat look ups on using OTX. etc, there is roughly 20k events a minute.

Situation
I have observed in Graylog server log that when I get rate limited , error =429 the processor buffer starts to fill and then the journal.

The pipeline rule is correctly configured to not do look ups against RFC 1918 address,if I remove the threat hunting rule from the pipeline the situation resolves.

Attempts to Resolve.
My first thought was to increase the caches for the OTX so that it would not do look ups. I have increase size to 20,000 entries and set the cache time to 30 mins as shown below.

image
image961×365 100 KB

Does anyone here have any other tweeks / ideas that might resolve the issue?

Does anyone know the OTX rate limits?

Cheers

Jake Smith

2

he @Magneton

easiest solution - pay for OTX to do not run into the ratelimit.

3

Hi Jan,

Would an alternative be to increase cache further?

If I increase cache , it will just use more ram on Graylog processing node, so I would have to increase ram?

Cheers

Jake

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.