Alert if unprocessed message count is above a certain number

[lmm5247 (Logan M)]

I run Graylog 3.3 in Docker at home on a single server. Sometimes my disk usage gets high on the Docker host, and as such, Elasticsearch goes read-only.

When this happens, I’m able to see the number of unprocessed messages on the Nodes page (screenshot below).

I’m able to manually set my Elasticsearch cluster back to read-write, and then Graylog starts processing messages again.

Is there a search I can use to alert me when unprocessed messages crossed a specific threshold? For example: unprocessed_messages > 2000?

[ttsandrew]

Hello @lmm5247,

I don’t believe there’s anything you can use directly built in, but we do this by monitoring via the API. For uncommitted messages, the following would work:

curl -u api_username -H ‘Accept: application/json’ -X GET ‘https://graylogserver.yourdomain.suffix:9000/api/system/metrics/org.graylog2.journal.entries-uncommitted’

If the above doesn’t work you can try with http or using the --insecure switch.

Hope this helps.

[lmm5247 (Logan M)]

@ttsandrew

curl -u api_username -H ‘Accept: application/json’ -X GET ‘https://graylogserver.yourdomain.suffix:9000/api/system/metrics/org.graylog2.journal.entries-uncommitted’

Thanks, that’s what I need!