Graylog can’t resolve events. If your search window contain messages with that string it will send out a notification unless the sliding window does not contain this string anymore.
You might want to search every 5 minutes for the last five minutes and you will more granular results.
Thank you for your answer. I understand that Graylog can’t resolve events and I already thought about about reducing the time window. But:
If the host producing the log only sends 1 log with the error message and 5 minutes later it still has a problem, the query would not show the alert message and alas there would not be another alert.
Do not use logs for stateful alerting … you really want some kind of monitoring system for that, cause how should a log tool know if something is “good” or “bad” it just search in your logs and return data.
You are totally right. We do have stateful alerting in form of Icinga2. The problem is, though, that we want the alert regarding this one log entry to be acknowledgable + resolvable.
I guess we have to come with something up ourselves, then. Thanks nontheless!
if you already have icinga2 - why not create a stream for this single (possible) alert - and search in this stream with icinga. If you get a result for the search, use icinga to alert…
… or you use the event search api to search for events you want to alert on.
I’ve already written something like that in Python.
Problem is still: I can not resolve this. So if I search in the last 24 hours, even if I actually resolve the issue it will still come up as alert because one can find the log entry triggering the alert.
