Back to normal alert


(Roman Toomarinson) #1

Hi All,

We are currently using Graylog v2.0.3 and I have defined the following alert condition:
Alert is triggered when there are more than 20 messages in the last 5 minutes. Grace period: 1 minute. Including last message in alert notification.

I am getting alerts when there are more than 20 messages counted in the last 5 minutes.
I am currently looking for some ‘back to normal’ alert so that in case there are less than 20 messages counted after 5 minutes I would expect to get an alert saying ‘number of messages counted in the last 5 minutes is for example 15 although the threshold is 20’ so I know it is all good again.

I will appreciate your assistance.

BR,
Roman


(Philipp Ruland) #2

Hey @Roman,

this is a topic that I’m about to open an Github issue about.
I’ll post the link to this here.

As a little workaround, you could define a second alert condition matching your “normal” values and make sure it is stateful. That will fire a notification if your values return to normal, but will always have an alert in the Alert tab.

Greetings - Phil


(Roman Toomarinson) #3

Hey @DerPhlipsi,

Thank you for your prompt response.
I appreciate your workaround but in this case I will receive an alert each 5 minutes saying that the status is normal.
We have an integration with Slack and currently looking for getting only 1 alert upon ‘back to normal’.
We know that we are good unless we get an alert saying that number of messages raised above 20 and just want to get 1 alert 5 minutes after the previous alert saying that everything is OK.
This expectation is to get an alert saying that we are back to normal (another option will be to wait for 5 minutes and see that no new alert was triggered).
This feature exist in Sensu and we got used to it before moving to Graylog.

BR,
Roman


(Philipp Ruland) #4

Oh, right, you are still on version 2.0.3.
I worked with the new stateful alerts introduced in the current version.


(Roman Toomarinson) #5

@DerPhlipsi

Does it work differently in the new version?


(Philipp Ruland) #6

Yes, see here for example:

Graylog got (the now optional) stateful alerts. This means, Graylog will only send one alert notification if an alert has been triggered and will then wait until the alert has been resolved plus the grace period for it to start checking the condition again.

Greetings - Phil


(Roman Toomarinson) #7

@DerPhlipsi - I will look for upgrading our version to 2.2.x yet I am not sure that it contains the feature I am looking for.


(Philipp Ruland) #8

Well, it is just a workaround that I use, it is not the optimal solution.

See the issue here:

Greetings - Phil