Graylog 3.1 Alert Notification using Event Definition with Aggregation

Hi,

We created an Event Definition using a Search Filter. We then set it up to create event based on Aggregation, using Grouping of a field (count of results for the specific field), which will trigger if 1 or more results in timeframe definition (within and interval) - we want to get only 1 notification per the value of Grouped field. Finally we set it to send notification via Email - used default email template.

For some reason this is not creating any events, and not sending any Email notification.
If we change the Event Definition from “Aggregation” to “Filter has Results”, it works fine - sends hundreds of emails, but works

Was wondering if you have a guide/example of such an Event Definition (using Filter not Stream), so we can see if perhaps we missed something.

Let me know what other info you need.

Thanks

he @mac300

you might want to share your exact configuration?

Here is the image for Event Condition.
No additional settings were added on the other screens, and notification was set as Email Notification with all default settings and email template.

Basically I have a Parent field (Level1), a Sub field (Level2) and details (Level 3).

On Parent field we currently can have 2 different values. Each of these can have about 10 Level2 values, and each of these Level2 can have hundreds of other values on Level3.

So, if filter has Result when searching every minute 1 minute back, we want to get 1 notification per Level1 & Level2.

Let me know if you need anything else…

Can you make sure that all your messages have fields for all levels?
What happens if you remove the Level3 field from the count() aggregation?

The filter I use brings back only specific Messages. No message = No alert.
I reference in Grouped fields and Aggregation only fields that exist.

As for removing the Count() aggregation, once it’s configured I cannot enter an empty field - I use “Message”, which is always available.
Even if I could, the purpose of this (as I understand it) is to limit amount of messages - so will send only when total messages in timeframe, for the Grouped Field, is equal or more than 1.

Appreciate the assistance do far :slight_smile:
Let me know what other info would be helpful…

Oh, that’s an UI bug. I’ll see this gets fixed.
In the meantime, you can work around this, by creating a new event definition.

Ok, so I updated the Event to not be aggregated, saved, and then edited again to be like it was before with the exception that no field is mentioned in the Aggregation condition.

No change.

Should I still try using a new Definition? Let me know…

1 thing I noticed, and not sure if relevant or how to change, is that the “Filter Preview” shows only 2 fields - Timestamp and Message. These are not the ones I use for Grouping. But when I use same filter on Graylog Search, I can see all fields. Might be what you asked before.

Any ideas?

Sorry for the delay,

If you’re running version 3.1.3, could you bump the loglevel to debug:


and look in your server.log for message like:

Skipping result <{}> because the conditions <{}> don't match

Maybe this will give us a clue

Thanks for the reply. :slight_smile:
I’ll get the Admin to set this up and check, and will let you know…

Where exactly do I search of that type of error?

Using parts of the string you copied, I searched in - Overview > Indexer failures > Show Errors - but did not find any.

Is this where I would have seen it?

If not, then where?
If yes, then would appreciate any other suggestions do you have

I meant the server.log in the filesystem.
Typically it should be /var/log/graylog-server/server.log
Depends on your setup:
https://docs.graylog.org/en/3.1/pages/faq.html#where-are-the-log-files-graylog-produces

1 Like

I see.
Ok, will ask our Admin to get us these logs.

In the meantime, is there any way you can perhaps create a test Alert on your end, similar to what I am trying to achieve? Perhaps you will have same issue as me, and then can find the cause. And if it does work ok, then you could share you you did so I can see what I did wrong :slight_smile:

Just a thought :smiley:

Ok, I finally managed to get it working. :smiley:

For some reason it works only when I add a Stream to the “Event Definition”, in the “Filter & Aggregation” section. That Stream is configured to - “Remove matches from ‘All messages’ stream”.

So, could be that when not using Stream in the Event, it looks for messages in the All Messages stream (default stream), and since what I need is not included there, it could not fire the Event.

Anyways, got it to work, so thanks for the assistance :slight_smile:

that is total reasonable @mac300

If you do not select a stream it will use the “default” search … so you should select the stream where your messages are in.

Yeah, well, Stream is only optional - I only found out about the “Remove matches from ‘All messages’ stream” option when I started to play around with Streams. Didn’t know that was the reason I was not getting the alerts…

At least now I know. :slight_smile:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.