ALERT Event Group By with multiple fields and email backlog

Graylog Central (peer support)
1

Hi - using Graylog 4.0.3 I have Event Definition using a filter on a single stream (Prod Env Stream), aggregating on a single field (container_name). When filter returns result I am successfully receiving an email notification for each container_name with correct message backlog.

Input data is routed into 1 of 3 streams based upon namespace_name (production, test, development). Next, I modified Event Definition to use multiple streams (Prod Env Stream, Test Env Stream, and Dev Env Stream), aggregating on 2 fields (namespace_name, container_name. I was expecting to have an email generated for each namespace_name and container_name. Each email appears to be correct with ${event.message} reporting correct aggregation counts, but the message backlog includes all messages for all container_name and namespace_name.

Any ideas why this works correctly when aggregating by a single field, but not for multiple fields?

Not sure if it is relevant, but I have also defined Event Fields extracting namespace_name and container_name as keys I was hoping to use these in the email template (so far unsuccessful).

Thanks

2

@Latitude41S

Hello and Welcome,

I do not believe you can get two email notifications for each field/s you stated on one Event Definition you might have to split them up. You can attach multiple Email notification to one Event Definition maybe going to two different groups when the Event Definiftion is satisfied I can see that happening.

It’s hard to tell, could you show us in greater detail how you set up your environment?.

If those are Syslog fields then maybe something like this would work.

--- [Event Definition] ---------------------------
Title:       ${event_definition_title}
Description: ${event_definition_description}
Type:        ${event_definition_type}
--- [Event] --------------------------------------
Timestamp:            ${event.timestamp}
Message:              ${event.message}
Source:               ${event.source}
Key:                  ${event.key}
Priority:             ${event.priority}
Alert:                ${event.alert}
Timestamp Processing: ${event.timestamp}
Timerange Start:      ${event.timerange_start}
Timerange End:        ${event.timerange_end}
Fields:
${foreach event.fields field}  ${field.key}: ${field.value}
${end}
${if backlog}
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
${foreach backlog message}
Namespace: ${message.fields.namespace_name}
Container: ${message.fields.container_name }
${end}
${end}

Have you seen this documentation?

Graylog Event Fields

3

Yes - I read the docs thoroughly before posting :). I have simplified the initial scenario to highlight the problem now described below using Alert Definitions 1 and 2:

Streams
Prod Env Stream

  • namespace_name must match exactly production
  • Remove matches from ‘All messages’ stream

Test Env Stream

  • namespace_name must match exactly test
  • Remove matches from ‘All messages’ stream

Dev Env Stream

  • namespace_name must match exactly development
  • Remove matches from ‘All messages’ stream

Alert Definition 1
Filter

  • Search Query: _l:“Errror”
  • Streams: Prod Env Stream

Aggregation: container_name
Definition: if count(_l) > 0

Using vanilla email notification, the emails are generated with the correct grouping (i.e. one email per container name) and the message backlog for each email is for that container_name only.

This is expected behavior, as per the Aggregation example given on the Filter and Aggregation UI page:
Select Fields that Graylog should use to group Filter results when they have identical values. Example:
Assuming you created a Filter with all failed log-in attempts in your network, Graylog could alert you when there are more than 5 failed log-in attempts overall. Now, add username as Group by Field and Graylog will alert you for each username with more than 5 failed log-in attempts.

Alert Definition 2 (identical to above, with the only change is grouping by 2 fields):
Aggregation: namespace_name and container_name

Note - I realize there is no point grouping by namespace_name when the filter is for Prod Env Stream only, this just makes the problem really obvious.

The emails are generated with the correct grouping. In particular, the following event fields are correct and can be verified by executing a search:
Message: ${event.message}
Source: ${event.source}
Key: ${event.key}

However, the message backlog is no longer grouped according to aggregation fields, and instead returns all messages. In other words, an email generated for Namespace 1 + Container A has event fields relevant to Namespace 1 + Container A only, but the message backlog includes messages for Namespace 1 + Container A, Namespace 1 + Container B… Namespace 1 + Container n.

The documentation says the Backlog is The list of messages or events which lead to this alert being generated. So I’m wondering why the ${event. } properties do not match the message backlog?

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.