Check_result empty, trying to get result of event filter

I’m trying to set up an alert for high average CPU utilization via Metricbeat logs.
I’m using a filter & aggregation as the event condition, which works as expected:
query: metricbeat_system_cpu_total_pct:>=0.50
group by: source
if avg() of metricbeat_system_cpu_total_pct >= 0.90

However, try as I might, I can’t find a way to actual report the CPU utilization that triggered the event in the email alert itself.
I’ve tried using a custom field, but since I can’t access the message itself this way i’m unable to get the total utilization as reported by the message. The only custom field I have that works is the hostname (${source.source})

I’ve also tried using check_result.triggeredcondition in the notification, but check_result appears to be completely empty in the resulting emails.

Is there some way to access the result of the aggregation in a notification?
I want the result of the aggregation (the average of the metricbeat fields) to appear in email notifications generated by this event. How do I do this?

The corresponding notification is below:
— [Event Definition] ---------------------------
Title: {event_definition_title} Description: {event_definition_description}
Type: {event_definition_type} --- [Event] -------------------------------------- Timestamp: {event.timestamp}
Source: {event.source} Priority: {event.priority}
Triggered condition: {check_result.triggeredCondition} Fields: {foreach event.fields field} {field.key}: {field.value}
{end} {if backlog}
— [Backlog] ------------------------------------
Last messages accounting for this alert:
{foreach backlog message} {end}

Check your Event Definition in Alerts - Event Definitions - tab Notifications, and setup parameter Message Backlog to 1 (by default it is not configured, so you could see any event details). This way, graylog can cycle through backlog, and show your cpu:

Use this snippet, or update:

{if backlog} {foreach backlog message}
Device {message.source} has high cpu utilization {message.fields.metricbeat_system_cpu_total_pct}
{end} {end}

Check also documentation, it’s described there:
https://docs.graylog.org/en/3.1/pages/streams/alerts.html#email-alert-notification

It’s funny you mention the documentation, because not only have I gone over that page of the docs several time, I later discovered that the documentation is out of date, and check_result is no longer a valid object at all: https://github.com/Graylog2/graylog2-server/issues/6203

I’m only slightly salty about it.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.