[SOLVED] message.cluster_name is ignored for notifications

[m_mlk]

1. Describe your incident:

Notifications does not seem to be able to parse anything ${message.*}-related except for ${message.message}, so other variables such as ${message.cluster_name} or ${message.namespace_name} are completely ignored, thus making our received emails with aggregated messages a bit useless…

2. Describe your environment:

• OS Information: Ubuntu 20.04 LTS

• Package Version: GL 4.3.9 + OpenSearch 1.3.3

• Service logs, configurations, and environment variables:

A sample message where you can see how container_name is populated:

How the event is defined:

Copied & pasted from ALERT Event Group By with multiple fields and email backlog - #2 by gsmith

--- [Event Definition] ---------------------------
Title:       ${event_definition_title}
Description: ${event_definition_description}
Type:        ${event_definition_type}
--- [Event] --------------------------------------
Timestamp:            ${event.timestamp}
Message:              ${event.message}
Source:               ${event.source}
Key:                  ${event.key}
Priority:             ${event.priority}
Alert:                ${event.alert}
Timestamp Processing: ${event.timestamp}
Timerange Start:      ${event.timerange_start}
Timerange End:        ${event.timerange_end}
Fields:
${foreach event.fields field}  ${field.key}: ${field.value}
${end}
${if backlog}
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
${foreach backlog message}
Namespace: ${message.namespace_name}
Container: ${message.container_name}
Message: ${message.message}
${end}
${end}

and the HTML counterpart:

<table width="100%" border="0" cellpadding="10" cellspacing="0" style="background-color:#f9f9f9;border:none;line-height:1.2"><tbody>
<tr style="line-height:1.5"><th colspan="2" style="background-color:#e6e6e6">Event Definition</th></tr>
<tr><td width="200px">Title</td><td>${event_definition_title}</td></tr>
<tr><td>Description</td><td>${event_definition_description}</td></tr>
<tr><td>Type</td><td>${event_definition_type}</td></tr>
</tbody></table>
<br /><table width="100%" border="0" cellpadding="10" cellspacing="0" style="background-color:#f9f9f9;border:none;line-height:1.2"><tbody>
<tr><th colspan="2" style="background-color:#e6e6e6;line-height:1.5">Event</th></tr>
<tr><td width="200px">Timestamp</td><td>${event.timestamp}</td></tr>
<tr><td>Message</td><td>${event.message}</td></tr>
<tr><td>Source</td><td>${event.source}</td></tr>
<tr><td>Key</td><td>${event.key}</td></tr>
<tr><td>Priority</td><td>${event.priority}</td></tr>
<tr><td>Alert</td><td>${event.alert}</td></tr>
<tr><td>Timestamp Processing</td><td>${event.timestamp}</td></tr>
<tr><td>Timerange Start</td><td>${event.timerange_start}</td></tr>
<tr><td>Timerange End</td><td>${event.timerange_end}</td></tr>
<tr><td>Source Streams</td><td>${event.source_streams}</td></tr>
<tr><td>Fields</td><td><ul style="list-style-type:square;">${foreach event.fields field}<li>${field.key}:${field.value}</li>${end}<ul></td></tr>
</tbody></table>
${if backlog}
<br /><table width="100%" border="0" cellpadding="10" cellspacing="0" style="background-color:#f9f9f9;border:none;line-height:1.2"><tbody>
<tr><th style="background-color:#e6e6e6;line-height:1.5">Backlog (Last messages accounting for this alert)</th></tr>
${foreach backlog message}
<tr>
<td>Namespace: ${message.namespace_name}<td>
</tr>
<tr>
<td>Container: ${message.container_name}<td>
</tr>
<tr>
<td>Message: ${message.message}<td>
</tr>
${end}
</tbody></table>
${end}

I’d be more than happy if this would work:

--- [Event] ---------------------------
Title:       ${event_definition_title}
Description: ${event_definition_description}
Type:        ${event_definition_type}
Timestamp:            ${event.timestamp}
${if backlog}
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
${foreach backlog message}
${message.container_name} ${message.message}
${end}
${end}

with this HTML template:

<table width="100%" border="0" cellpadding="10" cellspacing="0" style="background-color:#f9f9f9;border:none;line-height:1.2"><tbody>
<tr style="line-height:1.5"><th colspan="2" style="background-color:#e6e6e6">Event Definition</th></tr>
<tr><td width="200px">Title</td><td>${event_definition_title}</td></tr>
<tr><td>Description</td><td>${event_definition_description}</td></tr>
<tr><td>Type</td><td>${event_definition_type}</td></tr>
</tbody></table>
<br /><table width="100%" border="0" cellpadding="10" cellspacing="0" style="background-color:#f9f9f9;border:none;line-height:1.2"><tbody>
<tr><th colspan="2" style="background-color:#e6e6e6;line-height:1.5">Event</th></tr>
<tr><td width="200px">Timestamp</td><td>${event.timestamp}</td></tr>
<tr><td>Message</td><td>${event.message}</td></tr>
<tr><td>Source</td><td>${event.source}</td></tr>
<tr><td>Key</td><td>${event.key}</td></tr>
<tr><td>Priority</td><td>${event.priority}</td></tr>
<tr><td>Alert</td><td>${event.alert}</td></tr>
<tr><td>Timestamp Processing</td><td>${event.timestamp}</td></tr>
<tr><td>Timerange Start</td><td>${event.timerange_start}</td></tr>
<tr><td>Timerange End</td><td>${event.timerange_end}</td></tr>
<tr><td>Source Streams</td><td>${event.source_streams}</td></tr>
<tr><td>Fields</td><td><ul style="list-style-type:square;">${foreach event.fields field}<li>${field.key}:${field.value}</li>${end}<ul></td></tr>
</tbody></table>
${if backlog}
<br /><table width="100%" border="0" cellpadding="10" cellspacing="0" style="background-color:#f9f9f9;border:none;line-height:1.2"><tbody>
<tr><th style="background-color:#e6e6e6;line-height:1.5">Backlog (Last messages accounting for this alert)</th></tr>
${foreach backlog message}
<tr><td>${message.container_name} ${message.message}<td></tr>
${end}
</tbody></table>
${end}

This yields:

As you can see, the Namespace and Container are empty.

3. What steps have you already taken to try and solve the problem?

Read the docs for alerting: Alerting by Example - Alerts and Notifications
Searched in the forums and found the link shown above, among others.

I have also tried with ${kubernetes.container_name} or simply ${container_name} or even ${message.kubernetes.container_name} but no cigar…

4. How can the community help?

I assume that the filter shown in the screenshot above must not include the container_name in any way…

Is there any way to make it work?

TIA!

[tmacgbay]

I found it wonky to work with too, try ${message.fields.<fieldname>} I wonder if anyone has done a writeup or video on the how and why building alert message code ids more difficult than it needs to be … or at least seems that way.

[m_mlk]

Hi @tmacgbay

${messages.fields.container_name} worked. Thanks so much!

Best,

[gsmith]

@tmacgbay

${message.fields.}.

Oh boy, I need fix that post. Looks like my copy & paste malfunctioned.

[gsmith]

hey @m_mlk

Sorry about that post you copied from, it was incorrect.

Also have you seen these?

• Adding Stream URL to Notification Template
• ALERT template - HTML

[m_mlk]

hi @gsmith

no worries mate. I figured it out and the alerts work now as expected.

Nope, I haven’t taken a look at the other 2 links but thanks for sharing. I will keep them in mind if needed.