MS Teams Notification broken for me?

1. Describe your incident:

In my development graylog instance - Using the “Timestamp” field within an MS Teams notification in Graylog causes, um, odd behaviour.
It appears that the “:” within the field causes some sort of issue with the message.

See this image:
graylog issue

The above alert was generated by this notification text

Timestamp:            ${event.timestamp}
Message:              ${event.message}
Trigger:                  ${event.key}
Timestamp Processing: ${event.timestamp}

Event Fields:
${foreach event.fields field}
${field.key}: ${field.value}
${end}
${if backlog}
--- [Backlog] ------------------------------------
Last messages for this alert:
${foreach backlog message}
Time: ${message.timestamp} - user_name: ${message.fields.user_name} - Device: ${message.fields.user_domain} ${message.fields.IpAddress}
${end}${end}

I can reproduce by just having the timestamp field present, or even just manually typing in a timestamp.

Slack notifications do not have this issue, and messages are displayed as expected - using the same message template.

2. Describe your environment:

Ubuntu 20.04 LTS
Graylog 4.3.2, Elasticsearch 7.10.2

3. What steps have you already taken to try and solve the problem?

I have recreated the notification within Graylog and changed the “custom message” to no avail.
I have attempted to “escape” fields within the message using different teams escape strings

I’ve also added the notification to a few different alert definitions and they all exhibit the same problem for me.

I downloaded the original 3rd party plugin for Teams notifications and tried that, to no avail.

I have access to a production Graylog instance for a customer, and this does not have the same issue; although Graylog is version 4.2.x and uses a third party plugin for MS Teams notifications.

4. How can the community help?

Does anyone else experience the issue?
Am I being stupid? I feel like I’m doing something wrong but I’m not able to pinpoint it :frowning:

Hello,

Correct me if I’m wrong. this section of your notification does not work as expected?

Last messages for this alert:
${foreach backlog message}
Time: ${message.timestamp} - user_name: ${message.fields.user_name} - Device: ${message.fields.user_domain} ${message.fields.IpAddress}
${end}${end}

If so perhaps something like this as shown below, insure those fields are present in the message.

${if backlog}
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
${foreach backlog message}
Time:         ${message.timestamp}
user_name:    ${message.fields.user_name}
Device:       ${message.fields.user_domain}
IpAddress;    ${message.fields.IpAddress}

${end}
${end}

Hi GSmith,

Sorry for not being too clear, the issue I’m seeing is any colon “:” inside the Teams notification breaks the message.

For example: the timestamp field, will always contain a : - but Teams does not display the correct timestamp, it gets cut off right at the position of the : (as you can see in my original picture)

I can also reproduce the issue by simply manually typing text into the notification field too that contains a : - so if I manually type the timestamp of say - 2020-05-20T11:35:11.117Z - it will only display “2020-05-20T11”

I see, so this is a timestamp issue with the notification,is this correct?

Noticed that the timestamp example 2020-05-20T11:35:11.117Z is in Epoch Time which the notification is using this yyyy-MM-dd HH:mm:ss

Perhaps try to convert it before it sends.

rule "Epoch Convert"
when
  has_field("eventtime")
then
  let ts_millis = to_long($message.eventtime) / 1000;
  let new_date = parse_unix_milliseconds(ts_millis);
  set_field("epoch_timestamp", new_date);
  //set_field("timestamp", new_date);
end

HI,

I don’t know what the issue is to be honest, but it doesn’t appear to just a timestamp issue, I think its a notification issue. Its like the text being sent to MS teams is not being interpreted/parsed correctly.
Especially as typing anything with a : within the notification text field will trigger the same outcome.

I’ve tried converting it to a new field with a pipeline rule as you provided but no dice, same thing.
This happens even when simply clicking the “test notification” button also :frowning:

My next step is to get a “working” system to send me a notification to my webhook - this should indicate if the issue is teams side, or graylog side atleast. I will report back if I find anything

Appreciate your help.

Hi,

It seems that a lots of things are not going the way they should in Microsoft teams regarding Graylog’s notifications. I have myself a 4.3.2 graylog installation, ant I’m having the same issue. Also, when the fields are too long, they’re automatically cut by Teams (e.g : URIs). :clown_face:

So for me you’re right saying this is a notification issue.

Like you, I will report back my experience with this problem here.

Hello @C21.Scott

It could be, I compared my Notification to yours. The only difference was my Timestamp Processing

--- [Event Definition] ---------------------------
Title:       Windows: Permissions on an object were changed
Description: Windows:  Permissions on an object were changed
Type:        aggregation-v1
--- [Event] --------------------------------------
Timestamp:            2022-07-04T20:51:37.693Z
Message:              Windows: Permissions on an object were changed: greg.smith.domain.com - count()=14.0
Source:               graylog.domain.com
Key:                  
Priority:             2
Alert:                true
Timestamp Processing: 2022-07-04T20:51:37.693Z
Timerange Start:      2022-07-04T20:50:37.693Z
Timerange End:        2022-07-04T20:51:37.693Z
Fields:


--- [Backlog] ------------------------------------
Last messages accounting for this alert:

TimeStamp: 2022-07-04 15:50:53

If you noticed the Timestamp Processing is for 8:51 PM but my timestamp is for 3:50 PM, which I think is like your issue. To confirm I checked alerts.

Results:

Also check original message " timestamp" field.
results:

So at this point I had the same issue, What I noticed was this setting in my Notification template.

Change the time zone settings to Chicago

Results:

Timestamp Processing: 2022-07-04T22:49:18.126Z
Timerange Start:      2022-07-04T22:48:18.126Z
Timerange End:        2022-07-04T22:49:18.126Z
Fields:


--- [Backlog] ------------------------------------
Last messages accounting for this alert:

DWM-4
-
TimeStamp: 2022-07-04 22:48:18

Message Link: https://graylog.domain.com:9000/messages/graylog_1656/4ecb6cb4-fc15-11ec-8588-00155d601d11

Hope that helps

Hi all,
I’m experiencing the same issue. Not only does it apply to Timestamp field but to the Message one as well. If it contains colon(s) it gets split at the first one and the part following the first colon is discarded:
Screenshot - 11.07.2022 , 11_30_47
Looks to me like field content is being kind of interpreted when sent to Teams and colon works like in the template itself.
graylog 4.3.3 with Teams plugins 2.0.4 / 2.0.0 - neither seems to work properly

Hello,

My example above took care of that notification issue. Also check Web UI under System/Overview to insure all three date/times are correct. As for the plugin, I haven’t used that and are you sure Teams plugins 2.0.4 / 2.0.0 is able to work with GL version 4.3?

Hello,
In my opinion it is not timestamp-related issue. Please take a look at this:


On the left there is Graylog template with sample text entered (and highlighted). On the right is the Teams notification with the same text split by colon.
Here is HTML code for this notification:

<tr class="card-fact" ng-repeat="cardFact in ::ctrl.section.facts">
  <td class="card-fact-key" title="sam">sam</td>
  <td class="card-fact-value card-html card-html-hide-img" ng-bind-html="::ctrl.trustAsHtml(cardFact.value)"><p>ple text he</p></td>
</tr>

As you can see colons split the text into two-column table (text following the second colon gets discarded as there is no third column). Unfortunately the same applies to the message content field ${message.message}:

<tr class="card-fact" ng-repeat="cardFact in ::ctrl.section.facts">
  <td class="card-fact-key" title="Message">Message</td>
  <td class="card-fact-value card-html card-html-hide-img" ng-bind-html="::ctrl.trustAsHtml(cardFact.value)"><p>testvm MSWinEventLog	3	Security	3031	Mon Jul 11 21</p></td>
</tr>

The message content is rendered as table cell and when it contains colon the remainder following one gets discarded (the valid message content is: testvm MSWinEventLog 3 Security 3031 Mon Jul 11 21:44:32 2022 4625 Microsoft-Windows-Security-Auditing(…)).
Looks to me like fields content is not being escaped/sanitized (not a programmer) before being converted to html.
As for the plugins: I managed to load the one downloaded from github for testing:

2022-07-11T11:42:00.807+02:00 INFO  [CmdLineTool] Loaded plugin: Microsoft Teams Notification 2.0.0 [org.graylog.plugins.teams.TeamsNotificationPlugin]
2022-07-11T11:52:22.116+02:00 INFO  [CmdLineTool] Loaded plugin: Microsoft Teams Notification 2.0.4 [org.graylog.plugins.teams.TeamsNotificationPlugin]
[...]
2022-07-11T11:52:25.828+02:00 INFO  [ServerBootstrap] Graylog server 4.3.3+86369d3 starting up

yet the results were the same as with stock one.
BTW: all was fine with the old-fashioned notifications (cannot embed more than one image).

2 Likes

Thanks for all the input on this guys.

I installed a fresh copy of Graylog 4.2.11, and then used this plugin: GitHub - hidapple/graylog-plugin-teams: Graylog event notification plugin to send messages to the Microsoft Teams

Teams notifications send perfectly fine, so there does seem to be an issue with whatever happened in 4.3.

Is this something I should post over on the Github issues page?

To be honest, its probably something was over looked and were unable to find it. Since you stated a fresh installation fixed it that is what I’m assuming.

But it never hurt to post, It could very well be a issue with the plugin.

EDIT: there are some bug related issue with 4.3 so the plugin might not be compatible with that version yet.

Hello,

Did some more research on this issue. I just realized that Graylog Version 4.3 has this Plugin by default.

Shown here

After reading the Changelog I check my installation and it was there already.

Did you noticed this also @C21.Scott ?

Same problem graylog version - 4.3.2+313b6bc , build in plugin

Yeah, I believe it was included with 4.3 release.

Thanks for all the input, I think I will raise this as an issue on Github and link to this post so it can be reviewed just to check there isn’t something funky going on.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.