Server log getting flooded with notification warnings since 6.2 changes

Graylog Central (peer support)
,
1

Hi everyone!

1. Describe your incident:
Since the update to 6.2 the graylog-server logs are getting flooded with warning messages regarding notifications, see 2). Since it’s every few seconds there are tens of thousands of lines like this.
The changelog also states the following which probably provoked the messages:

Removed noisy system notification when event limit is exceeded. graylog2-server#20785 graylog2-server#21080

2. Describe your environment:

  • 3 VMs
  • 1x Graylog Server 6.2.2-1 with Nginx 1.22.1-9+deb12u1
  • 1x Graylog Datanode 6.2.2-1
  • 1x MongoDB 7.0.20
  • latest Debian 12.10 Bookworm

/var/log/graylog-server/server.log

[...]
2025-05-08T11:48:45.040+02:00 WARN  [NotificationServiceImpl] There is a notification type we can't handle: [event_limit_reached]
2025-05-08T11:48:48.048+02:00 WARN  [NotificationServiceImpl] There is a notification type we can't handle: [event_limit_reached]
2025-05-08T11:48:51.098+02:00 WARN  [NotificationServiceImpl] There is a notification type we can't handle: [event_limit_reached]
[...]

3. What steps have you already taken to try and solve the problem?
Restarting the whole stack didn’t help and I saw in another post it could be because the database still has the notification type stored but since Graylog won’t process it anymore, it will throw the warning. I’m not too keen to meddle with the database entries myself though.

I disabled all our alerts and also duplicated them and deleted the old ones but the warnings keep repeating.

4. How can the community help?
Anyone else with the same problem and/or knows a fix? I’m guessing it might not be too hard but since this is something I would have thought Graylog handles on it’s own, manually changing things might break even more.

2

What version were you on before 6.2?

3

Hello Joel,

we were on 6.1.9 before.

When I saw that 6.2 got released I first upgraded to 6.1.10 as that was available too and then to 6.2.

4

The noisy notification was removed. Unfortunately, you still had an active instance in your DB.
Since the UI polls for active notifications every few seconds, you end up with all those log messages.

Your best bet is to delete the offending entry from Mongo DB collection notifications.

5

Thank you @patrickmann !

I saw that there’s the MongoDB Compass which made it really easy to discover and delete the offending entry. It indeed was a single one that somehow didn’t get cleared.

Now the server log is quiet again.

1 Like