Notifications won't load or send

Graylog Central (peer support)
,
1

GL 4.1.2
We just recently setup event notifications but found they had stopped (Slack notifications). When I try to load the page for “Notifications” I only get a spinning circle and “Loading Notifications Information…”. I got a similar loading message when I tried to edit the event definition so I tried deleting that to no avail.

Initally this is the message we saw in the graylog.log file

2021-07-29T04:58:23.022Z ERROR [EventProcessorExecutionJob] Event processor <aggregation-v1/60ed1aa3e64b7d0138ec153f> failed to execute: Couldn't emit events for: EventDefinitionDto{id=60ed1aa3e64b7d0138ec153f, title=Account Lockout, description=, priority=2, alert=true, config=AggregationEventProcessorConfig{type=aggregation-v1, query=winlogbeat_winlog_event_id:4740, queryParameters=[], streams=[60cecc07ebede23cd3a01e64], groupBy=[], series=[], conditions=Optional[AggregationConditions{expression=Optional.empty}], searchWithinMs=60000, executeEveryMs=60000}, fieldSpec={username=EventFieldSpec{dataType=STRING, providers=[Config{type=template-v1, template=${source.winlogbeat_winlog_event_data_TargetUserName}, requireValues=true}]}, dc=EventFieldSpec{dataType=STRING, providers=[Config{type=template-v1, template=${source.winlogbeat_winlog_computer_name}, requireValues=true}]}}, keySpec=[username], notificationSettings=EventNotificationSettings{gracePeriodMs=0, backlogSize=0}, notifications=[Config{notificationId=60d678791876621ab5471bff, notificationParameters=Optional.empty}], storage=[Config{type=persist-to-streams-v1, streams=[000000000000000000000002]}]} (retry in 5000 ms)

After deleting the event definition that message stopped, but we get this when trying to load the Notifications page:

2021-07-29T06:05:43.451Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "type" (class org.graylog.events.notifications.EventNotificationConfig$FallbackNotificationConfig), not marked as ignorable (0 known properties: ])
 at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@598ea475; pos: 109] (through reference chain: org.graylog.events.notifications.AutoValue_NotificationDto$Builder["config"]->org.graylog.events.notifications.EventNotificationConfig$FallbackNotificationConfig["type"])

I imagine if I can delete the notification configuration I can recreate it, but I’m not sure how to proceed. Any thoughts?

2

  1. Try to use REST API, to list all notifications, get id of notification:
    curl -s -X GET -u admin:pass -H 'X-Requested-By: cli' -H 'Content-Type: application/json' 'http://172.28.128.18:9000/api/events/notifications?query=&page=1&per_page=100'|jq '.notifications[] | {id, title}'

  2. Then remove notification by it’s id:
    curl -i -X DELETE -u admin:pass -H ‘X-Requested-By: cli’ -H ‘Content-Type: application/json’ http://172.28.128.18:9000/api/events/notifications/61027f5b91c1f372f2eaa8fa

Replace 61027f5b91c1f372f2eaa8fa with your real id from step 1. Replace admin:pass with your user. Replace IP and port with your real values.

PS: My example uses jq command, either install it, or remove |jq '.notifications[] | {id, title}' and find id in json output in notification section.

Or use Rest Api Browser if you are not familiar with curl:
https://docs.graylog.org/en/4.1/pages/configuration/rest_api.html

2 Likes
3

Thanks so much for the fantastic response!

Using the curl command I got the same error but an error in the server.log file of “IOException encountered while reading from a byte array input stream”.

Once I Googled that, I found a similar issue here on the community (Graylog Content Packs page will not load after upgrading from v3.0 to v3.1) which lead me down the path of deleting the mongodb document for the notification in question:

db.event_notifications.remove({"_id" : ObjectId("60d678791876621ab5471bff"), "title" : "Slack Notification"})

Once I was able to delete that the page loads fine and I’m able to recreate alerts.
Thanks so much for giving me the hint I needed!

1 Like
4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.