Alerting/Notification when certain sources are not reporting

This is for Graylog 4. It is my understanding that the message count alert condition no longer exists.

I have ~30 source hosts reporting their messages. I’d like to receive an alert when any one of them stops reporting.

I can certainly program 30 separate notifications for EACH of the 30 hosts to Filter & Aggregate and then alert when the count() for each is zero, but is there a simpler way with a single notification and grouping by source and checking for sources not received?

1 Like

@dahacker
hello,
A quick search and I found these. I think its possible but I havent tried it yet. Hope this helps.

Content Pack - Event Source Not Sending Logs

From what I can tell, these all apply to earlier versions of Graylog. In particular, that plugin that is built for this problem is not for Graylog 4.

Sorry about that, I just seen that.
Were looking for a solution about tracking source also for GL 4.
Havent figured it out yet but here is our Idea.

We’re using Zabbix for trending data in our environment. What were looking at is something like this.
I created a widget on the " Sources" Dashboard called Number of Sources.

Then using the api-browser found in System/Nodes get the info from that widget.

    {
          "id": "5e4355886704f1e20679f322",
          "type": "DASHBOARD",
          "title": "Sources",
      
                "widget": {
                  "4a02a354-84b8-4b5b-a7c7-ceb8c5976a3a": "
                  "6c127c5d-be75-4157-b43f-ac0194ac0586": "Selected sources",
                  "212ea3e2-4484-4df2-8abd-266b9e660799": "
                  "4fa48004-28f5-41de-b6c0-f4d4651013ad": "Field Statistics for EventID",
                  "00637e63-d728-4b3e-932b-7c8696b4855d": "Messages over time",
                  "8f9dc4b3-5ccf-429b-93b1-58d062f9607c": 
                   "9c8c9d14-5c29-4a17-a958-d9ac80eff0a2": "Number of Sources",
                  "3f125c6c-adda-4945-ae70-284620351b13": 
                  "55296df7-5c21-41a3-ab78-dd684219b74e": 
                  "81011bcc-93e2-412c-84e8-759c791b76a9": 
                  "85eba10b-8922-48cb-9e82-7bc963540035": "Message Count 1 Day",
                  "7d11ff08-2ed6-471e-b606-254a1ea382db": "
                  "e3664cb9-9c47-46b2-894b-4c3a0587345f": 
                  "8aef2d13-da5c-4ac9-b71b-b20762735685
                  "92d63811-e4dd-47db-bd3b-db03c8a9bd53": "Messages per Source"
                }
              },

Maybe through a bash script and then sending it to Zabbix. From there I can create a tigger/alert if the number changes. But then how would i tell which node is not send data? Just a thought. Other then that I dont know either. Were still looking for a simple solution.

Thanks for the number of sources idea. We’ve got about 30 sources, so it isn’t out of the question to create 30 notifications, but it seems like a big hole in the Graylog design. Creating additional API reading scripts is certainly possible, but an over-complicated kludge for us.

I think it is weird these threads get automatically closed after 14 days. This would be a great topic to revisit if ever solved by configuration of current capabilities or a new feature.

Thanks again.

1 Like

@dahacker
No problem, sorry I couldnt be more help. Just an FYI I believe this can be done with the enterpise. version.

@gsmith Unfortunately there’s not a built in mechanism for this in the enterprise version either, we use it and ended up crafting our own solution. We couldn’t get the content pack to work the way we wanted either.

2 Likes

@ttsandrew
Thank you for the Information.
My team was acquiring about how to do this also.

Here is a suggestion : Create an event definition that runs every X minutes (define X as you see fit). In aggregation, define a condition based on the card() of the “source” field. If it is anything less than your number of sources, fire an alert.

2 Likes