Alerting/Notification when certain sources are not reporting

[dahacker]

This is for Graylog 4. It is my understanding that the message count alert condition no longer exists.

I have ~30 source hosts reporting their messages. I’d like to receive an alert when any one of them stops reporting.

I can certainly program 30 separate notifications for EACH of the 30 hosts to Filter & Aggregate and then alert when the count() for each is zero, but is there a simpler way with a single notification and grouping by source and checking for sources not received?

[gsmith]

@dahacker
hello,
A quick search and I found these. I think its possible but I havent tried it yet. Hope this helps.

Content Pack - Event Source Not Sending Logs

Notification alert if device is not sending log

Hi Team, we are facing some issue in the notification, In our setup, we are getting 300 network device logs but the issue is that if any device stops the sending log in Graylog server due to some reason so how we can know in Graylog. we are also monitoring our service and port, but please help on same how we can resolve the same issue.

Failed login sources alerting

The way I approached this was to have a dedicated stream for that particular dataset and then create a Message Count alert on it so that if the count is 0 after X minutes, send a notification. I also setup a Dashboard to monitor my forwarders and data over time (5 mins, 1 hr, 8hrs, 1day, etc.). I do the “source” in one panel and a data “type” in another - it provides a view of what what received by the server over time and let’s me know if something stopped forwarding. If you do something like…

Event definitions and alert if no logs graylog 3.2.5

Hi ! I’ve about 30 sources for my graylog and i want to generate an alert if some sources no longer sends logs for a given period of time. On my graylog it works if a create one stream and one event definition for each sources but i want to improve this way Can I do the same with only one stream and one event definitions ? I try on the “all messages” stream’s : search within the last 5 min, execute search every 5 min If count(source) <= 0 but it doesn’t work, what i’m doing wrong ? Lastly…

[dahacker]

From what I can tell, these all apply to earlier versions of Graylog. In particular, that plugin that is built for this problem is not for Graylog 4.

[gsmith]

Sorry about that, I just seen that.
Were looking for a solution about tracking source also for GL 4.
Havent figured it out yet but here is our Idea.

We’re using Zabbix for trending data in our environment. What were looking at is something like this.
I created a widget on the " Sources" Dashboard called Number of Sources.

Then using the api-browser found in System/Nodes get the info from that widget.

    {
          "id": "5e4355886704f1e20679f322",
          "type": "DASHBOARD",
          "title": "Sources",
      
                "widget": {
                  "4a02a354-84b8-4b5b-a7c7-ceb8c5976a3a": "
                  "6c127c5d-be75-4157-b43f-ac0194ac0586": "Selected sources",
                  "212ea3e2-4484-4df2-8abd-266b9e660799": "
                  "4fa48004-28f5-41de-b6c0-f4d4651013ad": "Field Statistics for EventID",
                  "00637e63-d728-4b3e-932b-7c8696b4855d": "Messages over time",
                  "8f9dc4b3-5ccf-429b-93b1-58d062f9607c": 
                   "9c8c9d14-5c29-4a17-a958-d9ac80eff0a2": "Number of Sources",
                  "3f125c6c-adda-4945-ae70-284620351b13": 
                  "55296df7-5c21-41a3-ab78-dd684219b74e": 
                  "81011bcc-93e2-412c-84e8-759c791b76a9": 
                  "85eba10b-8922-48cb-9e82-7bc963540035": "Message Count 1 Day",
                  "7d11ff08-2ed6-471e-b606-254a1ea382db": "
                  "e3664cb9-9c47-46b2-894b-4c3a0587345f": 
                  "8aef2d13-da5c-4ac9-b71b-b20762735685
                  "92d63811-e4dd-47db-bd3b-db03c8a9bd53": "Messages per Source"
                }
              },

Maybe through a bash script and then sending it to Zabbix. From there I can create a tigger/alert if the number changes. But then how would i tell which node is not send data? Just a thought. Other then that I dont know either. Were still looking for a simple solution.

[dahacker]

Thanks for the number of sources idea. We’ve got about 30 sources, so it isn’t out of the question to create 30 notifications, but it seems like a big hole in the Graylog design. Creating additional API reading scripts is certainly possible, but an over-complicated kludge for us.

I think it is weird these threads get automatically closed after 14 days. This would be a great topic to revisit if ever solved by configuration of current capabilities or a new feature.

Thanks again.

[gsmith]

@dahacker
No problem, sorry I couldnt be more help. Just an FYI I believe this can be done with the enterpise. version.

[ttsandrew (Andrew)]

@gsmith Unfortunately there’s not a built in mechanism for this in the enterprise version either, we use it and ended up crafting our own solution. We couldn’t get the content pack to work the way we wanted either.

[gsmith]

@ttsandrew
Thank you for the Information.
My team was acquiring about how to do this also.

[lisa87]

Here is a suggestion : Create an event definition that runs every X minutes (define X as you see fit). In aggregation, define a condition based on the card() of the “source” field. If it is anything less than your number of sources, fire an alert.