1. Describe your incident:
Have been using graylog for years and one of our common scenarios it to warn us of a certain log has become inactive such as to notify if a process has become hung up or stopped for some reason. This works great with specific stream setup and an event defined by condition “count() < 1” for the past 2 hours.
However we would now only like to throw this event if it is during specific hours or days. Some of these streams can normally become inactive or slow during overnight or weekends so that is normal. I would only like to be notified if they become inactive during say 8 am - 10 pm EST, M-F.
Currently we have these flowing into a separate ticketing system that can auto close during those time frames, and escalate during core hours, , but I would really love to to have it all work within GrayLog.
Is this possible? Thank you for any guidance!
We are currently running version Graylog 4.2.9 on Linux 5.1.
Thanks for asking us! There are several expert Graylog users who hang out here and will respond with experienced recommendations and suggestions. My response is based on the document. I hope it helps.
You can set up notifications in Graylog to alert you when a specific log becomes inactive during specific hours or days. You can use a Stream and an Event Definition to set up a notification based on the condition “count() < 1” for the past 2 hours, as you mentioned, but then set up a Scheduled Event to define the specific hours and days. Look under the “Triggers” section of the alert, and set up a schedule for when the alert should be active. For example, to only trigger the alert during 8 am - 10 pm EST, M-F, you would set the schedule to “Active: Mon-Fri 8:00-22:00” and “Timezone: America/New_York”.
You can find more information about setting up notifications and Scheduled Events in the Graylog Documentation:
David, That sounds great but I do not see anywhere to configure a ‘trigger’. Also, I do not see anything about ‘Scheduled Event’. Can you provide additional detail?
My explanation seems to have confused more than helped. My response was based on the information I got from the 4.x documentation. Here’s where I gleaned some of that information about triggers and scheduled events. While not the specific terms, these links go to documentation about those subjects:
David, thanks. From what I can tell, the abiliity I’m looking for is not available - Specifically the ability to filter events or notifications based on a time of day.
There is not a setting for a Specific time of day, there is not a setting/s to fire a alert at 9:32 PM on Monday. BUT when you create that alert for 24 hours and your Notification/Grace Period set for 24 hour then those alert will go off ever day if alert was triggered.
Graylog has a geat API and allows the usage of this. My recommendation would be to switch on and off the alerts of your choice with the API triggered by a litte python-cronjob:
I over looked that, good catch.