Graylog Alerts are really proving their worth here!
How can I limit stream input or condition or alert such that I am notified when a given process or activity being monitored fails (stops producing messages) only during business hours, say, 8a-5p? I won’t even try to tackle “and not school holidays,” unless you have some ideas there…
I have increasingly set these up as heartbeat alert for system processes that must be running (waiting) (and producing log messages) during business hours. After hours, I expect these processes to not produce messages, bc there is no user activity to trigger it.
With pipeline you can set a field eg. business_hour true or false, and set your stream based on this field. (Or you can put the message into a stream with pipeline)
And if you have too much time, you can also create eg. a lookup table for special days, and make a decision based on the ours, the weekday and the lookup table.
Ok that is clever!
Hmm… But once a message is marked, I still can’t tell Conditions to anticipate getting dead air (no inbound logs outside business hours). Conditions can only route message, not silence, right? What am I overlooking here?
You use the business_hours field as part of the condition, or rather, the query in the condition so you can include business hours - e.g. append AND business_hours:"yes" or something to it. Then your condition will be equal to something like “alert if less than X messages and it’s during business hours”.
Thanks @benvanstaveren - I’ll let my grey matter churn on this one: I am still not seeing past how I can give a condition a pipeline-calculated value business_hours:"yes" in a period during which no logs are expected to stream in: Normal Condition during business hours: Specific messages streaming in, pipeline-decorated with business_hours:yes. Normal Condition during off-business hours: Specific messages stopped streaming in. Alarm Condition: Specific messages stopped streaming in (and it is business hours, but there are no messages to decorate)
I appreciate your creative solution and hope my mind can grok what you’re handing.