Alert based on single field, but with multiple sources?

Graylog Central (peer support)
, ,
1

I have a simple, small powershell script installed on 400 Windows Server’s, that outputs disk, CPU and RAM utilization once per minute.

NXLog then picks this up, and sends it to Graylog;

C
43
D
19
cpuload
0
freememory
26720

I have fed these into a stream, and I want to send out an email alert, every time one of these numbers drops below a threshold. I then want a 30 minute grace-period.

Now, this is obviously easy to do if the stream takes EVERY event into account, but I need to differentiate between sources. So in other words, if souce:testserver01 triggers an alert, I want there to be a grace period, but I don’t want this grace period to prevent an alert being sent from a different source

Is there a ‘preferred’ way of achieving this?
The only option I can think of is to make 400 streams, one for each source!!!

Thanks,
Tom

2

That’s currently how you would model your requirements with Graylog.

I’d recommend using something different than Graylog for this particular use case, e. g. Prometheus, which has been designed to record and alert on metrics (rather than log messages like Graylog). Metrics and log messages (or structured data in general) are a bit different.

Just a side-note, but this could probably also be done with Metricbeat without the need for nxlog.

3

Thank you Jochen - I think I’ve found the middle-ground with our applications team, where they are happy to have every event on the same email, for the past hour. This means I can just use call-backs.

Going forward, I’ll see if there is anything more suited to the task.

Thanks,
Tom