Scale limit on the maximum number of streams/alerts/notifications


(Rohit) #1

Hello. My apologies as this post is a bit “long-ish” and many thanks in advance to those who read it through and provide their feedback!

I am looking into graylog for alerting based off of syslog messages for approximately 300 devices (this number could expand a lot more in the future). Here is the problem I am trying to solve:

-Each device is capable of sending approximately 10 different types of syslog messages.
-Alerts should be generated every time one of these syslog messages are received.
-Alerts should be generated for each individual device as each device could be sending syslog messages at the same time (or in close proximity of a grace period).

Since alerts have to be generated on a per device basis, if similar syslog messages from all devices are combined into one stream, it is not possible to alert for each device (multiple devices send in the same syslog message at the same time or within a short grace period of each other).

In order to resolve this issue I am thinking of creating a stream for each device times 10 (with the current numbers that would be 3000 streams and additional alert and notification conditions for each stream!). This way graylog should be able to alert on each syslog message type from each device independently of each other. Which brings about the scaling question.

Is there a limit to the number of streams, as well as equivalent alert and notification conditions that can be created within graylog?

Also, am I missing a better way to tackle this issue?

I am running graylog v2.4.3+2c41897 (it is currently running in AWS for internal evaluation purpose). Based on the response it maybe possible to run it as a bigger production type setup. Thank you again for suggestions and feedback!


(Jochen) #2

No, there’s no relevant limit. This being said, processing requirements (linearly) increase with an increasing number of streams and alert conditions.

Are you sure you need to have individual alerts for each device and each type of message? Usually it’s possible to group a certain number of devices and simply send an alert notification if one of these emits a specific message. Details about the source can be read from the alert notification.


(Rohit) #3

Thank you for the update! These are network devices and each device connects multiple end hosts. Hence the need for having individual alerts from each device for each type of actionable syslog message. Otherwise, with multiple devices grouped together for alerts configured on a specific message type, there is a possibility of an alert getting missed if the same syslog message is generated around the same time by more than one network device.


(system) #4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.