We are a university each departments have their own firewalls. We are gathering all the firewall logs in graylog in which we have a stream per department. Each department IT team has viewing permission on their stream and only their stream. We have a custom geocity lookup table that, instead of returning a city, returns the department that the IP belongs to. We also have a pipeline that add a department field using that lookp table. All this works well, as expected.
Now, we would like to provide to each department IT teams with all the firewall logs that matches their IPs even if it was received on a firewall of another department. We were planning to use our custom field “department” but since it’s being added by pipelines the addition is only done once the log line arrives in its final stream hence not being available for steam rules.
So here’s my question: Is there another way to had a custom field, depending on a lookup table and that would be available for stream rules? A way to add that field to log lines when they are received by Graylog’s inputs?
You could create and attach a new master pipeline that has the lookup table and rules to pass messages to one or more department streams where the department streams would only receive from the master (i.e. detach the department streams from the Input) if you use route_to_stream(), the message will finish all stages in the master stream before moving on to the department streams. Interestingly you could also simply attach the master stream AT THE SAME TIME… in this case you would make sure the master stream stages are of a smaller numeric priority (negative priorities are OK) because when pipelines are run in parallel, the stages execute in parallel based on priority which also means lower priority number stages are executed before higher number stages. So two ways you could inject the master rules.
