Move message between streams using route_to_stream and remove_from_stream

[_kotecek]

I would like to use Streams in a way that is maybe not following some general idea of Graylog - so this may a bit of architecture question of logs messages flow.

I’m using Graylog on prem with graylog-server 6.1.8-1.

I’m familiar with idea of Default Stream, but I would like to achieve following architecture:

1. create 2 Inputs input_servers and input_netdevices
2. create Stream stream_servers_all and set rule on it, saying gl2_source_input matches id of input_servers and point it to some index index_servers, rule set to remove message from Default Stream
3. create Stream stream_netdevices_all and set rule on it, saying gl2_source_input matches id of input_netdevices and point it to some index index_netdevices, rule set to remove message from Default Stream

After steps 1-3 there would be 2 Streams, each writing to it’s own Index, and this is business separation of 2 different origins of log messages.

Above is working, but then I would like to do so:

4. create Stream stream_servers_of_big_interest_or_whatever and point it to some different index

5. create pipeline in Stream stream_servers_all with following Pipeline Rule:

rule "route-logs-from-special-care-servers-to-their-own-stream"
when
 grok(value: to_string($message."designated_field"), pattern: "%{OUR_CRAFTED_REGEX}").matches == true
then
 route_to_stream( name : "stream_servers_of_big_interest_or_whatever");
 remove_from_stream(name : "stream_servers_all");
end

Above Pipeline Rule called in Pipeline for Stream stream_servers_all would take messages of special interest and route_to_stream them into some other stream.

But as I no longer need this message to be in Stream on which Pipeline Rule was called, I would like to remove_from_stream - and here goes a problem:

Inside Pipeline Rule that can be attached to many Pipelines like that - e.g I may have Stream servers_from_team_1 and servers_from_team_2 - I need to literally put a string with a name of Stream, I wish this message to be removed from.

Is there any pre-defined variable that is visible inside Pipeline Rule, saying on which Stream’s pipeline this Pipeline Rule was called?

Or do I really need to end up in calling code like that:

remove_from_stream(name : "i_know_i_run_this_pipeline_rule_on_pipeline_on_this_stream");
remove_from_stream(name : "but_i_also_run_this_pipeline_rule_on_pipeline_on_that_stream");
remove_from_stream(name : "oooooooooohhhhh_and_on_this_stream_i_call_it_too");

Or is it so, that having a messages only on one Stream, and effectively moving it from one Stream to another Stream like above is a kind of big misuse?

[Joel_Duffield]

So its not really that weird of a way to use streams.

I dont think that field you are after exists, however can you not just add the “remove from default stream” parameter to the route to stream function.

[_kotecek]

Thanks for your answer.

Function “remove from default stream” is not usable by me here, because messages I’m processing in a Pipeline are already not present in Default Stream.

My messages are moved to Stream stream_servers_all by a Rule on Stream (not in Pipeline) - just an example.

In fact what I’m trying to implement is a kind of multi-level routing of messages from less detailed Streams to more detailed Streams.

If I would like to use feature “remove from default stream” and I would have Streams - like in my example:
a) “stream_servers_all”
b) “stream_servers_of_big_interest_or_whatever”
c) “stream_netdevices_all”

Then I would need to setup Pipeline on Default Stream and - for example:
a) create Stage 10 - catching messages to stream_servers_of_big_interest_or_whatever - with rules message_is_from_input_servers and messages_is_of_big_interest
b) create Stage 40 (to be run later then Stage 10) - catching messages to stream_servers_all - with rule message_is_from_input_servers

So in above example - most of rules for these 2 Stages are defined twice (in above example message_is_from_input_server) and in my opinion it introduces risks:

• routing different types of Messages into needed Streams in a complex Stages and Pipeline Rules on Default Stream may lead to misconfiguring it and end up in state hard to debug and restore —> when routing would be set up multi-level, such misconfiguration would hit only subset of traffic
• when adjustment on conditions shared by some Stages is needed, it must be then changed (or at least confirmed) in all Stages which uses is —> when routing would be set up multi-level, a condition would be defined only in 1 Pipeline

So for now, the only strange aspect I see, when thinking about multi-level routing of Messages down from less detailed Streams to more detailed Streams, is that function remove_from_stream (which is defined in a Pipeline Rule) is not aware on which Stream’s Pipeline it has been called.

This is super strange, as Pipeline Rule may be attached to any Pipeline, and Pipelines are processing Messages from selected Streams, but in fact inside Pipeline Rule there is no variable showing which Stream (or maybe Streams) is processed by Pipeline that called this Pipeline Rule.

[patrickmann]

You can add fields to a message to help with tracking status.

A pipeline can be connected to multiple streams. A rule is running in the context of a pipeline, not in the context of a stream.

[Joel_Duffield]

First off, I was confused when responsing and yes it only removes from default stream.

As patrick said you could use a very early stage rule to write a field that states what “stream” it entered from, but in reality its just a rule that only ever runs on that stream. Then in the last stage you can delete that field etc.