JOIN GRAYLOG FOR OUR

ANNUAL CONFERENCE

Thursday,
October 21, 2021
10am-5pm CT
REGISTER NOW

Rename index set backing "All messages" stream?

Hello graylog community,
I have one more question this month:

Per Create/Cycle index set to specific ID? I am naming our index templates like “logs_r01m”, “logs_r06m”, “logs_r12m”. I am using stream filters to pick out logs for each retention period / index set. It works well, but lately I had to create a couple of special retention periods and realized that managing stream rules that route the messages to individual indices is getting harder and harder.

If I could use “All messages” stream as a fallback, then I would use it for default retention period and considerably simplify the stream rules. Unfortunately I see no way to change where logs from “All messages” go.They are forced to go to “Default index set” and then into the graylog_* index set.

I would very much like you to prove me wrong and to show me how to redirect them.

Thank you!

Oh, right, the answer to everything is “Pipeline processor”.
Obviously I can write a rule that reroutes everything from a default stream into another one and connect that to “All messages” stream.

route_to_stream(name:"logs_r12m", remove_from_default: true);

But that approach raises a few complications. First, it forces me to mix “message processing” and “message routing” in the pipelines, making them even more convoluted and making stream routing even more non-debugable. Second is a lack of documentation on what happens when pipelines re-route messages into a different stream:
I mean - Because messages are put through pipelines based on their stream membership, what happens when a pipeline rule changes stream membership?

  • Does pipeline processing continue like nothing happened?
  • Is pipeline processing aborted if the pipeline is no longer matching?
  • If so, does it abort after the step that changed streams, or after the pipeline runs to its end?
  • Is pipeline processing started in a newly matched pipeline?
  • If so, does it start at the “current” pipeline step, or the new pipeline start from its beginning?

… and all of these options sound bad.

PS: My processing order is set up like gsmith’s:

Hello,

Not sure about the pipeline but on the stream’s when you create one or one that exists, there is a tic box as shown below.

image

Yes, I am using this. But the problem is making stream rules so no message slips to graylog_* via ‘All messages’. All streams are set to “Remove matches from ‘All messages’ stream”.

Streams marked in red are there just to pull all remaining messages into the default retention period index set. If it was possible to redirect ‘All messages’, all these streams (50%!) would be unnecessary.

Rather than setting up so many stream rules, set up rules in a pipeline that creates flags-as-fields and use pipeline staging to decide what to do with the flags it finds. In later stages you can route to streams based on flag fields you find and even delete the flag field if you no longer want it. This might end up as a routing stream for all messages coming in and subsequent pipelines are attached to the the routed-to stream. You can use rule naming conventions to keep these routing rules alphabetically grouped for easier future edits.

  • Does pipeline processing continue like nothing happened?
    :point_right: I believe processing runs to the end of the pipeline regardless of when you route it. Easy to confirm in testing if you want to be sure
  • Is pipeline processing aborted if the pipeline is no longer matching?
    :point_right: Nope, once you start a pipeline, it runs to the end
  • If so, does it abort after the step that changed streams, or after the pipeline runs to its end?
    :point_right: see previous answer :stuck_out_tongue:
  • Is pipeline processing started in a newly matched pipeline?
    :point_right: Always. You can have multiple pipelines attached to a stream but I don’t think you can control sequence… yet. I think that’s what you were thinking…
  • If so, does it start at the “current” pipeline step, or the new pipeline start from its beginning?
    :point_right: Pipelines always start from the beginning and run all the way out. There was some questions in the forums recently asking about why pipelines continue to run after a drop_message() function… this is what leads me to believe there is nothing that currently aborts a pipeline.

It would be nice to choose pipeline sequence on a stream - or for that matter rule sequence in a pipeline stage… or even to have an abort_stream() function… all those are currently potential feature requests.

2 Likes

Thank your for this long answer.
Yes, I should have said I am asking “In case someone knows” and otherwise fall back to experimenting.

Let’s have a simple scenarios:

  • two streams ‘All mesages’ and ‘Alternate stream’
  • no stream rules, everything goes to ‘All messages’
  • two pipelines, ‘Default’ and matching ‘All messages’ and ‘Alternate’ and matching ‘Alternate stream’.
  • both pipelines have stages -2, -1, 0 and 1

Now, new messsages arrives and is routed to ‘All messages’ so processing in pipeline ‘Default’ starts.
Let’s imagine in step 0 there is route_to_stream(name:"Alternate stream", remove_from_default: true);

What do you expect to happen? OK, pipeline ‘Default’ will probably run all the way to its end.
You expect that newly matched pipeline ‘Alternate’ will start at this point. Will it start processing its stage -2 concurrently with Default pipeline stage 1? Or it will wait until pipeline Default finishes and start after that?

Good question - someone inside Graylog or who has tested that scenario would have to answer… Maybe @aaronsachs can provide some insight? Based on how Graylog handles rules in each stage, it would suggest that it would randomly handle that scenario - which is unhelpful. The result is to be mindful and place your routing and dropping at the end of pipeline staging.

@nisow95612
Hello,

Have you seen the Pipeline Simulator? I personally haven’t used it a long time but it did help me out on Staging and Rules. Maybe it will give you a better insight.