Pipeline does not work with custom stream, only with "All Messages"

Documentation Campfire
1

This post is similar to

I have a custom stream with a rule like " Field filebeat_tags must contain my_app". Stream works and I can see messages. Then I have a pipeline where I modify messages. Pipeline works (apply pipeline rule) only when it’s connected to “All Messages” stream but not my custom stream.

Similar posts suggest to use route_to_stream function in pipeline but none of them explain why. I’ve read docs but still could not figure out what is going on. It looks like Pipelines are run before Streams. And, in that case, Pipeline can only take input from “All Messages” and, when connected, it outputs messages to specified stream. Is that it?

2

Well, I triple read Stream connections — Graylog 3.2.0 documentation and it looks like indeed Pipeline are run before Stream unless otherwise is configured. Welp, Graylog UI is confusing in this case. They should have organizes UI in the order of data flow.

3

I did the following:

  1. Created new Pipeline rule
rule "Stream Router: To My App"
when
    has_field ("filebeat_tags") && contains(to_string($message.filebeat_tags), "my_app")
then
    route_to_stream (id: "626b51f8afee6f027798a1c7", remove_from_default: true);
end
  1. Added new Pipeline connected to “All Messages” and assigned the above rule to Stage 0
  2. Connected my original pipeline (with message processing) to “My App” stream

At this moment my pipeline rules were applied to messages. But, then I removed Stream Rule from My App stream and… while everything still works, UI shows 0 throughput on that stream. Well, I know why, because Stream UI things that nothing is routed to that stream since it has no Stream Rules.

This is mildly frustrating. I think I’m to simply this all by

  1. Remove routing pipeline and use of route_to_stream function
  2. Connect My Pipeline to All Messages and create and use same matching rule as I used before in route_to_stream
  3. Add Stream Rule to My Stream so it picks up needed messages after My Pipeline.
4

The processor order is the key

1 Like
5

To me it looks like there was one behavior, and then they changed it for what ever reason and “processor order” allows to revert it to old behavior. Old = deprecated. If not, then I’m not sure I understand the point of such configuration. I wonder if it’s an artifact of v.3.1 that we use and maybe old behavior was removed / streams configuration was changes in latest versions.