Pipeline rule that route to a stream

[wilsonshow]

I struggle with configuring pipelines, but I want to ensure that any mention of ‘filebeat’ in my messages directs them to the ‘Testing Filebeat’ stream while removing them from the ‘Default Stream’.

Below are my pipeline rules along with a sample log message:

Pipeline Rules:

rule "Testing Filebeat Pipeline"
when
    // Check if the message comes from filebeat
    contains(to_string($message.message), "filebeat")
then
    // Route filebeat logs to a specific stream
    route_to_stream("Testing Filebeat");
    // Remove logs from the default stream
    remove_from_stream("Default Stream");
end

Log Message:

graylog01 filebeat[66352]: 2024-02-29T14:22:03.177+0800#011INFO#011[input.harvester]#011log/harvester.go:310#011Harvester started for paths: [/var/log/* /home/user1/Andriod Log/*.log]#011{"input_id": "d91cf4a8-5179-4be1-bc83-0f21b19b48ed", "source": "/var/log/btmp", "state_id": "native::678-64768", "finished": false, "os_id": "678-64768", "old_source": "/var/log/btmp", "old_finished": true, "old_os_id": "678-64768", "harvester_id": "ff5fb596-016c-4367-ad88-31773a42219f"}

[drewmiranda-gl (Drew Miranda)]

I recommend using the remove_from_default argument when using the route_to_stream function.

For example:

rule "Testing Filebeat Pipeline"
when
    // Check if the message comes from filebeat
    contains(to_string($message.message), "filebeat")
then
    // Route filebeat logs to a specific stream
    route_to_stream(
        name: "Testing Filebeat",
        remove_from_default: true
        );
    
end

[wilsonshow]

Hey @drewmiranda-gl

Thank for you reply and learn about remove_from_default: true

Anyway I figure out why my pipeline is not working. The stream I created is call “Testing FileBeat” Is case-sensitive.