Remove fields function not working in pipeline

Hi! We have been using graylog for a long time already but only to storage system logs from Linux servers. I wanted to use it for more than that, mainly to get more information of our service logs (Apache, BIND, Haproxy, etc.) and create dashboards and alerts. So far i created a Test Graylog server with the lastest version (3.2) on Debian 10. I am receiving messages from 2 different servers, in both cases i installed the sidecar and i am using Filebeat to collect them. I also created an extractor with grok of the message field (have to improve it but is working as expected most of the cases) and i even created another strem to receive all the logs that come from the Filebeat. My problem is when i want to create a pipeline to remove the fields that filebeat add.
I created a pipeline that get logs from the strem that i mentioned, and in the pipeline there is a single stage (0) with a unique remove fields rule. This is the rule:

rule "function removeFields"
when
    has_field("beats_type")
then
    remove_field("@metadata_beat");
    remove_field("@metadata_type");
    remove_field("@@metadata_version");
    remove_field("agent_ephemeral_id");
    remove_field("agent_id");
    remove_field("agent_version");
    remove_field("agent_type");
end

The problem is that graylog is not removing the fields. What could be the problem?
I went to configuration and in the " Message Processors Configuration" i changed the order to Message Filter Chain -> Pipeline Processor and disabled the other 2 (AWS Instance Name Lookup and GeoIP Resolver).
I wanted to try the rule but in the simulation tab i have to put a raw message but i have no idea how to get a message in its raw state.
Any help will be appreciated.

Use the debug() function in your pipeline and watch the graylog logs for the results.

https://docs.graylog.org/en/latest/pages/pipelines/functions.html#debug

tail -f /var/log/graylog-server/server.log

Thanks for the tip. I added the debug

rule “function removeFields”
when
has_field(“beats_type”)
then
debug(“Debug”);
end

And apparently is entering the then statement (which should have the remove fields orders).
This is the graylog-server log

2020-02-13T13:54:47.747-03:00 INFO [Function] PIPELINE DEBUG: Debug
2020-02-13T13:54:47.750-03:00 INFO [Function] PIPELINE DEBUG: Debug
2020-02-13T13:54:47.752-03:00 INFO [Function] PIPELINE DEBUG: Debug
2020-02-13T13:54:47.752-03:00 INFO [Function] PIPELINE DEBUG: Debug
2020-02-13T13:54:47.749-03:00 INFO [Function] PIPELINE DEBUG: Debug
2020-02-13T13:54:47.748-03:00 INFO [Function] PIPELINE DEBUG: Debug
2020-02-13T13:54:47.757-03:00 INFO [Function] PIPELINE DEBUG: Debug
2020-02-13T13:54:47.751-03:00 INFO [Function] PIPELINE DEBUG: Debug

So, as far as i know the function is actually working, yet the fields remain.
Any other idea?

Wait, nevermind, apparently is working. Must have seen wrong when checking. Still the debug function is pretty neat, thanks for the help!

1 Like

check the processing order Sytstem > configuration you should have the processing pipeline after the message filter chain.

Is configured like that right now. Thanks for the response @jan