Hi! We have been using graylog for a long time already but only to storage system logs from Linux servers. I wanted to use it for more than that, mainly to get more information of our service logs (Apache, BIND, Haproxy, etc.) and create dashboards and alerts. So far i created a Test Graylog server with the lastest version (3.2) on Debian 10. I am receiving messages from 2 different servers, in both cases i installed the sidecar and i am using Filebeat to collect them. I also created an extractor with grok of the message field (have to improve it but is working as expected most of the cases) and i even created another strem to receive all the logs that come from the Filebeat. My problem is when i want to create a pipeline to remove the fields that filebeat add.
I created a pipeline that get logs from the strem that i mentioned, and in the pipeline there is a single stage (0) with a unique remove fields rule. This is the rule:
rule "function removeFields"
when
has_field("beats_type")
then
remove_field("@metadata_beat");
remove_field("@metadata_type");
remove_field("@@metadata_version");
remove_field("agent_ephemeral_id");
remove_field("agent_id");
remove_field("agent_version");
remove_field("agent_type");
end
The problem is that graylog is not removing the fields. What could be the problem?
I went to configuration and in the " Message Processors Configuration" i changed the order to Message Filter Chain -> Pipeline Processor and disabled the other 2 (AWS Instance Name Lookup and GeoIP Resolver).
I wanted to try the rule but in the simulation tab i have to put a raw message but i have no idea how to get a message in its raw state.
Any help will be appreciated.
