Correct use of remove_field in pipelines

Graylog Central (peer support)
1

Hi,

Thanks for the help you have provided me, I appreciate too much your guidance. Currently, I moved to connect filebeat directly to graylog using Beats Input. I used to remove some fields coming in filebeat in logstash configuration as follows:

mutate {
            remove_field => [ "[prospector][type]","[host][architecture]", "[host][containerized]", "[host][id]", "[host][os][platform]", "[host][os][family]","[beat][version]", "[offset]", "[input][type]", "[meta][cloud][provider]", "[meta][cloud][machine_type]", "[meta][cloud][instance_id]", "[meta][cloud][region]", "[meta][cloud][instance_name]", "[source]"]

        }

In graylog, I learned to use pipelines to remove any field. Then, I created a rule as follows to remove the fields:

rule "function removeFields"
when
    has_field("beats_type")
then
    remove_field("beats_type");
    remove_field("filebeat_@metadata_beat");
    remove_field("filebeat_@metadata_type");
    remove_field("filebeat_@metadata_version");
    remove_field("filebeat_@timestamp");
    remove_field("filebeat_beat_hostname");
    remove_field("filebeat_beat_name");
    remove_field("filebeat_host_architecture");
    remove_field("filebeat_host_containerized");
    remove_field("filebeat_host_id");
    remove_field("filebeat_host_name");
    remove_field("filebeat_host_os_codename");
    remove_field("filebeat_host_os_family");
    remove_field("filebeat_host_os_name");
    remove_field("filebeat_host_os_platform");
    remove_field("filebeat_host_os_version");
    remove_field("filebeat_input_type");
    remove_field("filebeat_meta_cloud_instance_id");
    remove_field("filebeat_log_file_path");
    remove_field("filebeat_meta_cloud_instance_name");
    remove_field("filebeat_meta_cloud_machine_type");
    remove_field("filebeat_meta_cloud_provider");
    remove_field("filebeat_meta_cloud_region");
    remove_field("filebeat_prospector_type");
end

I followed the next steps to test the change:
Stop filebeat, graylog, mongodb
Reset offset in registry
Delete graylog_0 index:

curl -XDELETE localhost:9200/graylog_0

Start mongodb, graylog, filebeat

However, I still having the fields in the search:
image

How to get rid of the filebeat fields?

Thanks a lot for your help

2

make sure your pipeline is after your extractor in the process chain, also make sure your pipeline rule is processing after the fields are created

Your rule is correct.

3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.