Illuminate for Windows logs keeps winlogbeat_ fields

[Hashcab]

1. Describe your incident
   Hello, in our environment we recently began collecting workstation logs using Graylog Sidecar+winlogbeat. One annoying thing about winlogbeat is that it adds the “winlogbeat_*” field to every message and the information is often repetitive and unnecessary. I thought that by enabling the Illuminate pack for Windows logs, these fields would be removed, instead these fields are kept but with even more repetitive info from GIM.

2. Environment details
   Graylog 5.2.3 with Security module. Winlogbeat 7.11.1 on Windows 10 host that is the Windows event collector for the network.

3. What steps have you already taken to try and solve the problem?
   I have tried to drop the fields that I do not want with the following part of my configuration file:

winlogbeat:
  event_logs:
   - name: ForwardedEvents
     forwarded: true
     ignore_older: 96h
  processors:
    - drop_fields:
        when:
          regexp:
            fields: ['winlogbeat_.*']
        ignore_missing: true

I am currently using this in the configuration file and the fields are still not dropped. I have also tried explicitly defining each field with “winlogbeat_@timestamp”, “winlogbeat_agent_ephemeral_id”, etc… but still no luck.

4. How can the community help?

First off, do you all keep these fields? Some of them look to be helpful but others are the exact data as a GIM field or a field from the event data.

If the solution is to just drop the fields as I have tried, what am I doing wrong with the configuration file?

Thanks for help!

[Joel_Duffield]

You will want to drop them in a pipeline after illuminate runs in Graylog. The easiest way to do that would be with the remove_field function which supports regex. Now if you want to keep a few maybe rename those first in the rule and then run the remove.

[Hashcab]

That would do the trick… I was hesitant to use this method because it may cause more processing on the Graylog end of things. Now that I think of it… I’d much rather more processing on Graylog’s end than my winlogbeat hosts. Thanks for the reply!