Drop fields using wildcards?

[ma645]

I’m collecting dns logs from packetbeat on a dns windows server and really often I receive the 1000 filed limit error on one custom index, I have an output to forward these messages to another system. I would drop all unnecessary fields like packetbeat_dns_additionals_N_type where ‘N’ is a increasing number.
I have a rule that is dropping some fields but I don’t know if it possible to drop all messages with a wildcard like this:

rule “function removeFields”
when
has_field(“beats_type”)
then
remove_field(“packetbeat_dns_additionals_*”);
end

or there’s any way to do so ?

Thanks !

[shoothub]

I don’t think you can use wildcard for this function. You need to remove it one by one.
You can also increase limit of 1000 fields, check this blog post:
https://www.graylog.org/post/what-to-do-when-you-have-1000-fields

[ma645]

Hi shoothub,

I’ve checked to increase the 1000 fields limit but I tought to filter out fields because I need only few,
there’s a way to drop all field and index only the few needed for my purpose ?

Thanks

[shoothub]

Best way will be probably to filter only required fields on paketbeat side, check this:
https://www.elastic.co/guide/en/beats/packetbeat/master/filtering-and-enhancing-data.html

[ma645]

This is what I’m looking for, even the 1000 field limit error is gone since I’ve dropped unnecessary fields directly on packetbeat configuration.

Thanks !