[collector][filebeat] lines to exclude using multiple regexp

#1

Hello,

I’m trying to use multiple regexp to exclude lines from logs sent by collector/filebeat.
In the web interface, I entered regexps in the format:
[’.test.domain.foo.bar.’, ‘.test.domain.foo.com.’, …]

In the resulting generated Filebeat log, I have my regexp under lines to exclude w/ one line per regexp (lines are dns.json from suricata):

Problem is that matching lines still get sent to Graylog.

Anyone with an idea or exact syntax to use it?

(marius) #2

Hi,
your syntax looks fine to me. Maybe the pattern doesn’t match the log messages you want to exclude?

I would try it with ['.*test.domain.foo.bar.*'] instead. Further readings can be found here:

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#exclude-lines

and

https://www.elastic.co/guide/en/beats/filebeat/current/regexp-support.html

Cheers,
Marius

#3

Hi Marius,

Just made a copy/paste error when posting my initial message, didn’t saw that your platform is escaping them :slight_smile: My regexp are indeed in the form like you propose:

  • .*test\.domain-1\.foo\.bar.*

Validated then in Go against full message and they are correct. Just upgraded to collector 0.1.1, will try to see if it changes anything.

#4

Ok, problem is in fact only on domains with dash ‘-’ in them that do not get filtered using:

  • .*test.domain-1.foo.bar.*
  • .*test.domain\-1.foo.bar.*
  • .*test.domain.*foo.bar.*