Thanks so much for your advice about using streams to send each log source to it’s own Elasticsearch index. That is a far better way to set this up and is helping me at least isolate the problem. Graylog is deep, I have a lot to learn XD.
To answer your question:
I am using the ‘replace with regex’ extractor to replace commas with spaces along with the key=value converter to create the fields for our WAF logs. (We have other sources which use normal extractors.) We did this for convenience and because it performed far better than using GROK/Regex to weed thru these enormous log entries and extract 20-30 fields (which is how we started out).
Our WAF which handles a metric ton of attacks and I think sometimes the key=value converter is taking arbitrary junk variables the attackers throw at us and turning them into fields. Otherwise it’s in some cases interpreting anything around an ‘=’ sign, even if there is no space delimiter, as a key=value pair and extracting it?
Given what’s happening do I have a choice aside from building a detailed extractor for just the fields we want? I can’t throw the rest of the message away because we need it for forensics.