Hi, I’m receiving large logs from a SOPHOS firewall of which I’m only interested in %20 of their content, so saving the whole log is taking so much storage, and as I’m already using extractors to give me the fields I’m interested in, Is there a way to strip the logs and only save specific fields to storage?
You can use the remove_field() function in a pipeline to drop the fields you don’t want. I have also used the regex_replace() function to remove standardized wording out of some of the larger $message.message results - I want the meat of the message but not the disclaimer that comes with with each one…
That should get you started - if you post more specific information (Obfuscated of course) I might be able to give you a more specific answer!
If you can work in extractors to only get the values you want out of the message, that would work - perhaps an well built GROK statement…either in extractors OR in the pipeline
You could leave the extractors in place and then drop the extra fields in the pipeline with the aforementioned functions… or you could drop the extractors all together and separate out your fields in the pipeline where you would only create the fields you want… Someone created a pipeline rule to do just that here It’s not necessarily efficient… but it works…
To be honest a pipeline /w regex probably be you best way to create all those fields, I use a lot REGEX extractors for those fields and noticed the resource consumption is greater then just using one pipeline.