Hello, logs from our apllications comes in Json format, and very often this json contain fields, that i don’t need to index. For exampe payload_session_123123, where 123123 is userid. Because of this fields i’ve run into limits in 1000 fileds to index in elasticsearch.
Ok, so i don’t need to index such fields and i just want to remove them after JSON extractor done it’s work.
What is the best way to do it ? I try to use Pipelines, but can’t find corresponding function, that allows me to delete fields by regexp.
Thanks a lot
The short is you can combine pipeline functions as such:
remove_field(to_string(regex(".[A-Za-z0-9][A-Za-z0-9-.]*\\.\\D{2,4}", to_string($message.the_json))));
But it would be way more efficient to extract only the things you want and ignore the rest…
Is it possible? I use JSON Extractor, and can’t find sutable options. The only option in this extractor is “Only attempt extraction if field matches regular expression”, but it seems, that it not check every fieldname, but through all of thme
Honestly, I don’t use extractors. I do all of the work I need in the pipeline… it works better for me in visualizing the message path and changes thereof. There are examples in the forums where other have asked how to pull distinct information from JSON with pipeline rules. It’s not that hard, particularly if you know the fields you want.
Thank’s a lot, in my case i only know fields that i don’t need ^)
In that case you I think you are stuck with grabbing them all then using regex to remove the ones you don’t want. 
1 Like
Thanks for show me the way, will try
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.