Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question. Don’t forget to select tags to help index your topic!
1. Describe your incident:
I have already implemented pipeline to drop message which contains specific words but I am looking for some example or help that how can I drop words from incoming log.
Let say I want to remove “I want to” from message “This is the word I want to delete and more words” and
I want to get only “This is the word delete and more words”
I want to keep my message that comes from FortiGate and I am interested in Src IP , Dst IP, Type of service or protocol, policy action if its allowed or denied and dont want to keep duration, sequence number and those kinds of information.
Will this help in reducing the size of daily logs as well?
2. Describe your environment:
OS Information:
Ubuntu 18.04
Package Version:
4.1
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
If you want to drop fields from a message coming in before it is stored you can use remove_field() if you are looking to remove words from a string you can use replace() and just replace the string portion with something like this:
set_field(to_string($message.some_field), replace(to_string($message.some_field), "I want to", ""));
depending on the volume it may or may not help reduce your message load.
Just a suggestion. You should be able to configure FortiGate depending on your model/version to prevent certain logs form being sent. This may help reduce the amount of log/s being ingested.
Not sure what Model/Version you have so here are a couple link for you.
Does this make sense?
I have few words in log like vd=“root” and I want to remove this.
rule “function Fortigate Fields”
when
has_field(“vd=root”)
then
remove_field(“vd=root”);
end
Is it a field name of “vd=root” or is it that the “vd” field contains the data “root”. I don’t think you can have an ‘=’ in a field name so it must be the latter? in which case your rule would look more like this: (Note: I am using the forum formatting tool </> to make the code look readable…)
rule "function remove Fortigate Fields"
when
$message.vd == "root"
then
remove_field("vd");
end
I try to keep the amount of fields low to save disk space so I only selected a few extractors that I needed.
Here is a list some regex. You can test these on your INPUT to see if they work for you. Just a reminder I’m using Raw/Plaintext UDP I just started learning regex on Graylog so these may be a little sloppy.
I understand now. Understanding your setup the only way I know of at this point would be what @tmacgbay suggested by using a pipeline/s. Sorry I cant be more help.
What did you try? It is still not clear what you have and what you want to do with it. Can you give an obfuscated and nicely formatted (using the forum tool </>) real example of:
The message you want to modify
The fields that message has been broken out into
What you want the results to look like
If you set your extractors to NOT leave the original message intact and only extract what you want, I think you would end up with only those fields and a leftover messy $message.message that you could then modify into something you want. @gsmith would confirm it could work that way…
I tried this in the pipeline but still I can see vd in the logs and also devid.
rule “function remove Fortigate Fields”
when
has_field(“devname”)
then
remove_field(“vd”);
end
and Also
rule “function remove Fortigate Fields”
when
has_field(“devname”)
then
set_field(to_string($message.some_field), replace(to_string($message.some_field), “devid=”, “”));
end
