For security reasons, I changed some data from the log before posting here, but the thing is: The url field is extracted correctly, but the value inside it is also extracted as a field.
The “subfield” was extracted, but it is not supposed to be.
How can I avoid this? I considered creating all fields on raw, instead of syslog input, but it’s a lot of work. Is there a way using pipelines or another thing?
How does the Syslog Input treat the message? I cannot understand why it extracted something inside another field. I think it’s because the equal sign, but it’s not clear why.
rule "Remove field"
when
has_field("some_field")
then
// added debug message to be notified about the removed fields
debug( concat("dropped field from ", to_string($message.source)));
remove_field("some_field");
end
The problem is that I don’t know the field name. It always create a new one, with the parameters in the URL. Sometimes it’s more verbose, sometimes it’s not.
I was trying to extract the value from the field URL and split it until I have the possible “subfield”, then use that string to drop the field with remove_field(). But it’s hard to do that, because sometimes it splits the value based not only on the “=” to separate key and value for the new field, but it also use “-”, “\” and “?” to define the “subfield” name (key). These are just few of what I found out. Maybe there is more characters.
Note that it split the subfield name (subfield key) based on the “?”.
I also tried to just replace the “=” with “&eq”, because URLs are treated like that sometimes, but it doesn’t change anything, because the field is already created.
If you have any idea on how to split the URL value and find the possible subfield, it would help a lot. Then I could use the remove_field() function.
My applogies I dont have access to my GL server rn, but a while back there was a post of something similar not sure if it will help.
EDIT: You could create a field “url” and then drop it if you real dont want it. What I did was create fields I needed then droped the rest.
here are some more good examples of regex in pipeline.
What is extracting the message into fields, there are ways to fix this but you need to do it before it’s extracted.
Because there can be all kinds of weird characters in that url field, and it’s not escaped with quotes or anything. It’s probably the equals sign.
What you need to do in cases like this is use regex to get that value first set it to a field manually, and remove it from the message. Then you can use the key value function to break up the rest easily.
