Drop field with random name

Hi everyone,

I’m receiving Fortigate logs using Syslog Input and it is extracting all fields from the message. That’s good.

But there is a field that should not exist at all.

Here is an example:

date=2024-02-28 time=11:24:21 devname=“xxx” devid=“xxx” eventtime=1709130261291966360 tz=“-0300” logid=“1059028704” type=“utm” subtype=“app-ctrl” eventtype=“signature” level=“information” vd=“root” appid=38131 user=“xxx” group=“xxx” srcip= srccountry=“Reserved” dstip=x.x.x.x dstcountry=“United States” srcport=57336 dstport=443 srcintf=“port1” srcintfrole=“lan” dstintf=“port1” dstintfrole=“lan” proto=6 service=“HTTPS” direction=“outgoing” policyid=5 sessionid=2037666607 applist=“xxx” action=“pass” appcat=“General.Interest” app=“Google.Accounts” hostname=“lh3.googleusercontent.com” incidentserialno=440754503 url=“/a-/ALV-UjUNeRFL8hvhQJAqqTSASIDJdj89182SDA3UVmDlH4lBWVkbnB1gI8VLpDXoK1o=s48-p-k-no-mo” msg=“General.Interest: Google.Accounts” apprisk=“elevated”

For security reasons, I changed some data from the log before posting here, but the thing is: The url field is extracted correctly, but the value inside it is also extracted as a field.

field: url
value: /a-/ALV-UjUNeRFL8hvhQJAqqTSASIDJdj89182SDA3UVmDlH4lBWVkbnB1gI8VLpDXoK1o=s48-p-k-no-mo

field: /a-/ALV-UjUNeRFL8hvhQJAqqTSASIDJdj89182SDA3UVmDlH4lBWVkbnB1gI8VLpDXoK1o
value: s48-p-k-no-mo

The “subfield” was extracted, but it is not supposed to be.

How can I avoid this? I considered creating all fields on raw, instead of syslog input, but it’s a lot of work. Is there a way using pipelines or another thing?

How does the Syslog Input treat the message? I cannot understand why it extracted something inside another field. I think it’s because the equal sign, but it’s not clear why.

Hey @eron.medeiros

You could use a pipeline and drop tht field.

Something like this.

rule "Remove field"
     // added debug message to be notified about the removed fields
    debug( concat("dropped field from ", to_string($message.source)));

The problem is that I don’t know the field name. It always create a new one, with the parameters in the URL. Sometimes it’s more verbose, sometimes it’s not.

I was trying to extract the value from the field URL and split it until I have the possible “subfield”, then use that string to drop the field with remove_field(). But it’s hard to do that, because sometimes it splits the value based not only on the “=” to separate key and value for the new field, but it also use “-”, “\” and “?” to define the “subfield” name (key). These are just few of what I found out. Maybe there is more characters.

For example:

key: url
value: /gps-proxy/ALd4DhH8KKbIvfD9ZvBUwheodEwWoWQ3R6eVN83AITkSMq-ur3DdvCdGIXd96a7Ii0V9zkcL16M4bkYbF11adnJpdtsrI8abdCWAtlimqGELjZjZlCBYSacV0pmLZ6N6uVR1teVaqoaTaDJ7lq3Ni80RKc8zQNK529ItJNJZy8l2h29pyHiUXH9eBhBE__yLYA=s436-k-no

create this:

key: ur3DdvCdGIXd96a7Ii0V9zkcL16M4bkYbF11adnJpdtsrI8abdCWAtlimqGELjZjZlCBYSacV0pmLZ6N6uVR1teVaqoaTaDJ7lq3Ni80RKc8zQNK529ItJNJZy8l2h29pyHiUXH9eBhBE__yLYA
value: s436-k-no

Note that it split the field name (subfield key) based on the “-”. It took everything before “-” out.

Another example:

key: url
value: /!api/2.0/repositories/%7B%7D/%8e46-4178-8a29-e3ae822823a9%7D?fields=uuid%2Cname%2Cworkspace.name%2Clinks.html.href%2Clinks.avatar.href%2Cfull_name%2Clanding_page%2Cfork_policy%2Cworkspace.slug%2Cowner.uuid

create this:

key: fields
value: uuid%2Cname%2Cworkspace.name%2Clinks.html.href%2Clinks.avatar.href%2Cfull_name%2Clanding_page%2Cfork_policy%2Cworkspace.slug%2Cowner.uuid

Note that it split the subfield name (subfield key) based on the “?”.

I also tried to just replace the “=” with “&eq”, because URLs are treated like that sometimes, but it doesn’t change anything, because the field is already created.

If you have any idea on how to split the URL value and find the possible subfield, it would help a lot. Then I could use the remove_field() function.

Hey @eron.medeiros

My applogies I dont have access to my GL server rn, but a while back there was a post of something similar not sure if it will help.

EDIT: You could create a field “url” and then drop it if you real dont want it. What I did was create fields I needed then droped the rest.
here are some more good examples of regex in pipeline.

What is extracting the message into fields, there are ways to fix this but you need to do it before it’s extracted.

Because there can be all kinds of weird characters in that url field, and it’s not escaped with quotes or anything. It’s probably the equals sign.

What you need to do in cases like this is use regex to get that value first set it to a field manually, and remove it from the message. Then you can use the key value function to break up the rest easily.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.