Fortimail "msg" field pipeline fix

Graylog Central (peer support)
,
1

Hey,

im struggling already since a few days to get this fixed but without success… i collect logs from our mail gateway Fortimail and have some issues with fixing the “msg” field inside of it.

Raw messages look like this:

date=2021-07-25 time=15:25:22.425 device_id=HOSTNAME log_id=0003002326 type=event subtype=smtp pri=information user=mail ui=mail action=NONE status=N/A session_id=“16PDPGVH002324-16PDPGVJ002324” msg=“STARTTLS=client, cert-subject=/C=CH/ST=GE/L=Plan-les-Ouates/O=Proton Technologies AG/CN=protonmail.com, cert-issuer=/C=CH/O=SwissSign AG/CN=SwissSign Server Gold CA 2014 - G22, verifymsg=unable to get local issuer certificate”

image
image1325×529 16.9 KB

…and as you can already see and notice is that the “msg” field is kind of unusable like this… so ive tried to send it through a pipeline to fix this… first i wanted to just remove the “msg” field, also remove the dashes between the subfields but the most im struggling with are “cert-subject” and “cert-issuer” with its content.
not only are “-” horrible because it means “substract” which actually means again that “cert-subject” will be listed as “subject” as a field but also fixing the content inside is quite tricky.

Pipeline looks like this right now (doesn’t work):

rule "Msg Fortimail"
when
    has_field("msg") && has_field("device_id")
then
    regex_replace("msg=", to_string($message.message), "");
    regex_replace(",", to_string($message.message), "", true);
    regex_replace("cert-issuer", to_string($message.message), "certissuer");
    regex_replace("cert-subject", to_string($message.message), "certsubject");
end

ive also search a lot on google and also here but couldn’t find anything helpful yet… maybe the one or other was already dealing with Fortimail logs and could help here… thanks a lot in advance!!

2

I have not workwith fortimail. But came accross similar situation recently. I have used message filter chains to extract IPV4 address out of the message.if you have list of messages just right click on the required field and assign into new field.

Thereafter you will be able to apply in in your pipeline rules. Put a debug message in your pipeline rule and check whether you are getting the exracted value.

3

but the message will already be split without any additional action from myside… just wrong because of the minus between cert and subject or cert and issuer.
the other issue im fighting with is also to disallow fake fields inside of some fields content… for example when you have:

url=https://whatever.com/istrue=yes

then you will get an additional field with “istrue” and its content “yes”… at more complex urls this is kind of not cool because they are spamming your index list and it will become really fast full = graylog web search = slow.
never the less thanks for your answer and help! ill focus on the debug function the next days.

debug output of the pipeline mentioned in the first post looks like this:

image
image1900×259 32.7 KB

somehow it is working but shouldnt the last message contain all changes?

4

Hi @0x3e4
That’s a known problem with Key value extractor, I’ve also have this problem in the past for fortigate logs, if there was a = inside one value, it was extracted as new field. I’ve ended up with log of unknown field. So better way is probably to extract only required field using regex or similar technique.

Graylog by default doesn’t extract Key value values. Just check, if you didn’t put KV extractor in one of the extractors in the Input that receive logs from FortiMail.

One I’ve also experienced similar problem in version 3.3, when KV values were extracted automatically without KV extractor in input, but it was probably KV extractors stuck somewhere in the input after I’ve removed it.

1 Like
5

well… i think that the “syslog” input does it on default because i only use extractors from one of the marketplace’s content pack which are related to our Fortigate logs but maybe then it would be better to make a “raw” input for that and generate the fields with own extractors by myself… maybe this is also the solution for my Fortimail “issue” haha
thanks for the help and reply!

7

im going with the way to log with a “raw” input instead of the “syslog” one because you can handle the messages easier with the extractors in my mind and so the issue with the default creatings of fields inside of urls/certissuer/certsubject will be prevented.
case closed… thanks for the hints and comments fam!

ps: ill update the thread as soon as ive configured and finished my extractors for the Fortimail logs.

edit: FortiMail extractors for raw input work in progress:
edit2: i couldnt update my last post so ive deleted it to make an update to the FortiMail extractors here again.

{
  "extractors": [
    {
      "title": "FMLdevice_id",
      "extractor_type": "regex",
      "converters": [],
      "order": 102,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "device_id",
      "extractor_config": {
        "regex_value": "^.*device_id=([\\w\\s]+)\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLlog_id",
      "extractor_type": "regex",
      "converters": [],
      "order": 103,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "log_id",
      "extractor_config": {
        "regex_value": "^.*log_id=([\\w\\s]+)\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLpri",
      "extractor_type": "regex",
      "converters": [],
      "order": 104,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "pri",
      "extractor_config": {
        "regex_value": "^.*\\spri=([\\w\\s]+)\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLuser",
      "extractor_type": "regex",
      "converters": [],
      "order": 105,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "user",
      "extractor_config": {
        "regex_value": "^.*\\suser=([\\w\\s]+)\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLui",
      "extractor_type": "regex",
      "converters": [],
      "order": 106,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "ui",
      "extractor_config": {
        "regex_value": "^.*\\sui=([\\w\\s]+)\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLsession_id",
      "extractor_type": "regex",
      "converters": [],
      "order": 107,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "session_id",
      "extractor_config": {
        "regex_value": "^.*\\ssession_id=\\\"([\\w\\s\\-]+)\\\"\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLSTARTTLS",
      "extractor_type": "regex",
      "converters": [],
      "order": 108,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "STARTTLS",
      "extractor_config": {
        "regex_value": "^.*STARTTLS=([\\w\\s]+)\\,+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLfrom",
      "extractor_type": "regex",
      "converters": [],
      "order": 109,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "from",
      "extractor_config": {
        "regex_value": "^.*\\sfrom=\\\"([a-zA-Z\\w\\s\\:\\-.\\.\\_\\@]+)\\\"\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLhfrom",
      "extractor_type": "regex",
      "converters": [],
      "order": 110,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "hfrom",
      "extractor_config": {
        "regex_value": "^.*\\shfrom=\\\"([a-zA-Z\\w\\s\\:\\-.\\.\\_\\@]+)\\\"\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLto",
      "extractor_type": "regex",
      "converters": [],
      "order": 111,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "to",
      "extractor_config": {
        "regex_value": "^.*\\sto=\\\"([a-zA-Z\\w\\s\\:\\-.\\.\\_\\@]+)\\\"\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLvirus",
      "extractor_type": "regex",
      "converters": [],
      "order": 112,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "virus",
      "extractor_config": {
        "regex_value": "^.*\\svirus=([a-zA-Z\\w\\s\\:\\\"\\-.\\.\\_\\@]+)\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLdisposition",
      "extractor_type": "regex",
      "converters": [],
      "order": 113,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "disposition",
      "extractor_config": {
        "regex_value": "^.*\\sdisposition=\\\"([a-zA-Z\\w\\s\\:\\\"\\-.\\.\\_\\@]+)\\\"\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLclient_ip",
      "extractor_type": "regex",
      "converters": [],
      "order": 114,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "client_ip",
      "extractor_config": {
        "regex_value": "^.*\\sclient_ip=\\\"([\\w\\.]+)\\\"\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLdst_ip",
      "extractor_type": "regex",
      "converters": [],
      "order": 115,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "dst_ip",
      "extractor_config": {
        "regex_value": "^.*\\sdst_ip=\\\"([\\w\\.]+)\\\"\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLdsn",
      "extractor_type": "regex",
      "converters": [],
      "order": 116,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "dsn",
      "extractor_config": {
        "regex_value": "^.*\\sdsn=([\\w\\.]+)\\,+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLrelay",
      "extractor_type": "regex",
      "converters": [],
      "order": 117,
      "cursor_strategy": "copy",
      "source_field": "msg",
      "target_field": "relay",
      "extractor_config": {
        "regex_value": "^.*\\srelay=([\\w\\.]+)\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTunauthuser",
      "extractor_type": "regex",
      "converters": [],
      "order": 90,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "unauthuser",
      "extractor_config": {
        "regex_value": "^.*unauthuser=\\\"([\\$\\.\\-\\w\\s]+)\\\"\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTcrlevel",
      "extractor_type": "regex",
      "converters": [],
      "order": 91,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "crlevel",
      "extractor_config": {
        "regex_value": "^.*crlevel=\\\"([\\w\\s]+)\\\"+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTcrscore",
      "extractor_type": "regex",
      "converters": [],
      "order": 92,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "crscore",
      "extractor_config": {
        "regex_value": "^.*crscore=([\\w\\s]+)\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTsource",
      "extractor_type": "regex",
      "converters": [],
      "order": 93,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "source",
      "extractor_config": {
        "regex_value": "^.+\\sdevname=\\\"(\\S+)\\\"\\s"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLclassifier",
      "extractor_type": "regex",
      "converters": [],
      "order": 120,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "classifier",
      "extractor_config": {
        "regex_value": "^.*\\sclassifier=\\\"([a-zA-Z\\w\\s\\:\\\"\\-.\\.\\_\\@]+)\\\"\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLto_dsn",
      "extractor_type": "regex",
      "converters": [],
      "order": 119,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "to",
      "extractor_config": {
        "regex_value": "^.*to=\\<([a-zA-Z\\w\\s\\:\\-.\\.\\_\\@]+)\\>\\,+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLto_dsn_multiple",
      "extractor_type": "regex",
      "converters": [],
      "order": 118,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "to",
      "extractor_config": {
        "regex_value": "^.*\\\"to=([a-zA-Z\\w\\s\\:\\-.\\.\\,\\_\\@]+)\\,\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTaction",
      "extractor_type": "regex",
      "converters": [],
      "order": 94,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "action",
      "extractor_config": {
        "regex_value": "^.+\\saction=\\\"(\\S+)\\\"\\s"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTlogid",
      "extractor_type": "regex",
      "converters": [],
      "order": 95,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "logid",
      "extractor_config": {
        "regex_value": "^.+\\slogid=\\\"(\\S+)\\\"\\s"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTdevid",
      "extractor_type": "regex",
      "converters": [],
      "order": 96,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "devid",
      "extractor_config": {
        "regex_value": "^.+\\sdevid=\\\"(\\S+)\\\"\\s"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTsubtype",
      "extractor_type": "regex",
      "converters": [],
      "order": 97,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "subtype",
      "extractor_config": {
        "regex_value": "^.+\\ssubtype=\\\"(\\S+)\\\"\\s"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTtransip",
      "extractor_type": "regex",
      "converters": [],
      "order": 98,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "transip",
      "extractor_config": {
        "regex_value": "^.+\\stransip=\\\"(\\S+)\\\"\\s"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTtype",
      "extractor_type": "regex",
      "converters": [],
      "order": 99,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "type",
      "extractor_config": {
        "regex_value": "^.+\\stype=\\\"(\\S+)\\\"\\s"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTtrandisp",
      "extractor_type": "regex",
      "converters": [],
      "order": 100,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "trandisp",
      "extractor_config": {
        "regex_value": "^.+\\strandisp=\\\"(\\S+)\\\"\\s"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FGTeventtype",
      "extractor_type": "regex",
      "converters": [],
      "order": 101,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "eventtype",
      "extractor_config": {
        "regex_value": "^.+\\seventtype=\\\"(\\S+)\\\"\\s"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLdirection",
      "extractor_type": "regex",
      "converters": [],
      "order": 121,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "direction",
      "extractor_config": {
        "regex_value": "^.+\\sdirection=\\\"(.+?)\\\""
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLmailer",
      "extractor_type": "regex",
      "converters": [],
      "order": 122,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "mailer",
      "extractor_config": {
        "regex_value": "^.+\\smailer=\\\"(.+?)\\\""
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLclient_cc",
      "extractor_type": "regex",
      "converters": [],
      "order": 123,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "client_cc",
      "extractor_config": {
        "regex_value": "^.+\\sclient_cc=\\\"(.+?)\\\""
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLpolid",
      "extractor_type": "regex",
      "converters": [],
      "order": 124,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "polid",
      "extractor_config": {
        "regex_value": "^.+\\spolid=\\\"(.+?)\\\""
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLsrc_type",
      "extractor_type": "regex",
      "converters": [],
      "order": 125,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "src_type",
      "extractor_config": {
        "regex_value": "^.+\\ssrc_type=\\\"(.+?)\\\""
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLmessage_length",
      "extractor_type": "regex",
      "converters": [],
      "order": 126,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "message_length",
      "extractor_config": {
        "regex_value": "^.+\\smessage_length=\\\"(.+?)\\\""
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLclient_name",
      "extractor_type": "regex",
      "converters": [],
      "order": 127,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "client_name",
      "extractor_config": {
        "regex_value": "^.*\\sclient_name=([a-zA-Z\\w\\s\\:\\\"\\-.\\.\\_\\@]+)\\s+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLstat",
      "extractor_type": "regex",
      "converters": [],
      "order": 129,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "stat",
      "extractor_config": {
        "regex_value": "^.*\\sstat=([a-zA-Z\\w\\s\\:\\-\\–\\_\\.\\,\\=\\/\\@\\|\\[\\]\\<\\>\\#\\ä\\ü\\ö\\(\\)\\Ä\\Ü\\Ö\\/\\;\\\"\\„\\“\\!\\&\\?]+)\\\"+"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "FMLsubject",
      "extractor_type": "regex",
      "converters": [],
      "order": 128,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "subject",
      "extractor_config": {
        "regex_value": "^.*\\ssubject=\\\"([\\d\\D]+)\\\"\\smessage_id"
      },
      "condition_type": "none",
      "condition_value": ""
    }
  ],
  "version": "4.1.2"
}
1 Like
8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.