Creating a log extractor for "key: value, key: value" logs

[amirmt71]

Hi everyone. I am trying to make an extractor to extract a very long log message with this format:
"value: key: value, key: value, …"

For example: " %FTD-1-430002: EventPriority: Low, DeviceUUID: xxxx-xxxx-xxx, InstanceID: 1, etc "

The problem is that the log is very long, having multiple key and values and it is not similar to any popular log types like JSON or something. Therefore, it takes a lot of processing power if I create a regular expression or try to extract each key with one extractor.
I actually did that, and made a regex for every key, it ended to having 20 or so extractors, but then my server was not able to process such heavy extractors and it overloaded.

So, what is your suggestion for making an extractor for this type of log message?

I appreciate your reply.

[shoothub]

Hi @amirmt71
try this one pipeline rule with key value function:

rule "Cisco FirePower KV"
when
  has_field("cisco_msg")
then
  set_fields(
    fields:
      key_value(
        value: to_string(to_string($message.cisco_msg)),
        delimiters:",",
        kv_delimiters:":"
  ));
end

cisco_msg is expected field which contains only key: value string extracted from message

[amirmt71]

Thank you @shoothub, I think that’s correct, but would you please let me know how I could remove the " %FTD-1-430002:" section before the rest?

[shoothub]

I’ve checked cisco docs, and found, that key value parameters uses only some message IDs: 430001, 43002, 430003, 430004 and 430005.

Cisco Firepower Threat Defense Syslog Messages - Security Event Syslog...

So best way would be probably to extract message ID and update pipeline rule for KV only for there messages.

1. Create pipeline rule to extract severity and message id:

rule "Cisco FirePower grok"
when
  contains(to_string($message.message), "%FTD", true)
then
 set_fields(grok(pattern: "\\%FTD-%{DATA:ftd_severity:int}-%{DATA:ftd_messageid}: %{GREEDYDATA:cisco_msg}", value: to_string($message.message), only_named_captures: true));  
end

2. Create pipeline rule KV only for message id 430001-430005

rule "Cisco FirePower KV"
when
  has_field("cisco_msg") AND (to_long($message.ftd_messageid) >= 430001 AND to_long($message.ftd_messageid) <= 430005)
then
  set_fields(
    fields:
      key_value(
        value: to_string(to_string($message.cisco_msg)),
        delimiters:",",
        kv_delimiters:":"
  ));
end

Create pipeline and 2 steps, on step 0 setup pipeline rule “Cisco FirePower grok” and for step 2: “Cisco FirePower KV”

[amirmt71]

Thank you very much @shoothub. It greatly worked, processing all the messages with lightning speed. (❁´◡`❁)