here is an scenario in my environment:
Logs from Palo Alto come in as type SYSTEM or TRAFFIC (there are more but lets stick with these two). I want to break out the fields in the two types of logs but they have different fields. To apply the relevant rule to the fields, first I have a rule that pulls out the type and creates a log_type field for it so it would be either log_type:SYSTEM or log_type:TRAFFIC. This is placed in Stage 0 that is set up to move to Stage 1 if any rules in Stage 0 are “true” (rule passes when section and is executed)
rule "PA-Firewall - ex0 - set log type"
regex(pattern: "(,TRAFFIC,|,THREAT,|,CONFIG,|,SYSTEM,)", value: to_string($message.message)).matches == true
let splitlog = split(",", to_string($message.message));
set_field("log_type", splitlog );
Now that I have marked the message for SYSTEM or TRAFFIC, in Stage 1 I can break the fields out based on the log_type field.
rule "PA-Firewall - ex1 - SYSTEM fields"
to_string($message.log_type) == "SYSTEM"
let message = to_string($message.message);
// Regex breaks out event description in quotes here.
// there are possibly commas in the description that messes up the split
// so regex the event description and ignore ending fields since not needed.
let snagy = regex(pattern: "(?<=,\")(.*)(?=.\",)",
let splitsys = split(",", message);
//set_field("log_type", splitsys); -- already handled
//set_field("virtual_system", splitsys); -- not used
This is just a pipeline for FW logs - I have a separate pipeline that handles messages coming in from the Windows machines and parses out event id’s before putting them into their own stream/index.
Hopefully that helps…