Working with Pipelines

Graylog Central (peer support)
4

here is an scenario in my environment:

Logs from Palo Alto come in as type SYSTEM or TRAFFIC (there are more but lets stick with these two). I want to break out the fields in the two types of logs but they have different fields. To apply the relevant rule to the fields, first I have a rule that pulls out the type and creates a log_type field for it so it would be either log_type:SYSTEM or log_type:TRAFFIC. This is placed in Stage 0 that is set up to move to Stage 1 if any rules in Stage 0 are “true” (rule passes when section and is executed)

rule "PA-Firewall - ex0 - set log type"
when
    regex(pattern: "(,TRAFFIC,|,THREAT,|,CONFIG,|,SYSTEM,)", value: to_string($message.message)).matches == true
then
    let splitlog = split(",", to_string($message.message));
    set_field("log_type", splitlog[3] );

 end

Now that I have marked the message for SYSTEM or TRAFFIC, in Stage 1 I can break the fields out based on the log_type field.

rule "PA-Firewall - ex1 - SYSTEM fields"
when
    has_field("log_type")           &&
    to_string($message.log_type) == "SYSTEM"
then
    let message     = to_string($message.message);

    // Regex breaks out event description in quotes here.
    // there are possibly commas in the description that messes up the split
    // so regex the event description and ignore ending fields since not needed.
    let snagy       = regex(pattern:    "(?<=,\")(.*)(?=.\",)", 
                            value:      message
                    );

    set_field("event_description", to_string(snagy["0"]));

	let splitsys   = split(",", message);
    set_field("hostname",                   splitsys[0]);
    set_field("receive_date_time",          splitsys[1]);
    set_field("serial_number",              splitsys[2]);
  //set_field("log_type",                   splitsys[3]); -- already handled
    set_field("log_subtype",				splitsys[4]);
    set_field("time_generated",				splitsys[6]);
  //set_field("virtual_system",				splitsys[7]); -- not used
    set_field("event_id_name",				splitsys[8]);
    set_field("session_object",				splitsys[9]);
    set_field("subtype_module",				splitsys[12]);
    set_field("event_severity",				splitsys[13]);
end

This is just a pipeline for FW logs - I have a separate pipeline that handles messages coming in from the Windows machines and parses out event id’s before putting them into their own stream/index.

Hopefully that helps…

1 Like