I tried to be good and create a content pack but man that interface is just soooo wonky…
All I was trying to do is put out pipelines and their rules and extract the extra stuff I have in there… AND you have to get it 100% right because although I can edit all I want in github, once you post in Graylog Marketplace, the only thing you can change is the title (LAME!) Ok… done complaining.
I am posting my rules in sequence here. I am assuming your input works and you know how to make all the connections, stream, indexes and rules to connect them etc.
STAGE 0:
rule "PA-Firewall - ex0 - set log type"
when
regex(pattern: "(,TRAFFIC,|,THREAT,|,CONFIG,|,SYSTEM,)", value: to_string($message.message)).matches == true
then
let splitlog = split(",", to_string($message.message));
set_field("log_type", splitlog[3] );
end
– And because PA 9.1.1 breaks out GLOBALPROTECT logs to different log file (used to be in SYSTEM) AND they decided that field 5 should contain the log type rather than field 3 like SYSTEM and TRAFFIC do. Sigh. Here is the second rule to handle marking that.
rule "PA-Firewall - ex0 - set log type - GP"
when
regex(pattern: "(,GLOBALPROTECT,)", value: to_string($message.message)).matches == true
then
set_field("log_type", "GLOBALPROTECT");
end
Now that we have separated out the log types we can delve into breaking out fields - >.
STAGE 1
rule "PA-Firewall - ex1 - SYSTEM fields"
when
has_field("log_type") &&
to_string($message.log_type) == "SYSTEM"
then
let message = to_string($message.message);
// Regex breaks out event description in quotes here.
// there are possibly commas in the description that messes up the split
// so regex the event description and ignore ending fields since not needed.
let snagy = regex(pattern: "(?<=,\")(.*)(?=.\",)",
value: message
);
set_field("event_description", to_string(snagy["0"]));
let splitsys = split(",", message);
set_field("hostname", splitsys[0]);
set_field("receive_date_time", splitsys[1]);
set_field("serial_number", splitsys[2]);
//set_field("log_type", splitsys[3]); -- already handled
set_field("log_subtype", splitsys[4]);
set_field("time_generated", splitsys[6]);
//set_field("virtual_system", splitsys[7]); -- not used
set_field("event_id_name", splitsys[8]);
set_field("session_object", splitsys[9]);
set_field("subtype_module", splitsys[12]);
set_field("event_severity", splitsys[13]);
end
As I said in a previous post: " I didn’t care about the fields following event_description (past field 13) but if you wanted them you could replace(message,to_string(snagy["0"]))
to remove the field before breaking the rest out."
for TRAFFIC:
rule "PA-Firewall - ex1 - TRAFFIC fields"
when
has_field("log_type") &&
to_string($message.log_type) == "TRAFFIC"
then
let splittraf = split(",", to_string($message.message));
set_field("hostname", splittraf[0]);
set_field("receive_date_time", splittraf[1]);
set_field("serial_number", splittraf[2]);
//set_field("log_type", splittraf[3]); -- already handled
set_field("log_subtype", splittraf[4]);
set_field("time_generated", splittraf[6]);
set_field("session_src_ip", splittraf[7]);
set_field("session_dst_ip", splittraf[8]);
set_field("session_nat_src_ip", splittraf[9]);
set_field("firewall_rule", splittraf[11]);
//set_field("source_user", splittraf[12]); -- not used
//set_field("destination_user", splittraf[13]); -- not used
set_field("application", splittraf[14]);
//set_field("virtual_system", splittraf[15]); -- not used
set_field("session_src_zone", splittraf[16]);
set_field("session_dst_zone", splittraf[17]);
set_field("ingress_interface", splittraf[18]);
set_field("egress_interface", splittraf[19]);
set_field("log_forward_profile", splittraf[20]);
set_field("session_id", splittraf[22]);
set_field("repeat_count", splittraf[23]);
set_field("session_src_port", splittraf[24]);
set_field("session_dst_port", splittraf[25]);
set_field("session_nat_src_port", splittraf[26]);
set_field("session_nat_dst_port", splittraf[27]);
set_field("session_flags", splittraf[28]);
set_field("session_ip_proto", splittraf[29]);
set_field("action", splittraf[30]);
set_field("session_total_bytes", splittraf[31]);
set_field("session_bytes_sent", splittraf[32]);
set_field("session_bytes_received", splittraf[33]);
set_field("session_total_packets", splittraf[34]);
set_field("session_start_time", splittraf[35]);
set_field("session_elapsed_time_sec", splittraf[36]);
set_field("url_category", splittraf[37]);
set_field("source_country", splittraf[41]);
set_field("destination_country", splittraf[42]);
set_field("pkts_sent", splittraf[44]);
set_field("pkts_received", splittraf[45]);
set_field("session_end_reason", splittraf[46]);
set_field("action_source", splittraf[53]);
end
This third rule below in STAGE 1 breaks out GLOBALPROTECT fields and shunts the message over to the Remote Access stream because remote access has a different stream/index to handle presentation and how long we hold onto the message.
rule "PA-Firewall - ex1 - GLOBALPROTECT fields"
when
has_field("log_type") &&
to_string($message.log_type) == "GLOBALPROTECT"
then
set_field(field: "ra_tag", value: "globalprotect");
let message = to_string($message.message);
//convert " , " to "-" in temp message
let message = replace( value: message,
search: " , 64-bit",
replacement: " - 64-bit"
);
let splittraf = split(",", message);
set_field("hostname", splittraf[0]);
set_field("receive_date_time", splittraf[1]);
set_field("serial_number", splittraf[2]);
set_field("seqno", splittraf[3]);
//set_field("action_flag", splittraf[4]); //Not used: panorama
set_field("log_type", splittraf[5]);
// items 6,7 are unknown at the moment - skipping
set_field("time_generated", splittraf[8]);
set_field("virtual_system", splittraf[9]);
set_field("event_id_name", splittraf[10]);
set_field("action", splittraf[10]); //created for consistancey with Pulse vpn
set_field("session_stage", splittraf[11]);
set_field("auth_method", splittraf[12]);
set_field("tunnel_type", splittraf[13]);
set_field("authenticated_user", splittraf[14]);
set_field("srcregion", splittraf[15]);
set_field("machinename", splittraf[16]);
set_field("client_ip", splittraf[17]);
//set_field("public_ipv6", splittraf[18]); //unused
set_field("private_ip", splittraf[19]);
//set_field("private_ipv6", splittraf[20]); //unused
set_field("hostid", splittraf[21]);
set_field("serialnumber", splittraf[22]);
set_field("client_ver", splittraf[23]);
set_field("client_os", splittraf[24]);
set_field("client_os_ver", splittraf[25]);
set_field("repeatcnt", splittraf[26]);
set_field("reason", splittraf[27]);
set_field("error", splittraf[28]);
set_field("opaque", splittraf[29]);
set_field("status", splittraf[30]);
set_field("location", splittraf[31]);
set_field("login_duration", to_long(splittraf[32])); //enforce as number
set_field("connect_method", splittraf[33]);
set_field("error_code", splittraf[34]);
set_field("portal", splittraf[35]);
//This is no longer a firewall event - its a remote access event - REMOVING form PA index so it's not double stored
route_to_stream(name: "Remote Access Global",
remove_from_default: true
);
end
Once the message gets to the Remote Access pipeline it hits a couple of rules I set up that I won’t include here for brevity ( ) but there is one you might like. If you use it in the current pipeline we are describing you would want to create a STAGE 2 to make sure that the STAGE 1 rule that breaks out the fields completes.
STAGE 2 (I had some community help creating this rule!)
rule "RA-Calc-Connect-Time"
when
has_field("login_duration") &&
to_long($message.login_duration) > 0
then
let vpn_duration = parse_unix_milliseconds(to_long($message.login_duration) * 1000);
let vpn_hours = vpn_duration.hourOfDay;
let vpn_minutes = vpn_duration.minuteOfHour;
let vpn_seconds = vpn_duration.secondOfMinute;
let build_message_0 = concat(to_string(vpn_hours), " hours, ");
let build_message_1 = concat(build_message_0, to_string(vpn_minutes));
let build_message_2 = concat(build_message_1, " minutes, ");
let build_message_3 = concat(build_message_2, to_string(vpn_seconds));
let build_message_4 = concat(build_message_3, " seconds");
set_field( field: "vpn_connection_time",
value: build_message_4
);
end
All of the detail that PA has on their logs can be found here for reference:
https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions.html
Whew! This was WAY easier than trying to create a Content Pack. Hope this helps to give you a view into PaloAlto logs and Graylog rules and stages.