Unable to Import Palo Alto Networks Content Packs

Hello,

I’m relatively new to Graylog so please forgive me if I miss anything.

I followed the Graylog documentation to perform a fresh installation on an Ubuntu Server 18.0.4.4 LTS machine. I’m running:

Graylog 3.2.4+a407287 on graylog (Private Build 1.8.0_252 on Linux 4.15.0-96-generic)

I’m consistently running into an issue with uploading/installing Content Packs designed for use with Palo Alto Networks Next-Generation Firewalls.

I’ve tried several:

I tried to enter links for 4 different Content Packs but when I tried to post the message, I got an error message stating that new users can only add two links in a message. That’s a very strange restriction! That’s the first time I’ve ever run into something like that! :slight_smile:

I already have inputs collected for my Palo Alto Networks NGFW - I have two separate inputs:

  • System Messages - syslog 5140/udp
  • Firewall Logs - syslog 5142/udp

I also happen to have a Netflow input on 5141/udp so that’s why the NGFW logs are using dis-contiguous port numbers.

I navigate to System -> Content Packs then either browse the Content Pack Marketplace and I also tried using ‘git clone’ to grab the repository for the aformentioned Content Packs. If I open the JSON from the Marketplace in a text editor, all of the content is merged into a single line. If I download them directly from GitHub and open them in a text editor, they are well-formed and look like what I would expect a JSON file to look like.

When I import, I receive the following message:

Could not import content pack

Error importing content pack, please ensure it is a valid JSON file. Check your Graylog logs for more information.
Null id at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 808, column: 1] (through reference chain: org.graylog2.contentpacks.model.AutoValue_LegacyContentPack$Builder[“id”])

I SSHed into the server and navigated to /var/log/graylog-server/ and searched through server.log which is the only log file therei. I also tried using ‘tail’ to watch the server.log in realtime as I tried to import and I can’t find anything.

I also tried to look at the logs within the application but I’m not sure what I’m looking for.

Any help would be greatly appreciated!!!

Best Regards,

-JeffH

The Marketplace Content Packs you listed are for a WAY old version of Graylog and the github ones are from two years ago so are very likely the same. Graylog has changed too much. PalotAlto has changed their logs a lot over time too. What version of PA are you running? We are running 9…1.1. I currently have Graylog set up to break out and manage the messages via pipeline rules rather than extractors but only for SYSTEM and TRAFFIC messages. It would be easy to create a new THREAT rule off them though. Happy to share the rules here if that’s the route you want to go.

1 Like

if you install and use the integrations plugin you get a palo alto input:

https://docs.graylog.org/en/3.2/pages/integrations/inputs/palo_alto_networks_input.html#palo-alto-network-input

1 Like

@jan - should have mentioned that too - there are a couple challenges that I have with the built in integration:

  • The Integrations plugin goes to 8.1 and PA is on version 9+ with notable adjustments in fields. You can adjust the integration but as an example, PA has broken out a new message type GLOBALPROTECT … which used to be under SYSTEM - not yet available in the current integration. Even more annoying, all the other log types are in CSV field position [3] but GLOBALPROTECT is CSV field position [5]…

  • The SYSTEM messages have a description field that has quotes around it contains zero or more commas and is semi key-value oriented. The integration kind of captures these but can’t break them into field name and data - until the pipeline… if you want them.

  • There is no method for ignoring fields

Having the message breakout in a pipeline gives me a lot more flexibility for this and as things evolve.

Example: (Assumes you already pulled out that it was logtype SYSTEM in a previous stage)

rule "PA-Firewall - ex1 - SYSTEM fields"
when
    has_field("log_type")           &&
    to_string($message.log_type) == "SYSTEM"
then
    let message     = to_string($message.message);

    // Regex breaks out event description in quotes here.
    // there are possibly commas in the description that messes up the split
    // so regex the event description and ignore ending fields since not needed.
    let snagy       = regex(pattern:    "(?<=,\")(.*)(?=.\",)", 
                            value:      message
                    );

    set_field("event_description", to_string(snagy["0"]));

	let splitsys   = split(",", message);
    set_field("hostname",                   splitsys[0]);
    set_field("receive_date_time",          splitsys[1]);
    set_field("serial_number",              splitsys[2]);
  //set_field("log_type",                   splitsys[3]); -- already handled
    set_field("log_subtype",				splitsys[4]);
    set_field("time_generated",				splitsys[6]);
  //set_field("virtual_system",				splitsys[7]); -- not used
    set_field("event_id_name",				splitsys[8]);
    set_field("session_object",				splitsys[9]);
    set_field("subtype_module",				splitsys[12]);
    set_field("event_severity",				splitsys[13]);
end

I didn’t care about the fields following event_description but if you wanted them you could replace(message,event_description) to remove the field before breaking the rest out.

OK WAY more information than asked for but it’s a slow day and someone may search for a similar answer in the future.

2 Likes

thank you @tmacgbay for sharing this!

I have linked this posting to the Github issue about to update the integration: https://github.com/Graylog2/graylog-plugin-integrations/issues/452

1 Like

Yes - I understand that installing the plugin adds an input for Palo Alto Networks firewalls - the point is that the plugin fails to install.

I am running PAN-OS 9.1.2 on my firewall. I actually work for Palo Alto Networks and am running several firewalls - one for production and 4 on various versions of code (8.1, 9.0, 9.1 and 9.2 beta).

Yes - there were additional logs added in PAN-OS 9.1 and there will be additional ones added in 9.2 - tentatively scheduled to ship in late May/early June but is likely going to be delayed.

I would think that PAN-OS 9.1 would still work with Graylog but the newer log types wouldn’t be detected/available for reporting. It is possible to configure PAN-OS with a granular policy controling what log events are forwarded to syslog, so it is likely possible to ensure that the logs that are not supported aren’t sent to Graylog in the first place.

I am not savvy enough to modify the plugins myself, but if I can provide someone with the details they need to update support for PAN-OS 9.1 and 9.2, I’d be happy to do so.

It seems like the vast majority of the plugins I’ve looked at are based on older versions of Graylog and are no longer functional with the newer releases. That begs the question - why do they even show up in the results of a search via the Graylog admin interface if they no longer work? It seems like they should be pruned from the database and a message should be added to the UI that provides more insight.

I see a lot of people asking questions about various plugins and the answer is almost always that the plugin that someone is asking about is based on an older version of Graylog and will no longer work. It would seem to me like it would make sense to add a very obvious message in the admin interface that speaks to this.

Case in point, I also tried to install a Graylog plugin to add support for Netflow. I found that the version I found on the Graylog GitHub repo is wicked old and no longer works with 3.2. I ended up discovering that there’s a Netflow input that is automatically part of Graylog 3.2. That’s an ideal situation but again, why are there so many references to outdated plugins?

Hi! Yes - I would love to see what your rules look like. Please know that I am brand new to Graylog, so the more detailed information you can provide, the better!

I was able to set up a new PANW TCP input and modify the configuration on my production firewall (PAN-OS 9.1.2) to send logs to 5555/tcp. That seems to be working nicely. I do need to spend some time going through the settings in the input to determine which log attributes are supported vs what PAN-OS 9.1.2 is sending.

Anything you’re willing to share would be greatly appreciated!

-JeffH

I found this repository on GitHub (https://github.com/jamesfed/PANOSGraylogExtractor) and there’s an extractor for PAN-OS 9.1. I was able to copy/paste the JSON code for 9.1 into the extractor for the PAN-OS input and I see it’s applied to the data.

I tried creating a dashboard and filtered by source to limit the data to only what is received from my firewall. I also have syslog from my AT&T UVerse router streaming to a generic UDP syslog input. Anytime I add the source manually, that seems to work, but if I try to save it to the dashboard, any time I leave the page and go back to the dashboard and select the dashboard I created, the source filter is gone and I have to apply it again.

I guess I have a lot of reading to do. :slight_smile:

I tried to be good and create a content pack but man that interface is just soooo wonky… :face_vomiting:

All I was trying to do is put out pipelines and their rules and extract the extra stuff I have in there… AND you have to get it 100% right because although I can edit all I want in github, once you post in Graylog Marketplace, the only thing you can change is the title (LAME!) Ok… done complaining.

I am posting my rules in sequence here. I am assuming your input works and you know how to make all the connections, stream, indexes and rules to connect them etc.

STAGE 0:

rule "PA-Firewall - ex0 - set log type"
when
    regex(pattern: "(,TRAFFIC,|,THREAT,|,CONFIG,|,SYSTEM,)", value: to_string($message.message)).matches == true
then
    let splitlog = split(",", to_string($message.message));
    set_field("log_type", splitlog[3] );
 end

– And because PA 9.1.1 breaks out GLOBALPROTECT logs to different log file (used to be in SYSTEM) AND they decided that field 5 should contain the log type rather than field 3 like SYSTEM and TRAFFIC do. Sigh. Here is the second rule to handle marking that.

rule "PA-Firewall - ex0 - set log type - GP"
when
    regex(pattern: "(,GLOBALPROTECT,)", value: to_string($message.message)).matches == true
then
    set_field("log_type", "GLOBALPROTECT");
 end

Now that we have separated out the log types we can delve into breaking out fields - >.

STAGE 1

rule "PA-Firewall - ex1 - SYSTEM fields"
when
    has_field("log_type")           &&
    to_string($message.log_type) == "SYSTEM"
then
    let message     = to_string($message.message);

    // Regex breaks out event description in quotes here.
    // there are possibly commas in the description that messes up the split
    // so regex the event description and ignore ending fields since not needed.
    let snagy       = regex(pattern:    "(?<=,\")(.*)(?=.\",)", 
                            value:      message
                    );

    set_field("event_description", to_string(snagy["0"]));

	let splitsys   = split(",", message);
    set_field("hostname",                   splitsys[0]);
    set_field("receive_date_time",          splitsys[1]);
    set_field("serial_number",              splitsys[2]);
  //set_field("log_type",                   splitsys[3]); -- already handled
    set_field("log_subtype",				splitsys[4]);
    set_field("time_generated",				splitsys[6]);
  //set_field("virtual_system",				splitsys[7]); -- not used
    set_field("event_id_name",				splitsys[8]);
    set_field("session_object",				splitsys[9]);
    set_field("subtype_module",				splitsys[12]);
    set_field("event_severity",				splitsys[13]);
end

As I said in a previous post: " I didn’t care about the fields following event_description (past field 13) but if you wanted them you could replace(message,to_string(snagy["0"])) to remove the field before breaking the rest out."

for TRAFFIC:

rule "PA-Firewall - ex1 - TRAFFIC fields"
when
    has_field("log_type")       &&
    to_string($message.log_type) == "TRAFFIC"
then
    let splittraf = split(",", to_string($message.message));
    set_field("hostname",                   splittraf[0]);
    set_field("receive_date_time",          splittraf[1]);
    set_field("serial_number",              splittraf[2]);
    //set_field("log_type",                 splittraf[3]); -- already handled
    set_field("log_subtype",				splittraf[4]);
    set_field("time_generated",				splittraf[6]);
    set_field("session_src_ip",				splittraf[7]);
    set_field("session_dst_ip",				splittraf[8]);
    set_field("session_nat_src_ip", 		splittraf[9]);
    set_field("firewall_rule",				splittraf[11]);
  //set_field("source_user",				splittraf[12]); -- not used
  //set_field("destination_user",			splittraf[13]); -- not used
    set_field("application",				splittraf[14]);
  //set_field("virtual_system",				splittraf[15]); -- not used
    set_field("session_src_zone",			splittraf[16]);
    set_field("session_dst_zone",			splittraf[17]);
    set_field("ingress_interface",			splittraf[18]);
    set_field("egress_interface",			splittraf[19]);
    set_field("log_forward_profile",		splittraf[20]);
    set_field("session_id",					splittraf[22]);
    set_field("repeat_count",				splittraf[23]);
    set_field("session_src_port", 			splittraf[24]);
    set_field("session_dst_port", 			splittraf[25]);
    set_field("session_nat_src_port",		splittraf[26]);
    set_field("session_nat_dst_port", 		splittraf[27]);
    set_field("session_flags", 				splittraf[28]);
    set_field("session_ip_proto",			splittraf[29]);
    set_field("action",						splittraf[30]);
    set_field("session_total_bytes",		splittraf[31]);
    set_field("session_bytes_sent",			splittraf[32]);
    set_field("session_bytes_received",		splittraf[33]);
    set_field("session_total_packets",		splittraf[34]);
    set_field("session_start_time",			splittraf[35]);
    set_field("session_elapsed_time_sec",	splittraf[36]);
    set_field("url_category",				splittraf[37]);
    set_field("source_country",				splittraf[41]);
    set_field("destination_country",		splittraf[42]);
    set_field("pkts_sent",					splittraf[44]);
    set_field("pkts_received", 				splittraf[45]);
    set_field("session_end_reason", 		splittraf[46]);
    set_field("action_source", 				splittraf[53]);
end

This third rule below in STAGE 1 breaks out GLOBALPROTECT fields and shunts the message over to the Remote Access stream because remote access has a different stream/index to handle presentation and how long we hold onto the message.

rule "PA-Firewall - ex1 - GLOBALPROTECT fields"
when
    has_field("log_type")           &&
    to_string($message.log_type) == "GLOBALPROTECT"
then
    set_field(field: "ra_tag", value: "globalprotect");

    let message     = to_string($message.message);

    //convert " , " to "-" in temp message
    let message = replace(  value:          message, 
                            search:         " , 64-bit",
                            replacement:    " - 64-bit"
            );

	let splittraf   = split(",", message);
    set_field("hostname",                   splittraf[0]);
    set_field("receive_date_time",          splittraf[1]);
    set_field("serial_number",              splittraf[2]);
    set_field("seqno",                      splittraf[3]); 
    //set_field("action_flag",				splittraf[4]);  //Not used: panorama
    set_field("log_type",                   splittraf[5]);
                                                            // items 6,7 are unknown at the moment - skipping
    set_field("time_generated",				splittraf[8]);
    set_field("virtual_system",				splittraf[9]); 
    set_field("event_id_name",				splittraf[10]);
    set_field("action",				        splittraf[10]); //created for consistancey with Pulse vpn
    set_field("session_stage",				splittraf[11]);
    set_field("auth_method",				splittraf[12]);
    set_field("tunnel_type",				splittraf[13]);
    set_field("authenticated_user",         splittraf[14]);
    set_field("srcregion",		            splittraf[15]);
    set_field("machinename",				splittraf[16]);
    set_field("client_ip",				    splittraf[17]);
    //set_field("public_ipv6",				splittraf[18]);  //unused
    set_field("private_ip",				    splittraf[19]);
    //set_field("private_ipv6",				splittraf[20]);  //unused
    set_field("hostid",				        splittraf[21]);
    set_field("serialnumber",			    splittraf[22]);
    set_field("client_ver",				    splittraf[23]);
    set_field("client_os",				    splittraf[24]);
    set_field("client_os_ver",		        splittraf[25]);
    set_field("repeatcnt",				    splittraf[26]);
    set_field("reason",				        splittraf[27]);
    set_field("error",				        splittraf[28]);
    set_field("opaque",				        splittraf[29]);
    set_field("status",				        splittraf[30]);
    set_field("location",				    splittraf[31]);
    set_field("login_duration",				to_long(splittraf[32]));  //enforce as number
    set_field("connect_method",				splittraf[33]);
    set_field("error_code",				    splittraf[34]);
    set_field("portal",				        splittraf[35]);
    
    //This is no longer a firewall event - its a remote access event - REMOVING form PA index so it's not double stored
    route_to_stream(name:                   "Remote Access Global",
                    remove_from_default:    true
                    );

end

Once the message gets to the Remote Access pipeline it hits a couple of rules I set up that I won’t include here for brevity ( :crazy_face:) but there is one you might like. If you use it in the current pipeline we are describing you would want to create a STAGE 2 to make sure that the STAGE 1 rule that breaks out the fields completes.

STAGE 2 (I had some community help creating this rule!)

rule "RA-Calc-Connect-Time"
when
    has_field("login_duration")            &&
    to_long($message.login_duration) > 0 
then
    let vpn_duration    = parse_unix_milliseconds(to_long($message.login_duration) * 1000);
    let vpn_hours       = vpn_duration.hourOfDay;
    let vpn_minutes     = vpn_duration.minuteOfHour;
    let vpn_seconds     = vpn_duration.secondOfMinute;
    let build_message_0 = concat(to_string(vpn_hours), " hours, ");
    let build_message_1 = concat(build_message_0, to_string(vpn_minutes));
    let build_message_2 = concat(build_message_1, " minutes, ");
    let build_message_3 = concat(build_message_2, to_string(vpn_seconds));
    let build_message_4 = concat(build_message_3, " seconds");

    set_field(  field: "vpn_connection_time", 
                value:  build_message_4
            );
end

All of the detail that PA has on their logs can be found here for reference:
https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions.html

Whew! This was WAY easier than trying to create a Content Pack. Hope this helps to give you a view into PaloAlto logs and Graylog rules and stages.

2 Likes

He @cyberjew

Yes - there were additional logs added in PAN-OS 9.1 and there will be additional ones added in 9.2 - tentatively scheduled to ship in late May/early June but is likely going to be delayed.

That is why the Input is created to be compatible and customizable: https://docs.graylog.org/en/3.2/pages/integrations/inputs/palo_alto_networks_input.html#palo-alto-network-input

I would think that PAN-OS 9.1 would still work with Graylog but the newer log types wouldn’t be detected/available for reporting. It is possible to configure PAN-OS with a granular policy controling what log events are forwarded to syslog, so it is likely possible to ensure that the logs that are not supported aren’t sent to Graylog in the first place.

That is why I have linked to the Github Issue above. You might want to add additional information that one of the developers can pick that up and make the software compatible with newer versions of PanOS.

Just for your information - the vendors do not contact us about their new versions or what they will change. We can only change when someone is telling us what has changed. As we do not run every device in every possible software version this might take some time and help from the person that is using the latest release already.

It seems like the vast majority of the plugins I’ve looked at are based on older versions of Graylog and are no longer functional with the newer releases. That begs the question - why do they even show up in the results of a search via the Graylog admin interface if they no longer work? It seems like they should be pruned from the database and a message should be added to the UI that provides more insight.

I’m not sure if you point here to the Graylog UI or the Marketplace or any other kind of search.

In the Graylog UI you see all provided inputs, when you open System > Inputs - What it can’t filter if the input is compatible with your specific device/version.
Should you refer to the Marketplace - yes there are Plugins created by others that are not compatible with all Graylog versions. But why should the Marketplace not display them? What it lags - the ability to show with what version of Graylog this is compatible, that is true.

I see a lot of people asking questions about various plugins and the answer is almost always that the plugin that someone is asking about is based on an older version of Graylog and will no longer work.

Not sure to what content you refer - but whomever have created that need to update it, from time to time when something has changed in Graylog.

Case in point, I also tried to install a Graylog plugin to add support for Netflow. I found that the version I found on the Graylog GitHub repo is wicked old and no longer works with 3.2. I ended up discovering that there’s a Netflow input that is automatically part of Graylog 3.2. That’s an ideal situation but again, why are there so many references to outdated plugins?

You might want to share where old, non working plugins for netflow are referenced? When it is possible for us we will change those links.

regards
Jan

Wanted to chime in with a thought here – would it be possible to add a ‘posted date’ and ‘updated date’ to the marketplace plugins listing or entries? When you go to take a look they’re just in alphabetical order and offer no information about how current they are – this could even pull dates from the associated github repos on most of them if the data isn’t stored on the Marketplace. There are tags for the version on some of them but that hasn’t been too helpful – some old ones work fine, some new(er) ones don’t.

I’ve stopped looking through the Marketplace because it’s not clear if the plugins will work and a good chunk of the ones I tried did not seem to work. If there was a ‘newest’ sorting option or the ability to see dates on these I would be much more likely to check out and try newer integrations.

2 Likes

you can’t use any Content Packs! ! it depends on PAN OS of Palo Alto firewall (PAN OS 8.0.9, 9 …) and on the level of palo firewall you must configure a syslog profile with the same port number as which of content pack, it is necessary that the same port,…and it also depended on the graylog version

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.