How to ignore comma in quoted strings?

[sert (Batuhan Sert)]

Hello

I want to extract Palo Alto Threat Logs PanOS 8.1
I tried Split and Index. I could not extract all fields because the log file is separated by commas and there are quotes that contain commas.

Icalso also tried Grok and was able to extract all the logs. But now I have a performance problem with Grok. Process Buffer will become full after a few minutes.

Is there another way to extract the logs and ignore the commas in quoted strings without overloading the system? Or to optimize the performance?
Here I have described the performance problem:

Process buffer gets full with the Grok pattern Extractor

Hello, To me, Im new here and this is my first post. I have a performance problem with Graylog Server running on Ubuntu 18.04 . As soon as I create grok extractors, process buffer will be full in a couple of minutes. Its a vm in azure cloud. ES and Graylog are running on the same vm with 1 node. The log rate is on average 50 logs/s. Graylog Version: v2.5.1+34194da Ressources: 4 VCPUs, 16GB RAM Reserved Rams for Graylog: 2 GB Reserved Rams for Elasticsearch: 8 GB I am grateful for any help…

Example log:
´´´
1,2019/03/22 14:55:43,012001026020,THREAT,url,2049,2019/03/22 14:55:43,192.168.200.216,148.251.64.134,95.143.56.76,148.251.64.134,200 allow_web-traffic,web-browsing,vsys1,internal,external,ae1.200,ethernet1/1,LFP_syslog,2019/03/22 14:55:43,74719,1,57822,443,9389,443,0x152b000,tcp,alert,“tracking.adalliance.io/ck?ck_1356234356=Schwangerschaft,Baby”,(9999),government,informational,client-to-server,3868243,0x2000000000000000,192.168.0.0-192.168.255.255,Germany,0,text/html,0,2,0,0,0,0,0,FW001,get,0,0,N/A,unknown,AppThreat-0-0,0x0,0,4294967295,
´´´

Grok:
´´´
{
“extractors”: [
{
“title”: “SYSTEM_LOGS”,
“extractor_type”: “grok”,
“converters”: ,
“order”: 0,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “message”,
“extractor_config”: {
“grok_pattern”: “%{BASE10NUM:UNWANTED},%{DATA:time_event;date;yyyy/MM/dd HH:mm:ss},%{BASE10NUM:serial},%{WORD:type},%{WORD:type_sub},%{BASE10NUM:UNWANTED},%{DATA:time_generated;date;yyyy/MM/dd HH:mm:ss},%{DATA:UNWANTED},%{DATA:id_event},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:severity},%{DATA:description},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{HOSTNAME:device_name}”
},
“condition_type”: “regex”,
“condition_value”: “^(.,SYSTEM,.)”
},
{
“title”: “THREAT_LOGS”,
“extractor_type”: “grok”,
“converters”: ,
“order”: 0,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “message”,
“extractor_config”: {
“grok_pattern”: “%{BASE10NUM:UNWANTED},%{DATA:time_event;date;yyyy/MM/dd HH:mm:ss},%{BASE10NUM:serial},%{WORD:type},%{WORD:type_sub},%{BASE10NUM:UNWANTED},%{DATA:time_generated;date;yyyy/MM/dd HH:mm:ss},%{IPV4:src},%{IPV4:dst},%{IPV4:src_nat},%{IPV4:dst_nat},%{DATA:rule},%{DATA:src_user},%{DATA:dst_user},%{DATA:app},%{DATA:UNWANTED},%{WORD:src_zone},%{WORD:dst_zone},%{DATA:interface_in},%{DATA:interface_out},%{DATA:action_log},%{DATA:UNWANTED},%{DATA:id_session},%{BASE10NUM:count},%{BASE10NUM:src_port},%{BASE10NUM:dst_port},%{BASE10NUM:src_nat_port},%{BASE10NUM:dst_nat_port},%{DATA:UNWANTED},%{DATA:protocol},%{DATA:action},%{QUOTEDSTRING:url},%{DATA:id_threat},%{DATA:category},%{DATA:severity},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:src_location},%{DATA:dst_location},%{DATA:UNWANTED},%{DATA:type_content},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{HOSTNAME:device_name},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:http_method},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED}”,
“named_captures_only”: false
},
“condition_type”: “regex”,
“condition_value”: “^(.,THREAT,.)”
},
{
“title”: “TRAFFIC_LOGS”,
“extractor_type”: “grok”,
“converters”: ,
“order”: 0,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “message”,
“extractor_config”: {
“grok_pattern”: “%{BASE10NUM:UNWANTED},%{DATA:time_event;date;yyyy/MM/dd HH:mm:ss},%{BASE10NUM:serial},%{WORD:type},%{WORD:type_sub},%{BASE10NUM:UNWANTED},%{DATA:time_generated;date;yyyy/MM/dd HH:mm:ss},%{IPV4:src},%{IPV4:dst},%{IPV4:src_nat},%{IPV4:dst_nat},%{DATA:rule},%{DATA:src_user},%{DATA:dst_user},%{DATA:app},%{DATA:UNWANTED},%{WORD:src_zone},%{WORD:dst_zone},%{DATA:interface_in},%{DATA:interface_out},%{DATA:action_log},%{DATA:UNWANTED},%{DATA:id_session},%{BASE10NUM:count},%{BASE10NUM:src_port},%{BASE10NUM:dst_port},%{BASE10NUM:src_nat_port},%{BASE10NUM:dst_nat_port},%{DATA:UNWANTED},%{DATA:protocol},%{DATA:action},%{NUMBER:bytes;long},%{NUMBER:bytes_sent;long},%{NUMBER:bytes_recieved;long},%{NUMBER:packets;long},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:category},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:src_location},%{DATA:dst_location},%{DATA:UNWANTED},%{NUMBER:packets_sent;long},%{NUMBER:packets_recieved;long},%{DATA:session_end_reason},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{HOSTNAME:device_name},%{DATA:src_action},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:type_tunnel},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED},%{DATA:UNWANTED}”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
}
],
“version”: “2.5.1”
}
´´´

I have created a Split and Index Extractor for Threat, System and Traffic Logs. Identical fields dont contain a regex rule.
´´´
{
“extractors”: [
{
“title”: “3_Serial Number”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 0,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “serial”,
“extractor_config”: {
“index”: 3,
“split_by”: “,”
},
“condition_type”: “none”,
“condition_value”:“”
},
{
“title”: “4_Type”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 1,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “type”,
“extractor_config”: {
“index”: 4,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “5_Threat/Content Type”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 2,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “type_sub”,
“extractor_config”: {
“index”: 5,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “8_Source IP”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 3,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “src”,
“extractor_config”: {
“index”: 8,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “9_Destination IP”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 4,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “dst”,
“extractor_config”: {
“index”: 9,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “10_NAT Source IP”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 5,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “src_nat”,
“extractor_config”: {
“index”: 10,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “11_NAT Destination IP”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 6,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “dst_nat”,
“extractor_config”: {
“index”: 11,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “12_Rule Name”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 7,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “rule”,
“extractor_config”: {
“index”: 12,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “13_Source User”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 8,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “src_user”,
“extractor_config”: {
“index”: 13,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “14_Destination User”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 9,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “dst_user”,
“extractor_config”: {
“index”: 14,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “15_Application”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 10,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “app”,
“extractor_config”: {
“index”: 15,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “17_Source Zone”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 11,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “src_zone”,
“extractor_config”: {
“index”: 17,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “18_Destination Zone”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 12,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “dst_zone”,
“extractor_config”: {
“index”: 18,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “19_Inbound Interface”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 13,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “interface_in”,
“extractor_config”: {
“index”: 19,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “20_Outbound Interface”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 14,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “interface_out”,
“extractor_config”: {
“index”: 20,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “25_Source Port”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 15,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “src_port”,
“extractor_config”: {
“index”: 25,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “26_Destination Port”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 16,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “dst_port”,
“extractor_config”: {
“index”: 26,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “27_NAT Source Port”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 17,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “src_nat_port”,
“extractor_config”: {
“index”: 27,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “28_NAT Destination Port”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 18,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “dst_nat_port”,
“extractor_config”: {
“index”: 28,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “30_Protocol”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 19,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “protocol”,
“extractor_config”: {
“index”: 30,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “31_Action”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 20,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “action”,
“extractor_config”: {
“index”: 31,
“split_by”: “,”
},
“condition_type”:“none”,
“condition_value”:“”
},
{
“title”: “32_TRAFFIC Bytes”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 21,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “bytes”,
“extractor_config”: {
“index”: 32,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “33_TRAFFIC Bytes Sent”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 22,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “bytes_sent”,
“extractor_config”: {
“index”: 33,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “34_TRAFFIC Bytes Recieved”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 23,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “bytes_recieved”,
“extractor_config”: {
“index”: 34,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “35_TRAFFIC Packets”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 24,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “packets”,
“extractor_config”: {
“index”: 35,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “38_TRAFFIC Category”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 26,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “category”,
“extractor_config”: {
“index”: 38,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “42_TRAFFIC Source Location”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 27,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “src_location”,
“extractor_config”: {
“index”: 42,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “43_TRAFFIC Destination Location”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 28,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “dst_location”,
“extractor_config”: {
“index”: 43,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “45_TRAFFIC Packets Sent”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 29,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “packets_sent”,
“extractor_config”: {
“index”: 45,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “46_TRAFFIC Packets Received”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 30,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “packets_recieved”,
“extractor_config”: {
“index”: 46,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “47_TRAFFIC Session End Reason”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 31,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “session_end_reason”,
“extractor_config”: {
“index”: 47,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “53_TRAFFIC Device Name”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 32,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “device_name”,
“extractor_config”: {
“index”: 53,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “54_TRAFFIC Action Source”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 33,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “src_action”,
“extractor_config”: {
“index”: 54,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “61_TRAFFIC Tunnel Type”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 34,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “type_tunnel”,
“extractor_config”: {
“index”: 61,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,TRAFFIC,.)”
},
{
“title”: “14_SYSTEM Severity”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 35,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “severity”,
“extractor_config”: {
“index”: 14,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,SYSTEM,.)”
},
{
“title”: “15_SYSTEM Description”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 36,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “description”,
“extractor_config”: {
“index”: 15,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,SYSTEM,.)”
},
{
“title”: “23_SYSTEM Device Name”,
“extractor_type”: “split_and_index”,
“converters”: ,
“order”: 37,
“cursor_strategy”: “copy”,
“source_field”: “message”,
“target_field”: “device_name”,
“extractor_config”: {
“index”: 23,
“split_by”: “,”
},
“condition_type”: “regex”,
“condition_value”: “^(.,SYSTEM,.)”
}
],
“version”: “2.5.1”
}
´´´

I am grateful for any help!

[jan (Jan Doberstein)]

have you seen this?

http://docs.graylog.org/en/3.0/pages/integrations.html

[sert (Batuhan Sert)]

thank you for your advice.

i have tried to install the plugin but get an error message “Forbiddden”

´´´
sysadmin@AVMU01:~$ sudo apt-get install graylog-plugin-enterprise-integrations
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages were automatically installed and are no longer required:
linux-azure-cloud-tools-4.15.0-1035 linux-azure-cloud-tools-4.15.0-1036 linux-azure-cloud-tools-4.15.0-1037
linux-azure-headers-4.15.0-1035 linux-azure-headers-4.15.0-1036 linux-azure-headers-4.15.0-1037
linux-azure-tools-4.15.0-1035 linux-azure-tools-4.15.0-1036 linux-azure-tools-4.15.0-1037
Use ‘sudo apt autoremove’ to remove them.
The following NEW packages will be installed:
graylog-plugin-enterprise-integrations
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 162 kB of archives.
After this operation, 170 kB of additional disk space will be used.
Ign:1 https://packages.graylog2.org/repo/debian stable/2.5 amd64 graylog-plugin-enterprise-integrations all 2.5.0+0
Err:1 https://packages.graylog2.org/repo/debian stable/2.5 amd64 graylog-plugin-enterprise-integrations all 2.5.0+0
403 Forbidden [IP: 52.218.97.50 443]
E: Failed to fetch https://packages.graylog2.org/repo/debian/pool/stable/2.5/g/graylog-plugin-enterprise-integrations/graylog-plugin-enterprise-integrations_2.5.0+0_all.deb 403 Forbidden [IP: 52.218.97.50 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
´´´