Parsing Suricata from OpnSense

accidentaladmin · December 21, 2023, 1:22am

Greetings:

I am running OpnSense with Suricata enabled. I have managed to ship Suricata logs to Graylog however the it comes as a big blob.

{"timestamp":"2023-12-20T20:12:30.285101-0500","flow_id":1582594801964295,"in_iface":"igc0","event_type":"alert","vlan":[100],"src_ip":"10.100.0.198","src_port":60794,"dest_ip":"144.217.225.162","dest_port":80,"proto":"TCP","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":2013504,"rev":6,"signature":"ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management","category":"Not Suspicious Traffic","severity":3,"metadata":{"created_at":["2011_08_31"],"former_category":["POLICY"],"updated_at":["2020_04_22"]}},"http":{"hostname":"download.proxmox.com","url":"/debian/pbs-client/dists/bookworm/InRelease","http_user_agent":"Debian APT-HTTP/1.3 (2.6.1)","http_method":"GET","protocol":"HTTP/1.1","status":304,"length":0},"app_proto":"http","flow":{"pkts_toserver":4,"pkts_toclient":3,"bytes_toserver":509,"bytes_toclient":391,"start":"2023-12-20T20:12:30.169223-0500"}}

It appears its all set to parse easily. What would be the best way to parse it at, say, the ,'s, utilizing the strings before the : as the field?

For instance,

{"pkts_toserver":4,"pkts_toclient":3,"bytes_toserver":509,"bytes_toclient":391,"start":"2023-12-20T20:12:30.169223-0500"}

Would become:

pkts_toserver
4
pkts_toclient
3
bytes_toserver
509
bytes_toclient
391
start
2023-12-20T20:12:30.169223-0500

Thank you!

gsmith · December 21, 2023, 1:40am

Hey @accidentaladmin

Happy Holidays.

I personally would use Pipeline with Key Values, (KV)something like this. You may have to adjust for your logs.

rule "KV-Parser"
when
    has_field("message")
then
    set_fields(
                fields:
                    key_value(
                    value: to_string($message.message), 
                    delimiters: ",",
                    kv_delimiters: ":",
                    trim_value_chars: "",
                    trim_key_chars:""
                    )
            );
end

accidentaladmin · December 21, 2023, 2:12am

gsmith:

rule "KV-Parser"
when
    has_field("message")
then
    set_fields(
                fields:
                    key_value(
                    value: to_string($message.message), 
                    delimiters: ",",
                    kv_delimiters: ":",
                    trim_value_chars: "",
                    trim_key_chars:""
                    )
            );
end

Hey G, Happy Holidays to you!

Unfortunately the rule you suggested didn’t work. I think its getting hung up at “value”. I switched over to the new Rule Builder feature and this is what I have so far; I just need to figure out what to put in “value”:

gsmith · December 21, 2023, 2:17am

Oh wow ,
I havent see the new version yet thats kool.

So then the Value would be something like this. Using your filed called application_name.

rule "KV-Parser"
when
    has_field("application_name")
then
    set_fields(
                fields:
                    key_value(
                    value: to_string($message.application_name), 
                    delimiters: ",",
                    kv_delimiters: ":",
                    trim_value_chars: "",
                    trim_key_chars:""
                    )
            );
end

gsmith · December 21, 2023, 2:18am

Normally if you go with to_string it would be the field where the messages are.

Joel_Duffield · December 21, 2023, 2:44am

That appears to be standard JSON format, so the flatten_json function in pipelines should work with it pretty well to transform it into individual fields.

accidentaladmin · December 21, 2023, 2:46am

Yeah, sorry, I should have said that up front.

So how would I use the flatten_json function?

gsmith · December 21, 2023, 5:06am

Hey @accidentaladmin
I have an old pipe, think something like this.

rule "Random User Data Flatten Json Rule"
// From sample data : https://randomuser.me/api/
// Api input path: *
when
    true
then
    let sJson = to_string($message.result);
    let sJson = regex_replace(
        pattern: "^\\[|\\]$",
        value: sJson,
        replacement: ""
        );
    let rsJson = flatten_json(to_string(sJson), "flatten");
    set_fields(to_map(rsJson));
    remove_field("result");
    set_field("message", "parsed user data");
end

accidentaladmin · December 21, 2023, 10:20pm

I feel so dumb when dealing with pipeline runs. Ive stared at that old pipeline rule and can’t make heads or tails of it.

accidentaladmin · December 21, 2023, 11:00pm

So here is where I am at:

rule "Suricata"
when
  has_field(
    field : "application_name"
  )
  AND
  ( has_field("application_name") &&
lowercase(to_string($message."application_name")) == lowercase("suricata")
)

then
  let output_1 = key_value(
    value : "message",
    delimiters : ",",
    kv_delimiters : ":",
    ignore_empty_values : false,
    allow_dup_keys : false,
    trim_key_chars : "\"\"",
    trim_value_chars : "\"\""
  );
  set_fields(
    fields : output_1,
    clean_fields : false
  );
end

But I get this:

Error evaluating action for rule <Suricata/6584c0bfc0330b78cc8fbb7a> (pipeline <Parse Suricata/65839935c0330b78cc8f2c27>) - In call to function 'key_value' at 12:17 an exception was thrown: Missing value for key message

As referenced above, when it comes to pipeline rules I am dumb

gsmith · December 21, 2023, 11:22pm

Hey @accidentaladmin

Oh your all good, I think you have double configurations under the “when” statement.

Here is a better reference, shoothub was pretty good at pipes. I basically get by,Im by no means really good at pipes but I have a library full if need be.

TBH, there is alot of examples in the forum over the years, not sure is you executed a global search here.

gsmith · December 21, 2023, 11:34pm

BTW @accidentaladmin you could use a JSON extractor on th input and enable the tic box for flatten json. Im still working on version 4 so Im not sure if things have changed

accidentaladmin · December 22, 2023, 12:39am

Okay, think I got it (still needs some clean up)

rule "key_value_parser"
when
     contains(
  value: to_string($message."application_name"),
  search: "suricata",
  ignore_case: true
)
then
    set_fields(
        fields:
            key_value(
                value: to_string($message.message),
                delimiters:",",
                kv_delimiters:":",
                trim_key_chars:"\"{}",
                trim_value_chars: "\"{}:[]")
            );
end

Produces this (partial):

From this:

{
  "process_id": "44826",
  "gl2_accounted_message_size": 2122,
  "level": 6,
  "gl2_remote_ip": "192.168.86.1",
  "gl2_remote_port": 14187,
  "streams": [
    "65838d32c0330b78cc8f2609"
  ],
  "gl2_message_id": "01HJ7EGZ0G0188QAJPJJYDYSPZ",
  "source": "OPNsense.thesmiths.management",
  "message": "{\"timestamp\":\"2023-12-21T19:24:10.756644-0500\",\"flow_id\":2023693415535850,\"in_iface\":\"igc0\",\"event_type\":\"alert\",\"vlan\":[100],\"src_ip\":\"10.100.0.198\",\"src_port\":58188,\"dest_ip\":\"144.217.225.162\",\"dest_port\":80,\"proto\":\"TCP\",\"tx_id\":0,\"alert\":{\"action\":\"allowed\",\"gid\":1,\"signature_id\":2013504,\"rev\":6,\"signature\":\"ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management\",\"category\":\"Not Suspicious Traffic\",\"severity\":3,\"metadata\":{\"created_at\":[\"2011_08_31\"],\"former_category\":[\"POLICY\"],\"updated_at\":[\"2020_04_22\"]}},\"http\":{\"hostname\":\"download.proxmox.com\",\"url\":\"/debian/pbs-client/dists/bookworm/InRelease\",\"http_user_agent\":\"Debian APT-HTTP/1.3 (2.6.1)\",\"http_method\":\"GET\",\"protocol\":\"HTTP/1.1\",\"status\":304,\"length\":0},\"app_proto\":\"http\",\"flow\":{\"pkts_toserver\":4,\"pkts_toclient\":3,\"bytes_toserver\":509,\"bytes_toclient\":391,\"start\":\"2023-12-21T19:24:10.644330-0500\"}}",
  "gl2_source_input": "65613c6bd8e1247f71e8771f",
  "sequenceId": "995584",
  "application_name": "suricata",
  "full_message": "<174>1 2023-12-21T19:24:10-05:00 OPNsense.thesmiths.management suricata 44826 - [meta sequenceId=\"995584\"] {\"timestamp\":\"2023-12-21T19:24:10.756644-0500\",\"flow_id\":2023693415535850,\"in_iface\":\"igc0\",\"event_type\":\"alert\",\"vlan\":[100],\"src_ip\":\"10.100.0.198\",\"src_port\":58188,\"dest_ip\":\"144.217.225.162\",\"dest_port\":80,\"proto\":\"TCP\",\"tx_id\":0,\"alert\":{\"action\":\"allowed\",\"gid\":1,\"signature_id\":2013504,\"rev\":6,\"signature\":\"ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management\",\"category\":\"Not Suspicious Traffic\",\"severity\":3,\"metadata\":{\"created_at\":[\"2011_08_31\"],\"former_category\":[\"POLICY\"],\"updated_at\":[\"2020_04_22\"]}},\"http\":{\"hostname\":\"download.proxmox.com\",\"url\":\"/debian/pbs-client/dists/bookworm/InRelease\",\"http_user_agent\":\"Debian APT-HTTP/1.3 (2.6.1)\",\"http_method\":\"GET\",\"protocol\":\"HTTP/1.1\",\"status\":304,\"length\":0},\"app_proto\":\"http\",\"flow\":{\"pkts_toserver\":4,\"pkts_toclient\":3,\"bytes_toserver\":509,\"bytes_toclient\":391,\"start\":\"2023-12-21T19:24:10.644330-0500\"}}",
  "facility_num": 21,
  "gl2_source_node": "d03cc833-8d4e-4ae5-bff7-8aba670a6c2a",
  "_id": "6d6e6503-a060-11ee-b303-7a59a036ff5d",
  "facility": "local5",
  "timestamp": "2023-12-22T00:24:10.000Z"
}

gsmith · December 22, 2023, 12:43am

Nice and thanks for sharing @accidentaladmin

will-ferrell-old-school

system · January 5, 2024, 12:43am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Suricata message extrator Graylog Add-ons	2	834	June 26, 2019
Suricata eve json logs Graylog Central (peer support)	1	1398	September 26, 2019
Sidecar to Suricata Graylog Central (peer support) sidecar	2	369	December 5, 2023
Graylog 4.X Not Parsing All JSON Fields with parse_json() Graylog Central (peer support) pipeline-rules	2	839	April 27, 2021
Suricata on PfSense 3 Different 'Message' fields Graylog Central (peer support) pipeline-rules	5	776	February 11, 2021

Parsing Suricata from OpnSense

Related topics