Parsing Suricata from OpnSense

accidentaladmin · December 21, 2023, 1:22am

Greetings:

I am running OpnSense with Suricata enabled. I have managed to ship Suricata logs to Graylog however the it comes as a big blob.

{"timestamp":"2023-12-20T20:12:30.285101-0500","flow_id":1582594801964295,"in_iface":"igc0","event_type":"alert","vlan":[100],"src_ip":"10.100.0.198","src_port":60794,"dest_ip":"144.217.225.162","dest_port":80,"proto":"TCP","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":2013504,"rev":6,"signature":"ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management","category":"Not Suspicious Traffic","severity":3,"metadata":{"created_at":["2011_08_31"],"former_category":["POLICY"],"updated_at":["2020_04_22"]}},"http":{"hostname":"download.proxmox.com","url":"/debian/pbs-client/dists/bookworm/InRelease","http_user_agent":"Debian APT-HTTP/1.3 (2.6.1)","http_method":"GET","protocol":"HTTP/1.1","status":304,"length":0},"app_proto":"http","flow":{"pkts_toserver":4,"pkts_toclient":3,"bytes_toserver":509,"bytes_toclient":391,"start":"2023-12-20T20:12:30.169223-0500"}}

It appears its all set to parse easily. What would be the best way to parse it at, say, the ,'s, utilizing the strings before the : as the field?

For instance,

{"pkts_toserver":4,"pkts_toclient":3,"bytes_toserver":509,"bytes_toclient":391,"start":"2023-12-20T20:12:30.169223-0500"}

Would become:

pkts_toserver
4
pkts_toclient
3
bytes_toserver
509
bytes_toclient
391
start
2023-12-20T20:12:30.169223-0500

Thank you!

gsmith · December 21, 2023, 1:40am

Hey @accidentaladmin

Happy Holidays.

I personally would use Pipeline with Key Values, (KV)something like this. You may have to adjust for your logs.

rule "KV-Parser"
when
    has_field("message")
then
    set_fields(
                fields:
                    key_value(
                    value: to_string($message.message), 
                    delimiters: ",",
                    kv_delimiters: ":",
                    trim_value_chars: "",
                    trim_key_chars:""
                    )
            );
end

accidentaladmin · December 21, 2023, 2:12am

gsmith:

rule "KV-Parser"
when
    has_field("message")
then
    set_fields(
                fields:
                    key_value(
                    value: to_string($message.message), 
                    delimiters: ",",
                    kv_delimiters: ":",
                    trim_value_chars: "",
                    trim_key_chars:""
                    )
            );
end

Hey G, Happy Holidays to you!

Unfortunately the rule you suggested didn’t work. I think its getting hung up at “value”. I switched over to the new Rule Builder feature and this is what I have so far; I just need to figure out what to put in “value”:

gsmith · December 21, 2023, 2:17am

Oh wow ,
I havent see the new version yet thats kool.

So then the Value would be something like this. Using your filed called application_name.

rule "KV-Parser"
when
    has_field("application_name")
then
    set_fields(
                fields:
                    key_value(
                    value: to_string($message.application_name), 
                    delimiters: ",",
                    kv_delimiters: ":",
                    trim_value_chars: "",
                    trim_key_chars:""
                    )
            );
end

gsmith · December 21, 2023, 2:18am

Normally if you go with to_string it would be the field where the messages are.

Joel_Duffield · December 21, 2023, 2:44am

That appears to be standard JSON format, so the flatten_json function in pipelines should work with it pretty well to transform it into individual fields.

accidentaladmin · December 21, 2023, 2:46am

Yeah, sorry, I should have said that up front.

So how would I use the flatten_json function?

gsmith · December 21, 2023, 5:06am

Hey @accidentaladmin
I have an old pipe, think something like this.

rule "Random User Data Flatten Json Rule"
// From sample data : https://randomuser.me/api/
// Api input path: *
when
    true
then
    let sJson = to_string($message.result);
    let sJson = regex_replace(
        pattern: "^\\[|\\]$",
        value: sJson,
        replacement: ""
        );
    let rsJson = flatten_json(to_string(sJson), "flatten");
    set_fields(to_map(rsJson));
    remove_field("result");
    set_field("message", "parsed user data");
end

accidentaladmin · December 21, 2023, 10:20pm

I feel so dumb when dealing with pipeline runs. Ive stared at that old pipeline rule and can’t make heads or tails of it.

accidentaladmin · December 21, 2023, 11:00pm

So here is where I am at:

rule "Suricata"
when
  has_field(
    field : "application_name"
  )
  AND
  ( has_field("application_name") &&
lowercase(to_string($message."application_name")) == lowercase("suricata")
)

then
  let output_1 = key_value(
    value : "message",
    delimiters : ",",
    kv_delimiters : ":",
    ignore_empty_values : false,
    allow_dup_keys : false,
    trim_key_chars : "\"\"",
    trim_value_chars : "\"\""
  );
  set_fields(
    fields : output_1,
    clean_fields : false
  );
end

But I get this:

Error evaluating action for rule <Suricata/6584c0bfc0330b78cc8fbb7a> (pipeline <Parse Suricata/65839935c0330b78cc8f2c27>) - In call to function 'key_value' at 12:17 an exception was thrown: Missing value for key message

As referenced above, when it comes to pipeline rules I am dumb

gsmith · December 21, 2023, 11:22pm

Hey @accidentaladmin

Oh your all good, I think you have double configurations under the “when” statement.

Here is a better reference, shoothub was pretty good at pipes. I basically get by,Im by no means really good at pipes but I have a library full if need be.

TBH, there is alot of examples in the forum over the years, not sure is you executed a global search here.

gsmith · December 21, 2023, 11:34pm

BTW @accidentaladmin you could use a JSON extractor on th input and enable the tic box for flatten json. Im still working on version 4 so Im not sure if things have changed

accidentaladmin · December 22, 2023, 12:39am

Okay, think I got it (still needs some clean up)

rule "key_value_parser"
when
     contains(
  value: to_string($message."application_name"),
  search: "suricata",
  ignore_case: true
)
then
    set_fields(
        fields:
            key_value(
                value: to_string($message.message),
                delimiters:",",
                kv_delimiters:":",
                trim_key_chars:"\"{}",
                trim_value_chars: "\"{}:[]")
            );
end

Produces this (partial):

From this:

{
  "process_id": "44826",
  "gl2_accounted_message_size": 2122,
  "level": 6,
  "gl2_remote_ip": "192.168.86.1",
  "gl2_remote_port": 14187,
  "streams": [
    "65838d32c0330b78cc8f2609"
  ],
  "gl2_message_id": "01HJ7EGZ0G0188QAJPJJYDYSPZ",
  "source": "OPNsense.thesmiths.management",
  "message": "{\"timestamp\":\"2023-12-21T19:24:10.756644-0500\",\"flow_id\":2023693415535850,\"in_iface\":\"igc0\",\"event_type\":\"alert\",\"vlan\":[100],\"src_ip\":\"10.100.0.198\",\"src_port\":58188,\"dest_ip\":\"144.217.225.162\",\"dest_port\":80,\"proto\":\"TCP\",\"tx_id\":0,\"alert\":{\"action\":\"allowed\",\"gid\":1,\"signature_id\":2013504,\"rev\":6,\"signature\":\"ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management\",\"category\":\"Not Suspicious Traffic\",\"severity\":3,\"metadata\":{\"created_at\":[\"2011_08_31\"],\"former_category\":[\"POLICY\"],\"updated_at\":[\"2020_04_22\"]}},\"http\":{\"hostname\":\"download.proxmox.com\",\"url\":\"/debian/pbs-client/dists/bookworm/InRelease\",\"http_user_agent\":\"Debian APT-HTTP/1.3 (2.6.1)\",\"http_method\":\"GET\",\"protocol\":\"HTTP/1.1\",\"status\":304,\"length\":0},\"app_proto\":\"http\",\"flow\":{\"pkts_toserver\":4,\"pkts_toclient\":3,\"bytes_toserver\":509,\"bytes_toclient\":391,\"start\":\"2023-12-21T19:24:10.644330-0500\"}}",
  "gl2_source_input": "65613c6bd8e1247f71e8771f",
  "sequenceId": "995584",
  "application_name": "suricata",
  "full_message": "<174>1 2023-12-21T19:24:10-05:00 OPNsense.thesmiths.management suricata 44826 - [meta sequenceId=\"995584\"] {\"timestamp\":\"2023-12-21T19:24:10.756644-0500\",\"flow_id\":2023693415535850,\"in_iface\":\"igc0\",\"event_type\":\"alert\",\"vlan\":[100],\"src_ip\":\"10.100.0.198\",\"src_port\":58188,\"dest_ip\":\"144.217.225.162\",\"dest_port\":80,\"proto\":\"TCP\",\"tx_id\":0,\"alert\":{\"action\":\"allowed\",\"gid\":1,\"signature_id\":2013504,\"rev\":6,\"signature\":\"ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management\",\"category\":\"Not Suspicious Traffic\",\"severity\":3,\"metadata\":{\"created_at\":[\"2011_08_31\"],\"former_category\":[\"POLICY\"],\"updated_at\":[\"2020_04_22\"]}},\"http\":{\"hostname\":\"download.proxmox.com\",\"url\":\"/debian/pbs-client/dists/bookworm/InRelease\",\"http_user_agent\":\"Debian APT-HTTP/1.3 (2.6.1)\",\"http_method\":\"GET\",\"protocol\":\"HTTP/1.1\",\"status\":304,\"length\":0},\"app_proto\":\"http\",\"flow\":{\"pkts_toserver\":4,\"pkts_toclient\":3,\"bytes_toserver\":509,\"bytes_toclient\":391,\"start\":\"2023-12-21T19:24:10.644330-0500\"}}",
  "facility_num": 21,
  "gl2_source_node": "d03cc833-8d4e-4ae5-bff7-8aba670a6c2a",
  "_id": "6d6e6503-a060-11ee-b303-7a59a036ff5d",
  "facility": "local5",
  "timestamp": "2023-12-22T00:24:10.000Z"
}

gsmith · December 22, 2023, 12:43am

Nice and thanks for sharing @accidentaladmin

will-ferrell-old-school

system · January 5, 2024, 12:43am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Indexing data from syslog RFC 5424 Graylog Central (peer support) sidecar , nxlog , nodatanx	9	2611	March 4, 2019
Ntop-NG to Graylog Graylog Central (peer support)	4	449	February 13, 2023
Graylog Extractors - BGP Graylog Central (peer support) cisco	2	92	February 24, 2024
Syslog Messages and Parsing Fortinet Graylog Central (peer support)	1	535	July 31, 2019
Reasons to graylog extractor stop working Graylog Central (peer support)	11	2704	April 25, 2017

Parsing Suricata from OpnSense

Related Topics