# Parsing Suricata from OpnSense **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [December 21, 2023, 1:22am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006 "2023-12-21T01:22:40Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![accidentaladmin](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@accidentaladmin](https://community.graylog.org/u/accidentaladmin)\ **Post date:** [December 21, 2023, 1:22am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/1 "2023-12-21T01:22:40Z")
Greetings: I am running OpnSense with Suricata enabled. I have managed to ship Suricata logs to Graylog however the it comes as a big blob. ```auto {"timestamp":"2023-12-20T20:12:30.285101-0500","flow_id":1582594801964295,"in_iface":"igc0","event_type":"alert","vlan":[100],"src_ip":"10.100.0.198","src_port":60794,"dest_ip":"144.217.225.162","dest_port":80,"proto":"TCP","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":2013504,"rev":6,"signature":"ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management","category":"Not Suspicious Traffic","severity":3,"metadata":{"created_at":["2011_08_31"],"former_category":["POLICY"],"updated_at":["2020_04_22"]}},"http":{"hostname":"download.proxmox.com","url":"/debian/pbs-client/dists/bookworm/InRelease","http_user_agent":"Debian APT-HTTP/1.3 (2.6.1)","http_method":"GET","protocol":"HTTP/1.1","status":304,"length":0},"app_proto":"http","flow":{"pkts_toserver":4,"pkts_toclient":3,"bytes_toserver":509,"bytes_toclient":391,"start":"2023-12-20T20:12:30.169223-0500"}} ``` It appears its all set to parse easily. What would be the best way to parse it at, say, the `,`'s, utilizing the strings before the `:` as the field? For instance, ```auto {"pkts_toserver":4,"pkts_toclient":3,"bytes_toserver":509,"bytes_toclient":391,"start":"2023-12-20T20:12:30.169223-0500"} ``` Would become: ```auto pkts_toserver 4 pkts_toclient 3 bytes_toserver 509 bytes_toclient 391 start 2023-12-20T20:12:30.169223-0500 ``` Thank you! ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [December 21, 2023, 1:40am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/2 "2023-12-21T01:40:37Z")
Hey @accidentaladmin Happy Holidays. I personally would use Pipeline with Key Values, (KV)something like this. You may have to adjust for your logs. ```auto rule "KV-Parser" when has_field("message") then set_fields( fields: key_value( value: to_string($message.message), delimiters: ",", kv_delimiters: ":", trim_value_chars: "", trim_key_chars:"" ) ); end ``` ---
**Author:** ![accidentaladmin](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@accidentaladmin](https://community.graylog.org/u/accidentaladmin)\ **Post date:** [December 21, 2023, 2:12am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/3 "2023-12-21T02:12:44Z")
> [@gsmith](#): > > ```auto > rule "KV-Parser" > when > has_field("message") > then > set_fields( > fields: > key_value( > value: to_string($message.message), > delimiters: ",", > kv_delimiters: ":", > trim_value_chars: "", > trim_key_chars:"" > ) > ); > end > > ``` Hey G, Happy Holidays to you! Unfortunately the rule you suggested didn’t work. I think its getting hung up at “value”. I switched over to the new Rule Builder feature and this is what I have so far; I just need to figure out what to put in “value”: ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/f/e/fe6ff47981170ed4f00b2dd26124e1315d2278d7.png) ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [December 21, 2023, 2:17am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/4 "2023-12-21T02:17:08Z")
Oh wow , I havent see the new version yet thats kool. So then the Value would be something like this. Using your filed called application\_name. ```auto rule "KV-Parser" when has_field("application_name") then set_fields( fields: key_value( value: to_string($message.application_name), delimiters: ",", kv_delimiters: ":", trim_value_chars: "", trim_key_chars:"" ) ); end ``` ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [December 21, 2023, 2:18am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/5 "2023-12-21T02:18:56Z")
> [@gsmith](#): > > `value: to_string($message.application_name),` Normally if you go with **to\_string** it would be the field where the messages are. ---
**Author:** ![Joel\_Duffield](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Joel\_Duffield](https://community.graylog.org/u/Joel_Duffield)\ **Post date:** [December 21, 2023, 2:44am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/6 "2023-12-21T02:44:34Z")
That appears to be standard JSON format, so the flatten\_json function in pipelines should work with it pretty well to transform it into individual fields. ---
**Author:** ![accidentaladmin](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@accidentaladmin](https://community.graylog.org/u/accidentaladmin)\ **Post date:** [December 21, 2023, 2:46am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/7 "2023-12-21T02:46:53Z")
Yeah, sorry, I should have said that up front. So how would I use the flatten\_json function? ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [December 21, 2023, 5:06am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/8 "2023-12-21T05:06:26Z")
Hey @accidentaladmin I have an old pipe, think something like this. ```auto rule "Random User Data Flatten Json Rule" // From sample data : https://randomuser.me/api/ // Api input path: * when true then let sJson = to_string($message.result); let sJson = regex_replace( pattern: "^\\[|\\]$", value: sJson, replacement: "" ); let rsJson = flatten_json(to_string(sJson), "flatten"); set_fields(to_map(rsJson)); remove_field("result"); set_field("message", "parsed user data"); end ``` ---
**Author:** ![accidentaladmin](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@accidentaladmin](https://community.graylog.org/u/accidentaladmin)\ **Post date:** [December 21, 2023, 10:20pm UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/9 "2023-12-21T22:20:49Z")
I feel so dumb when dealing with pipeline runs. Ive stared at that old pipeline rule and can’t make heads or tails of it. ---
**Author:** ![accidentaladmin](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@accidentaladmin](https://community.graylog.org/u/accidentaladmin)\ **Post date:** [December 21, 2023, 11:00pm UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/10 "2023-12-21T23:00:37Z")
So here is where I am at: ```auto rule "Suricata" when has_field( field : "application_name" ) AND ( has_field("application_name") && lowercase(to_string($message."application_name")) == lowercase("suricata") ) then let output_1 = key_value( value : "message", delimiters : ",", kv_delimiters : ":", ignore_empty_values : false, allow_dup_keys : false, trim_key_chars : "\"\"", trim_value_chars : "\"\"" ); set_fields( fields : output_1, clean_fields : false ); end ``` But I get this: ```auto Error evaluating action for rule (pipeline ) - In call to function 'key_value' at 12:17 an exception was thrown: Missing value for key message ``` As referenced above, when it comes to pipeline rules I am dumb 🤪 ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [December 21, 2023, 11:22pm UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/11 "2023-12-21T23:22:22Z")
Hey @accidentaladmin Oh your all good, I think you have double configurations under the “when” statement. Here is a better reference, shoothub was pretty good at pipes. I basically get by,Im by no means really good at pipes but I have a library full if need be. > [@Issue with Key=Value Parser Pipeline for specific log-messages](https://community.graylog.org/t/issue-with-key-value-parser-pipeline-for-specific-log-messages/19606/2): > > Try to use this, I’ve added parameters to remove "{} from key and values. But, it’s not perfect, because of Country contains json. rule "key\_value\_parser" when has\_field("Full\_Response") then set\_fields( fields: key\_value( value: to\_string($message.Full\_Response), delimiters:",", kv\_delimiters:":", trim\_key\_chars:"\"", trim\_value\_chars: "\"{}") ); end TBH, there is alot of examples in the forum over the years, not sure is you executed a global search here. ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [December 21, 2023, 11:34pm UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/12 "2023-12-21T23:34:59Z")
BTW @accidentaladmin you could use a JSON extractor on th input and enable the tic box for flatten json. Im still working on version 4 so Im not sure if things have changed ---
**Author:** ![accidentaladmin](https://avatars.discourse-cdn.com/v4/letter/a/7993a0/32.png) [@accidentaladmin](https://community.graylog.org/u/accidentaladmin)\ **Post date:** [December 22, 2023, 12:39am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/13 "2023-12-22T00:39:28Z")
Okay, think I got it (still needs some clean up) ```auto rule "key_value_parser" when contains( value: to_string($message."application_name"), search: "suricata", ignore_case: true ) then set_fields( fields: key_value( value: to_string($message.message), delimiters:",", kv_delimiters:":", trim_key_chars:"\"{}", trim_value_chars: "\"{}:[]") ); end ``` Produces this (partial): ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/6/d/6d0079e953f9c52076c2b61a6704a5bd7736e38f.png) From this: ```auto { "process_id": "44826", "gl2_accounted_message_size": 2122, "level": 6, "gl2_remote_ip": "192.168.86.1", "gl2_remote_port": 14187, "streams": [ "65838d32c0330b78cc8f2609" ], "gl2_message_id": "01HJ7EGZ0G0188QAJPJJYDYSPZ", "source": "OPNsense.thesmiths.management", "message": "{\"timestamp\":\"2023-12-21T19:24:10.756644-0500\",\"flow_id\":2023693415535850,\"in_iface\":\"igc0\",\"event_type\":\"alert\",\"vlan\":[100],\"src_ip\":\"10.100.0.198\",\"src_port\":58188,\"dest_ip\":\"144.217.225.162\",\"dest_port\":80,\"proto\":\"TCP\",\"tx_id\":0,\"alert\":{\"action\":\"allowed\",\"gid\":1,\"signature_id\":2013504,\"rev\":6,\"signature\":\"ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management\",\"category\":\"Not Suspicious Traffic\",\"severity\":3,\"metadata\":{\"created_at\":[\"2011_08_31\"],\"former_category\":[\"POLICY\"],\"updated_at\":[\"2020_04_22\"]}},\"http\":{\"hostname\":\"download.proxmox.com\",\"url\":\"/debian/pbs-client/dists/bookworm/InRelease\",\"http_user_agent\":\"Debian APT-HTTP/1.3 (2.6.1)\",\"http_method\":\"GET\",\"protocol\":\"HTTP/1.1\",\"status\":304,\"length\":0},\"app_proto\":\"http\",\"flow\":{\"pkts_toserver\":4,\"pkts_toclient\":3,\"bytes_toserver\":509,\"bytes_toclient\":391,\"start\":\"2023-12-21T19:24:10.644330-0500\"}}", "gl2_source_input": "65613c6bd8e1247f71e8771f", "sequenceId": "995584", "application_name": "suricata", "full_message": "<174>1 2023-12-21T19:24:10-05:00 OPNsense.thesmiths.management suricata 44826 - [meta sequenceId=\"995584\"] {\"timestamp\":\"2023-12-21T19:24:10.756644-0500\",\"flow_id\":2023693415535850,\"in_iface\":\"igc0\",\"event_type\":\"alert\",\"vlan\":[100],\"src_ip\":\"10.100.0.198\",\"src_port\":58188,\"dest_ip\":\"144.217.225.162\",\"dest_port\":80,\"proto\":\"TCP\",\"tx_id\":0,\"alert\":{\"action\":\"allowed\",\"gid\":1,\"signature_id\":2013504,\"rev\":6,\"signature\":\"ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management\",\"category\":\"Not Suspicious Traffic\",\"severity\":3,\"metadata\":{\"created_at\":[\"2011_08_31\"],\"former_category\":[\"POLICY\"],\"updated_at\":[\"2020_04_22\"]}},\"http\":{\"hostname\":\"download.proxmox.com\",\"url\":\"/debian/pbs-client/dists/bookworm/InRelease\",\"http_user_agent\":\"Debian APT-HTTP/1.3 (2.6.1)\",\"http_method\":\"GET\",\"protocol\":\"HTTP/1.1\",\"status\":304,\"length\":0},\"app_proto\":\"http\",\"flow\":{\"pkts_toserver\":4,\"pkts_toclient\":3,\"bytes_toserver\":509,\"bytes_toclient\":391,\"start\":\"2023-12-21T19:24:10.644330-0500\"}}", "facility_num": 21, "gl2_source_node": "d03cc833-8d4e-4ae5-bff7-8aba670a6c2a", "_id": "6d6e6503-a060-11ee-b303-7a59a036ff5d", "facility": "local5", "timestamp": "2023-12-22T00:24:10.000Z" } ``` ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [December 22, 2023, 12:43am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/14 "2023-12-22T00:43:07Z")
Nice and thanks for sharing @accidentaladmin ![will-ferrell-old-school](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/8/7/876a271624f400c63dd737b63beff1124d110633.gif) ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [January 5, 2024, 12:43am UTC](https://community.graylog.org/t/parsing-suricata-from-opnsense/31006/15 "2024-01-05T00:43:23Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.