Cisco FTD message extractor

[jonas_ibk]

Hey Guys,
im trying to split following Cisco FTD message with grok extractors:

%FTD-4-430003: EventPriority: Low, DeviceUUID: bawezh38-d5ewf-113c-8a08-adwde6ac, InstanceID: 3, FirstPacketSecond: 2024-09-04T10:06:25Z, ConnectionID: 21556, AccessControlRuleAction: Block, AccessControlRuleReason: SSL Block, SrcIP: 161.23.161.5, DstIP: 30.193.157.68, SrcPort: 62123, DstPort: 123, Protocol: tcp, IngressInterface: inside, EgressInterface: outside, IngressZone: inside, EgressZone: outside, IngressVRF: Global, EgressVRF: Global, ACPolicy: HDL, AccessControlRuleName: Web, Prefilter Policy: PreFilter_Std, Client: SSL client, ApplicationProtocol: HTTPS, WebApplication: Azure Authentication Service, ConnectionDuration: 0, InitiatorPackets: 3, ResponderPackets: 3, InitiatorBytes: 462, ResponderBytes: 2934, NAPPolicy: Balanced Security and Connectivity, SSLPolicy: SSL_Std, SSLRuleName: Block_unwanted, SSLFlowStatus: Success, SSLCipherSuite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, SSLCertificate: 7b85lk085abc854ecdd16d8ea2813658bbee187b, SSLVersion: TLSv1.2, SSLServerCertStatus: Valid, SSLActualAction: Block, SSLExpectedAction: Block, SSLSessionID: fbe30bca1387f700aa03865d4da8bc7836f571ed850ecda01e8e03931cf63cfc, URL: https://login.live.com, NAT_InitiatorPort: 62453, NAT_ResponderPort: 441, NAT_InitiatorIP: 154.78.20.232, NAT_ResponderIP: 31.195.194.68, EVE_Process: zscaler tunnel, EVE_ProcessConfidencePct: 44, EVE_ThreatConfidencePct: 0, EVE_ThreatConfidenceIndex: 1, ClientAppDetector: AppID

3. What steps have you already taken to try and solve the problem?

My Problem are not the grok patterns, i use Grok Debugger and the splitting works perfectly, even in the simulation in graylog, it works. But as soon as i save the extractor, the message stops apearing in my Stream. Even more confusing is, when i only extract the SrcIP with grok, it works, but as soon as i try to extract more from the message, new messages stop coming in.

I already tried old extractors, from old posts, but they are all 5-8 years old and dont work for me.

I also tried this Pipeline rule:
https://community.graylog.org/t/creating-a-log-extractor-for-key-value-key-value-logs/20820/2
it’s also not working, i can see that the messages are being put through the pipeline, but dont appear in the Stream.

4. How can the community help?

how do you guys extract cisco logs, do you have working pipeline Rules? Or are you using Extractors? I am having the same problems with splitting my Netscaler messages. It’s obviously some step i am missing, please help

[Wine_Merchant]

Hello @jonas_ibk,

Best to stick with pipelines rules.

The below rule appears to work on the message you provided, do you see any parsing errors within your /var/log/graylog/server.log file?

rule "Key Value"
when
true
then

set_fields(
   fields:key_value(
   value: to_string($message.message),
   delimiters:",",
   kv_delimiters:":"
)
);

[jonas_ibk]

Hey, firstly thanks for your reply @Wine_Merchant ,

i tried pipeline rules and they all work in the simulation, i tried your rule and the same problem arised, it split the message like i want but the messages stop showing up in the Stream.

I looked into the server file and found an errorr:

It was the same type of message from cisco ftd.

i was just trying to extract the DstIP with Grok pattern (DstIP: %{IPV4:Dst_IP}), im not sure how to fix the wrong date format, do you know how?

Edit: I just turned on your pipeline rule and the same error occurred.

[Wine_Merchant]

The error point to an issue with the date failing to process, are these messages arriving on a raw input and if no extractors or pipelines are enabled do you still see the error in the logs?

[jonas_ibk]

these messages are coming through a Linux-Syslog Input and without anything enabled the error does not occurr

[Wine_Merchant]

Just ran through a quick test sending your provided message to a syslog TCP input via the below

echo -e "%FTD-4-430003: EventPriority: Low, DeviceUUID: bawezh38-d5ewf-113c-8a08-adwde6ac, InstanceID: 3, FirstPacketSecond: 2024-09-04T10:06:25Z, ConnectionID: 21556, AccessControlRuleAction: Block, AccessControlRuleReason: SSL Block, SrcIP: 161.23.161.5, DstIP: 30.193.157.68, SrcPort: 62123, DstPort: 123, Protocol: tcp, IngressInterface: inside, EgressInterface: outside, IngressZone: inside, EgressZone: outside, IngressVRF: Global, EgressVRF: Global, ACPolicy: HDL, AccessControlRuleName: Web, Prefilter Policy: PreFilter_Std, Client: SSL client, ApplicationProtocol: HTTPS, WebApplication: Azure Authentication Service, ConnectionDuration: 0, InitiatorPackets: 3, ResponderPackets: 3, InitiatorBytes: 462, ResponderBytes: 2934, NAPPolicy: Balanced Security and Connectivity, SSLPolicy: SSL_Std, SSLRuleName: Block_unwanted, SSLFlowStatus: Success, SSLCipherSuite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, SSLCertificate: 7b85lk085abc854ecdd16d8ea2813658bbee187b, SSLVersion: TLSv1.2, SSLServerCertStatus: Valid, SSLActualAction: Block, SSLExpectedAction: Block, SSLSessionID: fbe30bca1387f700aa03865d4da8bc7836f571ed850ecda01e8e03931cf63cfc, URL: https://login.live.com, NAT_InitiatorPort: 62453, NAT_ResponderPort: 441, NAT_InitiatorIP: 154.78.20.232, NAT_ResponderIP: 31.195.194.68, EVE_Process: zscaler tunnel, EVE_ProcessConfidencePct: 44, EVE_ThreatConfidencePct: 0, EVE_ThreatConfidenceIndex: 1, ClientAppDetector: AppID" | nc -w 1 192.168.69.3 2020

I have one pipeline with the rule provided earlier attached to the stream

And with this the messages are split into key values

Is this setup the same as what you are working with?

[jonas_ibk]

its syslog UDP sorry if i left that out, but other than that, yes

[Wine_Merchant]

Gave it a run through with a syslog UDP input just in case but appears the same, what if you try using a raw UDP on the same port and then implementing the pipelines?

[jonas_ibk]

i have to run it by some coworkers, but i will try that , will probably get back to you

[jonas_ibk]

So changing the Input did not change anything, the messages still do not appear in the stream after applying the rule to the pipeline. But in the simulation the message is splitted perfectly in key value pairs. it seems like the messages are not routed back into the stream.

i even made a new rule for the latest stage, saying “Route message to stream” and sending it to the same or another stream, both didnt work.

Any more ideas on how to slove this?

[Wine_Merchant]

What does the processing order look like with your system, reference below

Are you still only seeing errors in the server log when you have the stream connected to a pipeline?

There are no rules within the pipeline that attempt to format the date?