Cisco FTD logging messages being ignored and wrong source

[pepperoni-pi]

Description of your problem

I set up a Graylog demo for myself and it has been going pretty well. The main issues that I am encountering are related to Cisco FTDs (which are essentially virtual ASAs). The Syslog UDP input has trouble indexing the various parts of the messages or even just ignores the logs entirely. Is there a way to see if logs are being discarded? The source is often indexed improperly and ends up showing as the month (Aug/Sep) instead of the source IP or hostname.

#Captured by Sylog UDP input but using a packet capture I found that it doesn’t have PRI field so no facility or log level are found leaving them as Unknown and -1.
Sep 02 2021 13:05:44 192.168.22.1 %FTD-6-430002: DeviceUUID: a12a9578-56ab-11eb-8010-87b13c2c7c5d, AccessControlRuleAction: Allow, SrcIP: 192.168.2.213, DstIP: 10.10.10.10, SrcPort: 61243, DstPort: 49680, Protocol: tcp, IngressInterface: outside, EgressInterface: inside, IngressZone: outside-fw1, EgressZone: inside-fw1, ACPolicy: Firewall, AccessControlRuleName: outside-in-1, Prefilter Policy: Default Prefilter Policy, User: No Authentication Required, InitiatorPackets: 2, ResponderPackets: 1, InitiatorBytes: 120, ResponderBytes: 66, NAPPolicy: Balanced Security and Connectivity

#Not captured by Syslog UDP input and does have PRI field
<190>Sep 02 14:04:59 192.168.22.1 : %FTD-6-302016: Teardown UDP connection 13584982 for outside:192.168.2.213/61654(LOCAL\user1) to inside:10.10.10.10/53 duration 0:00:00 bytes 0 (user1)

Description of steps you’ve taken to attempt to solve the issue

For now, I’m using Raw/Plaintext UDP and seeing all of the messages, but unfortunately nothing is indexed using this input and I’ll have to create all of the indexing myself. Obviously, this is not ideal especially because I can’t seem to find a way to index the portions in the PRI field (log level and facility). It would be nice to use the Syslog UDP input because it already handles this except that it seems to be discarding a large number of messages sent by the FTDs. To address the source issue, I had to create a pipeline to write the gl2_remote_ip as the source and then attempt a reverse dns lookup to convert it to the FQDN if it exists. Then an extractor removes the domain leaving just the hostname.

Environmental information

Integration with LibreNMS. Mostly Cisco devices.

Operating system information

Ubuntu 20.04

Package versions

Graylog - 4.1.3+9d79c05
MongoDB - v4.0.26
Elasticsearch - 7.10.2

[gsmith]

Hello && Welcome

When setting up our Input for Cisco Switches we had the same problems with Syslog UDP and just like yourself we switched to Raw/Plaintext UDP. I’ll try to answer some of your questions.

@pepperoni-pi

Is there a way to see if logs are being discarded?

On the Graylog Server, not that I know of. Basically, I had to do the same as yourself, run a packet capture.

@pepperoni-pi

unfortunately nothing is indexed using this input and I’ll have to create all of the indexing myself.

Could you elaborate further on this? Are you referring to the Syslog UDP input or Raw/Plaintext?

@pepperoni-pi

Obviously, this is not ideal especially because I can’t seem to find a way to index the portions in the PRI field (log level and facility).

There are probably a couple ways of doing this. What we did was use the Raw/Paintex UDP input. Then we created some regex extractors to have fields that were needed (log level and facility) for searching, dashboards, etc… I seen you are using a pipeline, that is another option if preferred. We had done the same with a pipeline for FQDN. One of our problems was settings. This was in System/Configurations. We had to adjusting our order in which these processors were applied.

Example:

Maybe something here can help

the NEW Marketplace

Sorry I cant give you a direct answer on the Syslog UDP input.

[shoothub]

Hi @pepperoni-pi
Cisco facility and serverity is also contained in messages, they uses syntax: %facility-severity-MNEMONIC:description

In case of FTD, facility is always FTD and severity is number from 1 - 7
So you can use this simple GROK to parse it:

\\%FTD-%{DATA:ftd_severity:int}-%{DATA:ftd_messageid}: %{GREEDYDATA:cisco_msg}

Cisco ASA 5500 Series Configuration Guide using the CLI, 8.4 and 8.6 -...

Cisco ASA Series Syslog Messages - Messages Listed by Severity Level...

Try to play with default syslog format or EMBLEM format.

Cisco ASA 5500 Series Configuration Guide using the CLI, 8.4 and 8.6 -...

If you want to include hostname of FTD in messages non EMBLEM format, configure it in cisco:
logging device-id hostname instead of ipaddress

Cisco ASA 5500 Series Configuration Guide using the CLI, 8.4 and 8.6 -...

Then you can extend GROK to include also hostname and save it as source.

[pepperoni-pi]

@gsmith

@gsmith

@pepperoni-pi

unfortunately nothing is indexed using this input and I’ll have to create all of the indexing myself.

Could you elaborate further on this? Are you referring to the Syslog UDP input or Raw/Plaintext?

I was referring to the Raw/Plaintext input. It only indexes the source and message fields.

@gsmith

@pepperoni-pi

Obviously, this is not ideal especially because I can’t seem to find a way to index the portions in the PRI field (log level and facility).

There are probably a couple ways of doing this. What we did was use the Raw/Paintex UDP input. Then we created some regex extractors to have fields that were needed (log level and facility) for searching, dashboards, etc… I seen you are using a pipeline, that is another option if preferred. We had done the same with a pipeline for FQDN. One of our problems was settings. This was in System/Configurations. We had to adjusting our order in which these processors were applied.

The problem I’m running into is that I can’t seem to find a way to extract the facility and log level from the syslog packet header. RFC5424. Clip from Wireshark capture:

The log level could be extracted from the “FTD-6-302016” portion, but the facility isn’t in the message portion of the packet. Is there another variable besides $message available in pipeline rules or a sub level like $message.header that contains the eight bit header value?

Thanks for letting me know about the Graylog Marketplace. I’m new to Graylog so I was not aware that even existed.

[pepperoni-pi]

@shoothub

@shoothub

Cisco facility and serverity is also contained in messages, they uses syntax: %facility-severity-MNEMONIC:description

In case of FTD, facility is always FTD and severity is number from 1 - 7

But FTD is not the facility. Facility is a number between 0 - 23 (RFC5424) that is found in the packet header. The log level can be extracted from “FTD-6-302016”, but the facility can only be found in the packet header. I’m trying to figure out what variable contains the packet header. In pipeline rules, I’m using things like $message.gl2_remote_ip. Is there a variable that contains the 8-bit header like $message.header?

[gsmith]

Hello,
I was researching this issue and came across this. Not sure if you seen this yet.

Configure Global Syslog Configuration

I was wondering is how you configured your Cisco setting for logging?

[pepperoni-pi]

@gsmith

Configure Global Syslog Configuration

That page looks familiar. Pretty sure I’ve used that when setting up the syslog configuration on the firewall. The firewall configuration is not the issue though. What I’m trying to figure out is how to extract the the facility and log level from the header.

[pepperoni-pi]

I’m giving up on the default Syslog UDP input and switching to Raw/Plaintext UDP and pipelining out all the data I need. I’m going to start a new thread just around getting the facility and log level out of the header.