Rsyslog -> syslog input fails parsing facility correctly sometimes

Graylog Central (peer support)
, ,
1

I’m trying to debug an issue where some messages have the field facility set correctly while others fail to set a correct facility.

Example of correctly parsed facility

Parsed facility: security/authorization
Parsed facility_num: 10
Full message from syslog host:
<86>1 2021-09-22T16:11:34.578583+02:00 mylogclient sudo - - - pam_unix(sudo:session): session closed for user root

Example of incorrectly parsed facility

Parsed facility: type=USER_END msg=audit(1632319894.575
Parsed facility_num: 1
Full message from syslog host:
<14>1 2021-09-22T16:11:34.579213+02:00 mylogclient audispd - - - node=mylogclient type=USER_END msg=audit(1632319894.575:2457): pid=26515 uid=0 auid=1000 ses=60 subj==unconfined msg='op=PAM:session_close grantors=pam_permit,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'

image
image1444×526 50.8 KB

How do I proceed debugging this issue?

Environmental information

All messages in this example are coming from the same host which is running Debian 10 with rsyslogd 8.1901.0 (aka 2019.01) using the following configuration:

$PreserveFQDN on                         # fixes FDQN issue

$ActionQueueType            LinkedList   # use asynchronous processing
$ActionQueueFileName        remote       # unique name prefix for spool files
$ActionQueueMaxFileSize     50m
$ActionQueueMaxDiskSpace    512m         # 512M space limit (use as much as possible)
$ActionResumeRetryCount     -1           # infinite retries if host is down
$ActionQueueSaveOnShutdown  on           # save messages to disk on shutdown
$ActionQueueTimeoutEnqueue  0            # Drop messages with no wait when they can’t be queued

*.*     @@mylogserver;RSYSLOG_SyslogProtocol23Format

Package versions

The graylog server is running Debian 11.

  • Graylog 4.1.5-1 (installed from graylog repository)
  • MongoDB 4.4.9 (installed from mongodb repository)
  • Elasticsearch 7.15.0 (installed from elasticsearch repository)
2

As a new user I couldn’t embed more than 1 picture. Sending in the picture of the correctly parsed message.

image
image1419×469 28 KB

3

Hello && welcome

Need to ask a couple of questions.

  • What type of Input are you using to ingest logs?
  • Are you aware of the supported version of Elasticsearch?
  • Do you have any extractors/pipeline created on you Input?

image
image972×463 34.3 KB

I personally haven’t tried version 7.15 yet

4

Syslog TCP and/or UDP. Both have the same result.

Thank you, I actually wasn’t aware, I have removed Elasticsearch 7 and installed Elasticsearch 6.8.19 from Elasticsearch repo, removed indexes and started with new indexes, still the same.

Yes, but none of them should adjust the facility field. Just to be on the safe side I have removed all extractors from the Syslog TCP input and removed all pipeline connections, looks like this works. Gonna check all sorts of messages and report back.

5

Hello,

I was asking about extractors/pipelines was because the field data you shown above looks like it truncate.

Parsed facility_num: 10 
Parsed facility_num: 1

As for your Rsyslog I assume you are aware of these types of configurations below. Reason I ask was I’m not sure when you changed Inputs types that you changed you Rsyslog configuration file when testing.

*.* @@somehost:5140  //UDP connection     
*.* @somehost:5140  //TCP  connection

For troubleshooting purposes, have you tried to use different log shippers (i.e. Nxlog, FileBeat)? If so do you get the same results?

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.