Configuring Rsyslog

gsmith · November 23, 2017, 2:45am

Hello All,
Having troubles with Rsyslog TLS/SSL Configuration on Linux clients send messages to graylog server with certificates.

Environment;
Total of 6 CentOS 7.3 Servers minimal install.
3 Servers with Graylog version 2.3 and Mongo version 3.4 ‘Clustered’
3 Servers with Elasticsearch 5.6.4 ‘Clustered’

Client Rsyslog Version;
rsyslog-8.24.0-12.el7.x86_64

Graylog Input

Rsyslog.conf file on Linux Client

#### MODULES #### The imjournal module bellow is now used as a message source instead of imuxsock.

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
Provides UDP syslog reception
#$ModLoad imudp
#$ModLoad imudp
#$UDPServerRun 514

Provides TCP syslog reception
#$ModLoad imtcp
#InputTCPServerRun 514

GLOBAL DIRECTIVES
Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

File syncing capability is disabled by default. This feature is usually not required,
not useful and an extreme performance hit
#$ActionFileEnableSync on

Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
Turn off message reception via local log socket;
local messages are retrieved through imjournal now.
$OmitLocalLogging on

File to store the position in the journal
$IMJournalStateFile imjournal.state

RULES
Log all kernel messages to the console.
Logging much else clutters up the screen
#kern.* /dev/console

Log anything (except mail) of level info or higher.
Don’t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

The authpriv file has restricted access.
authpriv.* /var/log/secure

Log all the mail messages in one place.
mail.* -/var/log/maillog

Log cron stuff
cron.* /var/log/cron

Everybody gets emergency messages
.emerg :omusrmsg:

Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

Save boot messages also to boot.log
local7.* /var/log/boot.log

Begin forwarding rule
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down

Set gtls driver
$DefaultNetstreamDriver gtls

certificate files
$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/cert.pem
$DefaultNetstreamDriverCertFile /etc/pki/tls/certs/graylog-certificate.pem
$DefaultNetstreamDriverKeyFile /etc/pki/tls/certs/graylog-key.pem
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *..net
$ActionSendStreamDriverMode 1 # run driver in TLS-only mode

Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#. @@remote-host:514
#. @@remote-host:514
. @@lab-graylog-001..net:51423
end of the forwarding rule

Rsyslog.log from Linux Client;

Rsyslog Status running

Graylog Server Logs

I tried to follow these links, but was unsuccessful;

http://kb.kristianreese.com/index.php?View=entry&EntryID=148

github.com

Graylog2/graylog-guide-syslog-linux/blob/master/README.md#rsyslog

## Sending syslog from Linux systems into Graylog

The two most popular syslog deamons (the programs that run in the background to accept and write or forward logs) are [rsyslog](https://www.rsyslog.com/) and [syslog-ng](https://syslog-ng.com/). One of these will most likely be running on your Linux distribution.

Please refer to the documentation of your distribution if you are not sure about this.

### ⚠️ Warning ⚠️

**These instructions configure rsyslog and syslog-ng to send log messages _unencrypted_ over the network. This is generally not recommended on public networks.**

### rsyslog

Forwarding syslog messages with rsyslog is easy. The only important thing to get the most out of your logs is following
[RFC 5424](http://www.ietf.org/rfc/rfc5424.txt). The following examples configure your `rsyslog` daemon to send
RFC 5424 date to Graylog syslog inputs:

##### UDP:

    *.* @graylog.example.org:514;RSYSLOG_SyslogProtocol23Format

This file has been truncated. show original

Wondering if some one could enlighten me where I’m going wrong. Perhaps I over looked something?
Thanks in advance. I tried to follow these links, but was unsuccessful;

rfinney · November 23, 2017, 5:01am

The last log that points to rsyslog.com/e/2078 indicates a permission issue on the cert Graylog created for the input. Have you checked to see whatever user Graylog is running under has permissions to the cert? Since Graylog created the cert it should, but something might have changed if it was touched with a file editor or something else.

Also, is it possible to use a valid signed cert and test? Some vendors such as digicert allow for unlimited duplicate certs. I’ve run into issues with self signed certs in the past.

rfinney · November 23, 2017, 5:10am

Just realized it was rsyslog throwing the error I mentioned, but I noticed Graylog complaining it couldn’t find your TLS key file. I would check the permissions on that.

As of that log entry it isn’t using your key for that input.

gsmith · November 24, 2017, 12:12am

@rfinney
Thank you for your quick response.

I fixed my input on Graylog Server. I’m using one that works previously for my Windows machines, as shown below;
NOTE: What’s weird is that I just copied this input to make one for my Linux machines and all I did was change the Port Number and Name.

I used SCP command for certificate file/s from Graylog Server to Remote Linux client.
I did change Certificate location as show below;
NOTE:They used to be located in DIR /etc/pki/tls/certs/

On the Remote Linux Machine/s I have tried to set the permission on certificates and Directories to 777 (I know, not a good Idea) , but for testing purposes, this still did not work.

Just a side note I dropped my firewall and disabled Selinux.

The error on Rsyslog Status stays the same and no data In Graylog Input (test-tcp) from remote Linux clients. I know the TCP/TLS Input works because I have machine’s sending data to that Input as we speak.
Following error on Rsyslog Status;

I believe there is a permission/authentication issue as stated from above, but where? I was looking into configurations in NXLOG on Windows machines, and In the Configuration file I set the Pass Key for the certs. During my research I did not find how to config rsyslog.conf for a pass key.

Thank you for you help in advance.

gsmith · November 25, 2017, 1:21am

To Summarize;
Created an Input Called Linux-TCP, using TLS cert, key, and password configuration.
Transferred Certificates using SCP from Graylog Server to a remote Linux server.
Certificate are placed in /etc/pki/rsyslog/ directory on remote Linux server.

Configure Remote Linux Server rsyslog.conf as;
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down

#driver
$DefaultNetstreamDriver gtls

#certificate files
$DefaultNetstreamDriverCAFile /etc/pki/rsyslog/cert.pem
$DefaultNetstreamDriverCertFile /etc/pki/rsyslog/graylog-cert.pem
$DefaultNetstreamDriverKeyFile /etc/pki/rsyslog/graylog-key.pem

#actions
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer lab-graylog-001.nseva-labs.net
$ActionSendStreamDriverMode 1 # run driver in TLS-only mode

#remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*. * @@lab-graylog-001.enseva-labs.net:6514

Restarted Rsyslog Service
No Messages came through Linux-TCP Input (fixed Input error “TLS key file or certificate file does not exist, creating a self-signed certificate for input”) There was a space in full path.

Rsyslog Status error “unexpected GnuTLS error -24 in nsd_gtls.c:205: Decryption has failed.”.
I check permissions on Certificates, and directory’s.
I went as far as giving everyone excess to certs and directory’s, No Joy.
Do I need to make Cert’s on remote Linux server, then transfer them to Graylog server?
Looking for a way on the remote Linux server to read Graylog Certs’, I think that is where the problem located, but I’m unsure. If so how do I get rsyslog to read these cert’s?

NOTE: I tired just using TCP Connection without Certs, no problems occurred, messages came through.

jochen · November 27, 2017, 8:13am

What’s the output of the following command?

namei -l /etc/pki/rsyslog/graylog-key.pem

What’s the system user running Graylog?

gsmith · November 27, 2017, 11:01pm

This is from the Remote CentOS 7 FTP Server

System user running Graylog

gsmith · November 28, 2017, 3:58am

I’m unsure if this information would help Identify my problem, but I execute the following command on Graylog Certificate from the Remote CentOS 7 (ftp-server) ;
root# curl -v --cacert graylog-certificate.pem https://lab-graylog-001.enseva-labs.net:9000

jochen · November 28, 2017, 8:35am

There’s your problem. The directory /etc/pki/rsyslog/ is lacking the executable bit (see Execute vs Read bit. How do directory permissions in Linux work? - Unix & Linux Stack Exchange) and the file /etc/pki/rsyslog/graylog-key.pem is only readable for the user “root”.

gsmith · November 29, 2017, 12:56am

@jochen

I execute the new configuration as suggested

Graylog-Server Input for linux-tcp/tls looks like it started up fine as shown below;

Input Config

Remote Linux Server Rsyslog.service status does show same error;
“unexpected GnuTLS error -24 in nsd_gtls.c:205: Decryption has failed.”

I’m using the same certificates made on the Graylog server that Windows OS is using in NXLOG. No problems are occurring for the Syslog-TCP input, with TLS enabled.

So i research the error URL provided it states the following;

http://kb.monitorware.com/kbeventdb-detail-id-6789.html
"This message occurs with TLS netstream driver. For TLS, certificates (.pem files) are needed to provide security credentials.
This error is issued if there is a problem with these files. The message contains a more precise error description. That error text is taken directly from the underlying TLS library.
A common cause is that the file can not be found or accessed (permissions!). In that case, a rsyslogd-2040 error will follow.

This is my rsyslog.conf file, not sure what drivers i would need.

Is rsyslog service unable to use graylog Certificates?
Or did I not give the correct ownership to cert files?
NOTE:I did try to make graylog users and chown cert files, but that did not seem to work, same errors occurred

gsmith · November 30, 2017, 6:13am

@jochen
I’ve been trying to every permission for the remote Linux server Cert’s and nothing seems to work. I do realize its on the Remote Linux server, rsyslog service. When I disable TLS in the input and comment out
"# $ActionSendStreamDriverMode 1 # run driver in TLS-only mode"
Graylog starts receiving messages, obviously messages are only being sent through TCP,
Any other Ideas would be appreciated.

gsmith · December 2, 2017, 6:01pm

@jochen
I might have found my solution about the error from rsyslog,
Creating a different Certificate/s for my Linux servers.

gsmith · December 6, 2017, 1:46am

I created new Certificates for the remote Linux Servers, No joy. Same error occurred;

“unexpected GnuTLS error -24 in nsd_gtls.c:205: Decryption has failed.”

So, I have Isolated the problem to the Remote Linux Server using Rsyslog. Unable to use Working Certificates I made for Graylog Cluster. Any help would be appreciated
Thank in advance.

gsmith · December 6, 2017, 3:40am

After researching ,think I found the problem why Graylog Certs are not working for Linux Servers.
http://www.rsyslog.com/doc/v8-stable/tutorials/tls.html#certificates

To sum it up,
I was using “Openssl” and “keytool” commands not “certtools”. This might be culprit, but still unsure.

Tried making certificates from the following
http://www.rsyslog.com/doc/v8-stable/tutorials/tls.html#ca-certificate

This corrected the error from Rsyslog on the remote Linux Server., But graylog was unable to use these certs in the Input.
Think I’m getting closer to a solution again.
Trying to find another solution/s on how to use Certificates that Linux Servers and Graylog Servers are able to use together. perhaps an example what other peoples use for TCP/TLS

gsmith · December 8, 2017, 1:39am

Well… I gave up on using Rsyslog. I’ve tried to make certs that work for Rsyslog, but that did not work for Graylog. Tried converting those certs so graylog would be able to use them, No joy
I’ve joined https://kb.monitorware.com (i.e. was a bad experience to even register), Still no Joy
I reinstall NXLOG using this site;
https://www.allcloud.io/how-to/configure-nxlog-send-logs-to-graylog2/
Before I could not get nxlog to work correctly. I just found out that if “rsyslog.service” is not started “NXLOG” will not work. Ummmm
So… NXLOG uses rsyslog.service to get messages then sending them by GELF module. I tried this by using UDP Input. All good for UDP Input test.
Then I created GELF=TCP Input

Log/s output of the remote Linux server using NXLOG, No problems

Log/s From Graylog Cluster. Error occur

How messages are received from input

So Graylog is complaining about something.

Here is my nxlog.conf

Any Advice would be appreciated
Thank in advance

gsmith · December 8, 2017, 3:22am

Problem Solved:
Make sure rsyslog service started/enabled on client (remote Linux server).
Installed NXLOG on client

https://www.allcloud.io/how-to/configure-nxlog-send-logs-to-graylog2/

Configured Graylog Input as follow;

001-Linux-input-tcp_tls

Configure NXLog File as follow;

Test Input messages by restarting nxlog service
Log file from NXLog.conf as follow;

Log file from Graylog server as follow;

Messages from Input as follow;

I’m happy now

By using NXLog on CentOS 7 made life easier to configure 100+ client machine.

jochen · December 8, 2017, 8:20am

I’m just wondering why you would use syslog when NXLOG perfectly supports GELF.

gsmith · December 8, 2017, 10:40pm

@jochen
My first task was getting a ssl connection between Client and Server. Yes, I do agree using GELF, but unfortunately I ran into a error with it. Here is the log error from Graylog Server;

My configurations to GELF are stated in the previous reply’s.
I’m just glad I was able to get something to work, also I learned a lot more about how Graylog functions. :)

This week I’ll be working on GELF Input.

jtkarvo · December 10, 2017, 5:47pm

Difficult to see, what the problem is, without the conf files.

Did you remember to use output type GELF_TCP instead of GELF with SSL output?

gsmith · December 11, 2017, 9:55pm

@jtkarvo
Ah, No I did not.
nxlog-gelf-configuration

Topic		Replies	Views
Join Graylog and Rsyslog Graylog Central (peer support)	6	714	October 24, 2018
Rsyslog not pushing the logs Graylog Central (peer support)	7	1255	October 12, 2022
Configuration Rsyslog on Server and Client Graylog Central (peer support)	8	3745	September 24, 2020
Input TLS configuration Graylog Central (peer support)	2	3911	April 21, 2017
Rsyslog to graylog logs sending Graylog Central (peer support)	2	788	May 10, 2018

Configuring Rsyslog

Related topics