Configuring Rsyslog


(Greg Smith) #1

Hello All,
Having troubles with Rsyslog TLS/SSL Configuration on Linux clients send messages to graylog server with certificates.

Environment;
Total of 6 CentOS 7.3 Servers minimal install.
3 Servers with Graylog version 2.3 and Mongo version 3.4 ‘Clustered’
3 Servers with Elasticsearch 5.6.4 ‘Clustered’

Client Rsyslog Version;
rsyslog-8.24.0-12.el7.x86_64

Graylog Input

Rsyslog.conf file on Linux Client

#### MODULES #### The imjournal module bellow is now used as a message source instead of imuxsock.

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
Provides UDP syslog reception
#$ModLoad imudp
#$ModLoad imudp
#$UDPServerRun 514

Provides TCP syslog reception
#$ModLoad imtcp
#InputTCPServerRun 514

GLOBAL DIRECTIVES
Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

File syncing capability is disabled by default. This feature is usually not required,
not useful and an extreme performance hit
#$ActionFileEnableSync on

Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
Turn off message reception via local log socket;
local messages are retrieved through imjournal now.
$OmitLocalLogging on

File to store the position in the journal
$IMJournalStateFile imjournal.state

RULES
Log all kernel messages to the console.
Logging much else clutters up the screen
#kern.* /dev/console

Log anything (except mail) of level info or higher.
Don’t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

The authpriv file has restricted access.
authpriv.* /var/log/secure

Log all the mail messages in one place.
mail.* -/var/log/maillog

Log cron stuff
cron.* /var/log/cron

Everybody gets emergency messages
.emerg :omusrmsg:

Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

Save boot messages also to boot.log
local7.* /var/log/boot.log

Begin forwarding rule
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down

Set gtls driver
$DefaultNetstreamDriver gtls

certificate files
$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/cert.pem
$DefaultNetstreamDriverCertFile /etc/pki/tls/certs/graylog-certificate.pem
$DefaultNetstreamDriverKeyFile /etc/pki/tls/certs/graylog-key.pem
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *..net
$ActionSendStreamDriverMode 1 # run driver in TLS-only mode

Remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#. @@remote-host:514
#. @@remote-host:514
. @@lab-graylog-001..net:51423
end of the forwarding rule

Rsyslog.log from Linux Client;

Rsyslog Status running

Graylog Server Logs

I tried to follow these links, but was unsuccessful;

http://www.rsyslog.com/doc/v8-stable/tutorials/tls_cert_scenario.html

https://help.papertrailapp.com/kb/configuration/encrypting-remote-syslog-with-tls-ssl/#download-root-certificates

http://kb.kristianreese.com/index.php?View=entry&EntryID=148

Wondering if some one could enlighten me where I’m going wrong. Perhaps I over looked something?
Thanks in advance. I tried to follow these links, but was unsuccessful;


#2

The last log that points to rsyslog.com/e/2078 indicates a permission issue on the cert Graylog created for the input. Have you checked to see whatever user Graylog is running under has permissions to the cert? Since Graylog created the cert it should, but something might have changed if it was touched with a file editor or something else.

Also, is it possible to use a valid signed cert and test? Some vendors such as digicert allow for unlimited duplicate certs. I’ve run into issues with self signed certs in the past.


#3

Just realized it was rsyslog throwing the error I mentioned, but I noticed Graylog complaining it couldn’t find your TLS key file. I would check the permissions on that.

As of that log entry it isn’t using your key for that input.


(Greg Smith) #4

@rfinney
Thank you for your quick response.

I fixed my input on Graylog Server. I’m using one that works previously for my Windows machines, as shown below;
NOTE: What’s weird is that I just copied this input to make one for my Linux machines and all I did was change the Port Number and Name.

I used SCP command for certificate file/s from Graylog Server to Remote Linux client.
I did change Certificate location as show below;
NOTE:They used to be located in DIR /etc/pki/tls/certs/

On the Remote Linux Machine/s I have tried to set the permission on certificates and Directories to 777 (I know, not a good Idea) , but for testing purposes, this still did not work.

Just a side note I dropped my firewall and disabled Selinux.

The error on Rsyslog Status stays the same and no data In Graylog Input (test-tcp) from remote Linux clients. I know the TCP/TLS Input works because I have machine’s sending data to that Input as we speak.
Following error on Rsyslog Status;


I believe there is a permission/authentication issue as stated from above, but where? I was looking into configurations in NXLOG on Windows machines, and In the Configuration file I set the Pass Key for the certs. During my research I did not find how to config rsyslog.conf for a pass key.

Thank you for you help in advance.


(Greg Smith) #5

To Summarize;
Created an Input Called Linux-TCP, using TLS cert, key, and password configuration.
Transferred Certificates using SCP from Graylog Server to a remote Linux server.
Certificate are placed in /etc/pki/rsyslog/ directory on remote Linux server.

Configure Remote Linux Server rsyslog.conf as;
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down

#driver
$DefaultNetstreamDriver gtls

#certificate files
$DefaultNetstreamDriverCAFile /etc/pki/rsyslog/cert.pem
$DefaultNetstreamDriverCertFile /etc/pki/rsyslog/graylog-cert.pem
$DefaultNetstreamDriverKeyFile /etc/pki/rsyslog/graylog-key.pem

#actions
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer lab-graylog-001.nseva-labs.net
$ActionSendStreamDriverMode 1 # run driver in TLS-only mode

#remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*. * @@lab-graylog-001.enseva-labs.net:6514

Restarted Rsyslog Service
No Messages came through Linux-TCP Input (fixed Input error “TLS key file or certificate file does not exist, creating a self-signed certificate for input”) There was a space in full path.

Rsyslog Status error “unexpected GnuTLS error -24 in nsd_gtls.c:205: Decryption has failed.”.
I check permissions on Certificates, and directory’s.
I went as far as giving everyone excess to certs and directory’s, No Joy.
Do I need to make Cert’s on remote Linux server, then transfer them to Graylog server?
Looking for a way on the remote Linux server to read Graylog Certs’, I think that is where the problem located, but I’m unsure. If so how do I get rsyslog to read these cert’s?

NOTE: I tired just using TCP Connection without Certs, no problems occurred, messages came through.


(Jochen) #6

What’s the output of the following command?

namei -l /etc/pki/rsyslog/graylog-key.pem

What’s the system user running Graylog?


(Greg Smith) #7

This is from the Remote CentOS 7 FTP Server

System user running Graylog


(Greg Smith) #8

I’m unsure if this information would help Identify my problem, but I execute the following command on Graylog Certificate from the Remote CentOS 7 (ftp-server) ;
root# curl -v --cacert graylog-certificate.pem https://lab-graylog-001.enseva-labs.net:9000


(Jochen) #9

There’s your problem. The directory /etc/pki/rsyslog/ is lacking the executable bit (see https://unix.stackexchange.com/a/21252) and the file /etc/pki/rsyslog/graylog-key.pem is only readable for the user “root”.


(Greg Smith) #10

@jochen

I execute the new configuration as suggested

Graylog-Server Input for linux-tcp/tls looks like it started up fine as shown below;

Input Config

Remote Linux Server Rsyslog.service status does show same error;
“unexpected GnuTLS error -24 in nsd_gtls.c:205: Decryption has failed.”

I’m using the same certificates made on the Graylog server that Windows OS is using in NXLOG. No problems are occurring for the Syslog-TCP input, with TLS enabled.

So i research the error URL provided it states the following;

http://kb.monitorware.com/kbeventdb-detail-id-6789.html
"This message occurs with TLS netstream driver. For TLS, certificates (.pem files) are needed to provide security credentials.
This error is issued if there is a problem with these files. The message contains a more precise error description. That error text is taken directly from the underlying TLS library.
A common cause is that the file can not be found or accessed (permissions!). In that case, a rsyslogd-2040 error will follow.

This is my rsyslog.conf file, not sure what drivers i would need.

Is rsyslog service unable to use graylog Certificates?
Or did I not give the correct ownership to cert files?
NOTE:I did try to make graylog users and chown cert files, but that did not seem to work, same errors occurred


(Greg Smith) #11

@jochen
I’ve been trying to every permission for the remote Linux server Cert’s and nothing seems to work. I do realize its on the Remote Linux server, rsyslog service. When I disable TLS in the input and comment out
"# $ActionSendStreamDriverMode 1 # run driver in TLS-only mode"
Graylog starts receiving messages, obviously messages are only being sent through TCP,
Any other Ideas would be appreciated.


(Greg Smith) #12

@jochen
I might have found my solution about the error from rsyslog,
Creating a different Certificate/s for my Linux servers.


(Greg Smith) #13

I created new Certificates for the remote Linux Servers, No joy. Same error occurred;

“unexpected GnuTLS error -24 in nsd_gtls.c:205: Decryption has failed.”

So, I have Isolated the problem to the Remote Linux Server using Rsyslog. Unable to use Working Certificates I made for Graylog Cluster. Any help would be appreciated
Thank in advance.


(Greg Smith) #14

After researching ,think I found the problem why Graylog Certs are not working for Linux Servers.
http://www.rsyslog.com/doc/v8-stable/tutorials/tls.html#certificates

To sum it up,
I was using “Openssl” and “keytool” commands not “certtools”. This might be culprit, but still unsure.

Tried making certificates from the following
http://www.rsyslog.com/doc/v8-stable/tutorials/tls.html#ca-certificate

This corrected the error from Rsyslog on the remote Linux Server., But graylog was unable to use these certs in the Input.
Think I’m getting closer to a solution again.
Trying to find another solution/s on how to use Certificates that Linux Servers and Graylog Servers are able to use together. perhaps an example what other peoples use for TCP/TLS


(Greg Smith) #15

Well… I gave up on using Rsyslog. I’ve tried to make certs that work for Rsyslog, but that did not work for Graylog. Tried converting those certs so graylog would be able to use them, No joy
I’ve joined https://kb.monitorware.com (i.e. was a bad experience to even register), Still no Joy
I reinstall NXLOG using this site;
https://www.allcloud.io/how-to/configure-nxlog-send-logs-to-graylog2/
Before I could not get nxlog to work correctly. I just found out that if “rsyslog.service” is not started “NXLOG” will not work. Ummmm
So… NXLOG uses rsyslog.service to get messages then sending them by GELF module. I tried this by using UDP Input. All good for UDP Input test.
Then I created GELF=TCP Input


Log/s output of the remote Linux server using NXLOG, No problems

Log/s From Graylog Cluster. Error occur

How messages are received from input

So Graylog is complaining about something.

Here is my nxlog.conf

Any Advice would be appreciated
Thank in advance


(Greg Smith) #16

Problem Solved:
Make sure rsyslog service started/enabled on client (remote Linux server).
Installed NXLOG on client

Configured Graylog Input as follow;

001-Linux-input-tcp_tls

Configure NXLog File as follow;

Test Input messages by restarting nxlog service
Log file from NXLog.conf as follow;

Log file from Graylog server as follow;

Messages from Input as follow;

I’m happy now :slight_smile:

By using NXLog on CentOS 7 made life easier to configure 100+ client machine.


(Jochen) #17

I’m just wondering why you would use syslog when NXLOG perfectly supports GELF. :thinking:


(Greg Smith) #18

@jochen
My first task was getting a ssl connection between Client and Server. Yes, I do agree using GELF, but unfortunately I ran into a error with it. Here is the log error from Graylog Server;

My configurations to GELF are stated in the previous reply’s.
I’m just glad I was able to get something to work, also I learned a lot more about how Graylog functions. :):grin:

This week I’ll be working on GELF Input.


#19

Difficult to see, what the problem is, without the conf files.

Did you remember to use output type GELF_TCP instead of GELF with SSL output?


(Greg Smith) #20

@jtkarvo
Ah, No I did not.