RSyslog over TLS

Graylog Central (peer support)
1

Hello,

I recently installed Graylog on a server inside our enterprise network and everything works like a charm with UDP syslog.

Now we would like to secure the log transfers with TLS. I generated certificates with our CA, configured syslog on monitored servers to use TCP over TLS instead of UDP, and added a new input in the Graylog configuration.
The input started successfully but when a message arrives, I get the following error (I’ll post the full stacktrace if anyone needs it) : java.security.cert.CertificateException: No X509TrustManager implementation available

Input configuration :

    allow_override_date: true
    bind_address: 0.0.0.0
    expand_structured_data: false
    force_rdns: false
    max_message_size: 2097152
    override_source: <empty>
    port: 10514
    recv_buffer_size: 1048576
    store_full_message: false
    tcp_keepalive: false
    tls_cert_file: /var/lib/graylog-server/ssl/machine.cert.pem
    tls_client_auth: required
    tls_client_auth_cert_file: <empty>
    tls_enable: true
    tls_key_file: /var/lib/graylog-server/ssl/machine.key.der
    tls_key_password: ********
    use_null_delimiter: false

I already tried to add the CA cert in the java keystore. I also tried to add BouncyCastle to Java. But nothing seems to change anything…

Graylog version : 2.2.1+4829190
Java : openjdk version “1.8.0_121”

2

You need to install and define tls_client_auth_cert_file, if you require authentication.

I suggest the following:

  1. try without SSL. If it works, then
  2. try with SSL but with “optional” authentication
  3. try with required authentication

Making this work in steps helps you check where the problem is.

BTW: when switching from UDP to TCP with rsyslog, you need to know what you are doing. Rsyslog is able to throttle the server; read about that before doing that in a production server.

3

Not sure if your Graylog can access your keystore. When I added my certs to the keystore I also had to manually add the keystore as a start-up parameter in my init script so Graylog knew where the keystore was located.

-Djavax.net.ssl.trustStore=/path/to/cacerts.jks
1 Like
4

Hi,

I have been trying to figure out how to contribute to the documentation, but this is how we do it on CentOS6.

On the Graylog Server, configure an input:

allow_override_date: true
bind_address: 0.0.0.0
expand_structured_data: false
force_rdns: false
max_message_size: 2097152
override_source:
port: 12450
recv_buffer_size: 1048576
store_full_message: false
tcp_keepalive: false

We bundled the root and intermediate, order in the file is important.

tls_cert_file: /etc/ssl/your_CA.pem
tls_client_auth: disabled
tls_client_auth_cert_file:
tls_enable: true
tls_key_file: /etc/ssl/private/your_private.key
tls_key_password: ********
use_null_delimiter: false

On the Graylog client:

Install packages:

yum install rsyslog-gnutls
yum install libsemanage-python
yum install policycoreutils-python

Create Spool Directory

mkdir /var/spool/rsyslog
chown root /var/spool/rsyslog

Add rsyslogd Configuration File

Create graylog.conf file in /etc/rsyslog.d directory

Add contents to graylog.conf

Setup disk assisted queues

$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down

#RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/ssl/your_CA.pem
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.your.company.lcl

. @@111.55.200.24:12450;RSYSLOG_SyslogProtocol23Format

Set permissions so root can read

chown root /var/rsyslog.d/10-graylogp.conf

Add your root CA to /etc/ssl

cp your_CA.pem /etc/ssl/your_CA.pem

Make sure root owns file

chown root /etc/ssl/your_CA.pem

If you have SELinux

View SELinux status

getenforce
Enforcing

View SELinux ports allowed for rsyslog

semanage port -l | grep syslog
syslogd_port_t tcp 6514, 601
syslogd_port_t udp 514, 6514, 601

Add SELinux port
semanage port -a -t syslogd_port_t -p tcp 12450

View SELinux ports allowed for rsyslog and verify your port was added.

semanage port -l | grep syslog
syslogd_port_t tcp 12450, 6514, 601
syslogd_port_t udp 514, 6514, 601

Authorize directories

semanage fcontext -a -t syslog_conf_t "/var/spool/rsyslog/"
semanage fcontext -a -t syslog_conf_t "/etc/rsyslog.d/"
semanage fcontext -a -t syslog_conf_t "/etc/ssl/"
restorecon -R -v /var/spool/rsyslog
restorecon -R -v /etc/rsyslog.d
restorecon -R -v /etc/ssl

Restart rsyslog
service rsyslog restart

Test:

logger TestWithSELinux

Warning!!!

We had a nightmare with our Microsoft certificate authority. You have to make sure that the root CA is added to the Java keystore on the Graylog server, and I also added it to the server’s root CA’s.

To import the root certificate, we decided to “pollute” the default Java keystore for Graylog.

  1. Make sure the public certificate is in pem format and Linux file (no Windows characters)
  2. The default trust store of an installed Java runtime environment can be found at $JAVA_HOME/jre/lib/security/cacerts or /etc/ssl/certs/java/cacerts (due to all the symbolic links).
  3. Import:
  4. keytool -importcert -keystore /etc/ssl/certs/cacerts -storepass changeit -alias YourCompany -file your_CA.pem
  5. View the YourCompany root certificate:
    keytool -list -v -keystore /etc/ssl/certs/java/cacerts | grep your.company.com
4 Likes
5

You can create a pull request for the Graylog documentation at GitHub - Graylog2/documentation: Archived Graylog documentation. See https://docs.graylog.org/ for the new documentation. or specifically to the syslog guide at GitHub - Graylog2/graylog-guide-syslog-linux: How to send syslog from Linux systems into Graylog.

6

Hello,

tls_client_auth_cert_file was effectively the missing part in my configuration.
I created a folder and put the certificate of every server that needs to send logs to Graylog.

Everything works fine now :slight_smile:

I’m very used to Rsyslog, throttling should not be a problem here as we only have a dozen of servers sending logs and not everything is sent through Rsyslog.

7

For everyone using Let’s Encrypt Certificates on the Graylog server, user this line in your Clients:
$DefaultNetstreamDriverCAFile /etc/ssl/certs/DST_Root_CA_X3.pem

Thank me later…

1 Like