My problem is that I cannot get logs from my remote app to log to a Graylog2 input using TLS. The remote app was logging correctly to the configure input before I attempted to enable TLS with self-signed certs.
Here is some background. First, I created some self-signed certs following these steps:
$ openssl genrsa -out CA.key 2048
$ openssl req -x509 -new -nodes -key CA.key -sha256 -days 3650 -out rootCA.crt
$ openssl genrsa -out graylog-server.key 2048
$ openssl req -new -key graylog-server.key -out graylog-server.csr
$ openssl x509 -req -in graylog-server.csr -CA rootCA.crt -CAkey CA.key -CAcreateserial -out graylog-server.crt -days 3640 -sha256
source: https://gist.github.com/fntlnz/cf14feb5a46b2eda428e000157447309
Where I filled in the DNS name for the graylog server in the graylog-server.crt
I then imported the rootCA.crt to the keystore on the graylog server:
$ keytool -importcert -keystore ./graylog-cacerts -alias graylogRootCaCert -file rootCa.crt
Then, I log into the Graylog web interface and I reconfigure the input fields to this:
port: 51xx
TLS Cert File: /path/to/certs/graylog-server.csr
TLS Private Key File: /path/to/certs/graylog-server.key
Enable TLS: True
TLS Client authentication: Optional
TLS Client Auth Trusted Certs: /path/to/certs*
all the created certs are in the /path/to/certs/ directory which is owned by the graylog. All keys are read only.
The remote app running on Ubuntu 16.04 has an rsyslog config that looks like this:
$ cat 60-remote-rsyslog.conf
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/ssl/graylog/rootCA.pem
$ActionSendStreamDriverAuthMode anon
$ActionSendStreamDriverMode 1 # run driver in TLS-only mode
#$ActionSendStreamDriverPermittedPeer graylog.name.com
. @@graylog.name.com:51XX;RSYSLOG_SyslogProtocol23Format
As for the rsyslog.conf on the server, it is using all the defaults. I have seen a solution where the TLS is set up through rsyslog directly and then Graylog picks up the logs from there, but this is not ideal for my application. We need to use the inputs. That solution is here:
https://serverfault.com/questions/807651/forwarding-logs-from-rsyslog-to-graylog-over-tls
I have tried following the steps in these links, but it is still not working:
https://community.graylog.org/t/rsyslog-over-tls/263/4
https://community.graylog.org/t/encrypt-traffic-from-rsyslog/5649
I also reviewed these links to enabled TLS on rsyslog for the client. I did not apply the server config to the rsyslog config file because I did not expect it was needed (Please correct if I am wrong):
https://www.rsyslog.com/doc/v8-stable/tutorials/tls_cert_summary.html
https://www.rsyslog.com/doc/v8-stable/tutorials/tls.html
I expect that the logs will come from the client TLS secured into the configured input. What happens is the logs do not arrive in the input, and the graylog-server logs do not have any errors to speak of.
I was able to test that Graylog can see receive logs over TLS by running this command on the graylog server:
openssl s_client -connect localhost:51XX -tls1_2 -tlsextdebug
Any help or suggestions are appreciated. Thanks.