Graylog 3.0 doesn't get syslog messages anymore from newish rsyslog versions

Graylog Central (peer support)
1

Hi,

after managing to update to Graylog 3.0 (because the UI was bugging me to update), I noticed that all our newer servers do not send syslog messages to Graylog anymore. The only servers for which logs still come in are the old ones (we are talking wheezy here). The error thrown by rsyslogd is

3661.491220099:main Q:Reg/w0  : Called LogMsg, msg: unexpected GnuTLS error -28 in nsd_gtls.c:1755: Resource temporarily unavailable, try again.

rsyslogd: unexpected GnuTLS error -28 in nsd_gtls.c:1755: Resource temporarily unavailable, try again.  [v8.24.0 try http://www.rsyslog.com/e/2078 ]

gnutls-cli works fine and connects. Does someone have a clue to why this is? I have been trying for a few hours to figure out what the problem is but I can’t put my finger on it.
GELF still seems to work fine.

Thanks

2

The UI does not nack you for a beta version …

How did you configure what and how could we try to rebuild what you have to check where the error might be?

3

I’ve got a Syslog TCP input with TLS setup. It uses a Let’s Encrypt Certificate. Single node.
On the client I have the same config on all servers (mostly Debian 8 & 9).

$DefaultNetstreamDriverCAFile /etc/ssl/certs/DST_Root_CA_X3.pem

$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon
*.* @@(o)SERVERADDRESS:8514;RSYSLOG_SyslogProtocol23Format

The UI was showing me something that looked like an error message telling me to upgrade following a link. When I clicked on the link it showed me that the latest version was 3.0 and how to upgrade. I did not realise until now that you told me that 3.0 is still in beta. I reckon it wasn’t clear enough and I didn’t see it. But now here I am and I’m not going to downgrade.

Thanks

4

This issue appears to be with rsyslog and not Graylog.

The thread which can be found via the link provided in the error message states:

A common cause is that the file can not be found or accessed (permissions!).

First thing to check is that /etc/ssl/certs/DST_Root_CA_X3.pem exists and is readable.

1 Like
5

Yes, the file does exist and is readable. It did work before I upgraded Graylog. Just to test I even took the intermediate LE X3 certificate and put it somewhere else and tried to use that.
I did read through that link when I saw the error message. It’s not like I came directly here before doing some research. Posting in forums is a last resort for me. Thank you for understanding.

6

just some tips for debuging
use openssl in client mode to verify the graylog’s cert over TLS connection.
check your java settings, as far as i know you need to modify the java certstore, the new GL install use your old certstore?

7

I’ve used both gnutls-cli and the openssl client to test the connection; it works and when I type something in it appears in Graylog.
I have never set a certstore in Graylog or changed the Java default certstore. Certstore wise it’s all default.
I will try to create another input on a different port, just to see if the config of the input got messed up.
Thanks.

8

It works with the second input I created. I’ve set only the necessary and left everything else default. Here is the diff of the config that is shown in the UI:

$ diff --color input1.txt input2.txt
5a6
> number_worker_threads: 2
7c8
< port: 8514
---
> port: 9999
12c13
< tls_client_auth: optional
---
> tls_client_auth: disabled

Something must have gotten messed up in the internal configs. Is there some debugging I can do for you?

9 
tls_client_auth : disabled

the GL won’t auth the client.
If you want you have to add the CA to the java cert store AND/OR (i’m not sure) you can also put the acepted cert in a dire, and config the dir under the input. ( tls_client_auth_cert_file)
Check the official docs:
http://docs.graylog.org/en/2.4/pages/configuration/https.html

So as I see you set client auth without tell GL who are the trusted clients.

10

That’s because I have never had authentication setup and optional vs. disabled should not make a difference. I guess the default for tls_client_auth has changed from optional to disabled so that’s why it’s different. I’ll change it anyways to disabled in the original input to see if a config change triggers something and repairs the input. Thank you anyways.

11

Setting tls_client_auth to disabled fixes the original input. Putting it back to optional breaks it again, so I’m going to leave it to disabled.
So this has to be some sort of bug because the behaviour of ‘optional’ has changed since the last release.
As I said, before the upgrade it worked. Thanks for the help.

1 Like
12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.