TLS Encrypted Communication between Graylog and Graylog Forwarder failing (GRPC Error)

Graylog Central (peer support)
1

Hey there,

I am trying to encrypt the traffic between a Graylog Forwarder and another Graylog Node.

On the Graylog Node i have setup a ca.pem, a key.pem, and a certificate.pem.
I have also created the Forwarder Input which is using the certificate.pem and the key.pem.

(I would add a picture of the Input I have setup but as a new user in this Forum I can only post a single Image)

On the Forwarder i have setup a stream and the corresponding Output which points at the input of the other Graylog Node.
I then transferred the ca.pem to the Forwarder and setup the Output to use the ca.pem file so that it can verify the Server certificate.

gr2

  • When I am trying to connect to the Graylog Node via the openssl s_client program everything works fine. (Here I am also specifying the ca.pem)
  • When I look at the Stream Output, the messages are shown as being forwarded

BUT

  • When Graylog is trying to Forward the Log messages to the Server I am getting the following errors:

    2020-07-23T09:32:20.299+02:00 INFO [ForwarderOutput] Launched Forwarder output [Output/5f193ca70cb76c4e060291d8] successfully.
    2020-07-23T09:32:30.203+02:00 ERROR [GRPCStreamingSenderClient] A gRPC client error occurred: [UNAVAILABLE: io exception
    Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]]
    2020-07-23T09:32:30.211+02:00 INFO [GRPCStreamingSenderClient] Restarting Forwarder network sender in [10000 MILLISECONDS]…
    2020-07-23T09:32:30.708+02:00 ERROR [GRPCStreamingSenderClient] An error occurred while sending message [call was cancelled]

And on the Graylog Node itself I am not recieving anything.

Any Ideas regarding the GRPC Error Messages would be helpful. :slight_smile:

2

he, maybe a stupid question:

  • is the time on both devices synced with ntp?
3

Hey,

no the time is not synced with ntp but it is the exactly same on both servers.

Should i sync it with ntp?

4

most important is that the time is in sync.

Possible reason that it does not work might be that the certificate can’t be verified but that should be given as something in the log.

Did you checked if the firewall allows the connection between those servers?

5

Yes the time on both servers is in sync and firewall is disabled on both servers.
When i disable TLS between those two, then the messages are being forwarded without any problem.

I also thought it could be a problem with the certificates but then i would expect that there are messages in the logs about it.

Is there any way to verbose the logging of Graylog?

6

Another useful Information:

When using GELF TCP with TLS enabled as Input on the Server instead of Forwarder Input everything works as expected. Even while using the same Certificate, CA and Key.

7

@bjarneeins how was the certificate issued? You may need to add the “Graylog Node”'s certificate to the forwarder’s jvm trust store so that the forwarder trusts it.

8

I have used the following commands to issue the certificates:

openssl genrsa -aes256 -out ca-key.pem 2048

openssl req -x509 -new -nodes -extensions v3_ca -key ca-key.pem -days 1024 -out ca-cert.pem -sha512

openssl genrsa -out key.pem 4096

openssl req -new -key key.pem -out request.csr -sha512

openssl x509 -req -in request.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out cert.pem -days 365 -sha512

openssl pkcs8 -in key.pem -topk8 -nocrypt -out key-plain.pem

I then used the ca-cert.pem on the Forwarder and the cert.pem and the key-plain.pem on the Server to which the Logs are being forwarded.

9

I have also tried adding the Graylog Nodes certificate (cert.pem) to the JVM Truststore on the Forwarder but i am still getting the same error.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.