Graylog5 Input 'SYSLOG-2222/TCP' could not be started Request to start input 'SYSLOG-2222/TCP' failed

Graylog Central (peer support)
1

1. Describe your incident:
I migrated my Graylog4 data and Elasticsearch logs over to a new graylog. I can see my old data but all my inputs will not start.

Here is an example. I create a TCP input for port 2222 and that looks good in logs:

2023-05-05T16:45:56.050-07:00 INFO  [InputStateListener] Input [Syslog TCP/6455953456ec7834db63184d] is now STARTING
2023-05-05T16:45:56.053-07:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=SYSLOG-2222/TCP, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=bcd3b26f-bffb-4a51-be16-4d927637cf6d} (channel [id: 0x86b8399d, L:/10.1.1.14:2514]) should be >= 1048576 but is 425984.
2023-05-05T16:45:56.055-07:00 INFO  [InputStateListener] Input [Syslog TCP/6455953456ec7834db63184d] is now RUNNING

But when I try and start the input I see a red banner at the bottom of the screen that says:

Input 'SYSLOG-2222/TCP' could not be started
Request to start input 'SYSLOG-2222/TCP' failed. Check your Graylog logs for more information.

Im usually pretty good about checking log files but this failed condition does not appear on my server logs.

2. Describe your environment:

  • OS Information: RHEL9.1

  • Package Version:
    Graylog 5.0.7+7758557 on localhost (Eclipse Adoptium 17.0.6 on Linux 5.14.0-162.23.1.el9_1.x86_64)
    OpenSearch 2.5.0
    mongodb v6.0.5
    nginx/1.20.1

  • Service logs, configurations, and environment variables:
    GRAYLOG_SERVER_JAVA_OPTS=“$GRAYLOG_SERVER_JAVA_OPTS -Djavax.net.ssl.trustStore=/etc/graylog/server/cacerts.jks”

3. What steps have you already taken to try and solve the problem?
temporarily turned off selinux and fapolicyd while I troubleshoot (no change)
opened port 2222/tcp up with firewalld (no change)
tail -f /var/log/graylog-server/server.log (no logs other than the input creation!)
Restarted and rebooted the server a couple times.

4. How can the community help?
Any input on debugging this further.

2

hey @SoMoney

What is the status of Opensearch,Graylog and Mongo? Have you check those other logs too? Are you using HTTPS by chance?

Here some add info on that WARN. This working for my CentOS/Red Hat node.

1 Like
3

Okay figured it out. It was TLS certificate related. Looking for any error in /var/log/graylog-server/server.log I saw the following

2023-05-06T12:27:44.636-07:00 WARN  [ProxiedResource] Unable to call https://10.1.1.14:9000/api/system/metrics/multiple on node <bcd3b26f-bffb-4a51-be16-4d927637cf6d>: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

I was adding my certificates to /etc/graylog/server/cacerts.jks with keytool and then told /etc/sysconfig/graylog-server:

GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Djavax.net.ssl.trustStore=/etc/graylog/server/cacerts.jks -Djavax.net.ssl.trustStoreType=jks"

HOWEVER, THIS OMITS GRAYLOGS v5.0 cacerts file /usr/share/graylog-server/jvm/lib/security/cacerts.
So My issue:

  1. I cant use an IP_ADDRESS http_publish_uri it has to be a name (because my TLS cert used a name).
http_bind_address = 0.0.0.0:9000
http_publish_uri = https://FULLYQUALIFIED_DOMAIN.com/
  1. -Djavax.net.ssl.trustStore option bypassed graylog v5 built in /usr/share/graylog-server/jvm/lib/security/cacerts file.

My FIX:

  1. comment out my GRAYLOG_SERVER_JAVA_OPTS change and just use the cacerts file graylog comes with (/usr/share/graylog-server/jvm/lib/security/cacerts).
keytool -importcert -keystore /usr/share/graylog-server/jvm/lib/security/cacerts -storepass changeit -alias REDACTED -file ./REDATCED.crt
keytool -importcert -keystore /usr/share/graylog-server/jvm/lib/security/cacerts -storepass changeit REDACTED -file ./REDATCED.crt
keytool -importcert -keystore /usr/share/graylog-server/jvm/lib/security/cacerts -storepass changeit -alias REDACTED -file ./REDATCED.bundle.crt

Now my inputs start up…

1 Like
4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.