Beats not running

[arafat16]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

I’ve enabled HTTPS on Graylog Open 6.1 by creating a self-signed Certificate. I’ve followed the guidelines from the following :

HTTPS
How-To Guide: Securing Graylog with TLS

For clarity I’m providing the following steps that I took for enabling HTTPS with a self-signed certificate.
Step 1 :
echo "[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
req_extensions = v3_req

[ dn ]
CN = Graylog Server

[ v3_req ]
subjectAltName = IP:10.35.4.20" > openssl-graylog.cnf

Step 2: openssl req -x509 -days 365 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem

Step 3: openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem

Step 4: sudo mv cert.pem /etc/graylog/server/graylog.crt
sudo mv pkcs8-plain.pem /etc/graylog/server/graylog.key

Step 5: sudo chown graylog:graylog /etc/graylog/server/graylog.crt /etc/graylog/server/graylog.key
sudo chmod 644 /etc/graylog/server/graylog.crt
sudo chmod 600 /etc/graylog/server/graylog.key

Step 6: In the server.conf file the following changes have been made -
http_enable_tls = true
http_tls_cert_file = /etc/graylog/server/graylog.crt
http_tls_key_file = /etc/graylog/server/graylog.key

Step 7: systemctl restart graylog-server

Then I’ve installed Graylog Sidecar on Windows following the instructions from here
[Install Sidecar on Windows](Install Sidecar on Windows?

In the sidecar.yml I’ve also edited the following :
tls_skip_verify: true

After that when I tried to add an input for Beats with port 5044 selecting Global, it shows “0 Running”

2. Describe your environment:

• OS Information: Red Hat Enterprise Linux release 9.5 (Plow)

• Package Version:
  MongoDB-org 7.0
  Graylog-6.1

• Service logs, configurations, and environment variables:

After adding a Beats I did a tail -f /var/log/graylog-server/server.log, this is the output :

This is the image of my inputs for Beats showing “0 running”

I also added another Beats inputs to my only node and the following screenshot of the web interface and the logs are below:

Log: As you can see it’s showing it’s running on the logs but the web interface is showing not running on the node.

3. What steps have you already taken to try and solve the problem?

I’m trying to figure out even what causing my inputs not to run. Previously I did not have OpenJDK installed. I have installed OpenJDK 17 now and I’m trying to see if I have to add my self signed certificate to JVM Trust Store or not? Is this problem related to certificate trust issue that is preventing the web interface from showing inputs as running?

4. How can the community help?
If the community can point out how to run Beats and troubleshoot why my inputs for Beats aren’t running on the web interface, I would really appreciate it. I’m trying to get windows log files on the Graylog server. I am also thinking that it could be a simple GUI issue. My inputs beats could be running but it’s not showing in the web interface.

Thank you in advance!

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[Joel_Duffield]

After configuring tls did you change publish uri in server.conf to https from http

[arafat16]

Hi @Joel_Duffield

Thank you for your feedback. I’ve checked the server.conf to make sure that I’ve set the publish uri to https .
This is what it looks like http_publish_uri = https://10.35.4.20:9000/
Any other things that I should look into?

[Joel_Duffield]

I dont think your graylog server trusts its own certificates, i know that sounds weird but it talks to itself. You need to make sure the certs you made anre in the java keystore and are trusted.

[arafat16]

@Joel_Duffield Thank you for your feedback and I know it sounds weird that graylog server don’t trust it’s own certificate.

So, I did little a research and I found that I did a typo error when adding the cacerts into the JVM trust store. The updated JVM options in /etc/sysconfig/graylog-server looks like this now.

Please note that this is where I added the cacerts to JVM options setting that is JAVA Trust Store

The bold part is what I did added the JAVA Trust Store
GRAYLOG_SERVER_JAVA_OPTS="-Xms16g -Xmx16g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks"

Here’s a screenshot of the jvm options (/etc/sysconfig/graylog-server) below:

Previously for a typo error when I added the cacerts to jvm options /etc/sysconfig/graylog-server this resulted with a broken WEB UI.

Just to give you a rundown of the steps I took to add the certs in Java Trust Store:

1. Located the Java Trust Store: There were several options to choose between them but I chose java-17-openjdk-17.0.15.0.6-2.el9.x86_64

java-17-openjdk-17.0.15.0.6-2.el9.x86_64 jre jre-17 jre-17-openjdk jre-17-openjdk-17.0.15.0.6-2.el9.x86_64 jre-openjdk

Screenshot below:

cd /usr/lib/jvm/java-17-openjdk-17.0.15.0.6-2.el9.x86_64/lib/security/cacerts/etc/pki/java/etc/pki/ca-trust/extracted/java/cacerts

2. I’ve copied the default cacerts file:
   cp cacerts /etc/graylog/graylog.jks

3. Imported the certificate:
   keytool -importcert -keystore /etc/graylog/graylog.jks -storepass changeit -alias graylog-self-signed -file /etc/graylog/server/graylog.crt

Pressed y when prompted to trust the certificate and the default password for cacerts was changeit.

4. I went to the JVM options to append GRAYLOG_SERVER_JAVA_OPTS
   vi /etc/sysconfig/graylog-server and added -Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks

Here’s a screenshot :
/etc/sysconfig/graylog-server

5. Then I’ve set the ownership : chown graylog:graylog /etc/graylog/graylog.jks

6. Restarted the graylog server : systemctl restart graylog-server

7. So, I ran a tail -f /var/log/graylog-server/server.log and here is the log:
   

   

8. I added two inputs to test. As you can see from the screenshots the WEB UI is showing as “0 Running” for both of them:
   Beats (Port 5044)
   

   

You can see from the server.conf logs it started the Input Beats and is running

Same for Syslog UDP (Port 1514)

The server.conf logs it started the Input Syslog UDP and is running

I’ve also checked the sidecar.log in my windows machine and I believe it’s running properly. Thought of sharing this just incase -

For test purpose to see if my inputs will be running without https , so in the server.conf I’ve also set the - http_enable_tls = false

Still no luck

Question I have is now : Did I miss any steps the way I added the certificate to JVM trust store?
Why is my WEB UI is still showing as “0 Running” but from the server.conf log I can see both the inputs are running. Is it a WEB UI issue I’m experiencing and I’m guessing it’s a simple fix if that’s the case.

I would really appreciate any pointer/suggestions to trouble shoot this issue. Thanks in advance

[arafat16]

@Joel_Duffield
In order for the inputs to work as “Running”, I had to disable https and revert back to all the changes I made for https. For some reason the JVM trust store is not liking the self-signed certificate which I’ve created. Currently, it’s running on http and the beats showing as running. Thank you again for all the help!

Now I have to figure out to secure the Graylog Web UI with TLS so that the input beats can work.

[elster]

Hi arafat,

I can’t really help you with enabling HTTPS within Graylog itself but if all you want is to enable HTTPS to access the web UI and the API, maybe take a look at using a reverse proxy:

The Web Interface

It’s also rather easy to use Let’s Encrypt/ZeroSSL/… certificates in your reverse proxy of choice and it can be easily fully automated but it depends on your environment of course.

[arafat16]

Thank you for your feedback; I will take it into consideration.