Configuration ssl/tls graylog https

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I have probles to configure clients filebeat and NXLOG and winlog to send beats with sidecar in ssl/tls mode.

2. Describe your environment:

  • OS Information: debian 11

  • Package Version: graylog 4.3

  • Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?
I have made the configuration of the server to HTTPS but the problem is configure the clients

4. How can the community help?
I hace a certificate cert.pem and a private key pkcs8-encrypted.pem but the configuration of filebeat is:
output.logstash:
hosts: [“logs.mycompany.com:5044”]
ssl.certificate_authorities: [“/etc/ca.crt”]
ssl.certificate: “/etc/client.crt”
ssl.key: “/etc/client.key”

How can I get the ca.cert, client.cert and client .cert ?

Thanks a lot to help me it meams a lot to me.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

Hey, Jose,

Thanks for asking your question in the community. Here’s some information from the 4.x documentation. I hope it helps. Our community has many excellent users and experts who are willing to help. Look for additional posts to this thread from them. My response is based mostly on the documentation.

To obtain the ca.cert, client.cert, and client.key files, you can use the cert.pem and pkcs8-encrypted.pem files you have. You can use OpenSSL to split the cert.pem file into the ca.cert and client.cert files and to decrypt the pkcs8-encrypted.pem file to obtain the client.key file.

The exact commands you would use would depend on the format and encoding of your original files, but a possible set of commands would be:

Split the cert.pem file into the ca.cert and client.cert files:


openssl pkcs12 -in cert.pem -nokeys -out ca.cert
openssl pkcs12 -in cert.pem -nokeys -out client.cert -nodes
Decrypt the pkcs8-encrypted.pem file to obtain the client.key file:
Copy code
openssl pkcs8 -in pkcs8-encrypted.pem -out client.key -nocrypt

Then you can configure Filebeat with these certificates and keys


output.logstash:
hosts: [“logs.mycompany.com:5044”]
ssl.certificate_authorities: [“/path/to/ca.cert”]
ssl.certificate: “/path/to/client.cert”
ssl.key: “/path/to/client.key”

Remember to restart the filebeat service after configuration changes.

The community can help you by providing more detailed information on how to use OpenSSL to split and decrypt your files, troubleshoot any issues you encounter, and provide guidance on how to configure Filebeat and NXLOG to send beats to your Graylog server in SSL/TLS mode.

For more detail on @dscryber’s reply, look here:

https://go2docs.graylog.org/4-x/setting_up_graylog/secured_graylog_and_beats_input.html?Highlight=ca.cert%2C%20client.cert%2C%20and%20client.key%20

1 Like

Hey @jgutie45

What do you see in the log files (i.e., Sidecar, Graylog,etc…) this may improve troubleshooting the issue.

Hi mister dscryber
First thanks to all people, it means a lot to me.
I used the exact comands that I saw in these videos, these are the liks:
02. Enable https for Graylog 2.4 | Graylog 2.4 - YouTube
Graylog - HTTPS setup v4.0.6 - YouTube
the exact procedure that i made is this:
ssl/tls en graylog

cd /etc/ssl/certs
mkdir graylog
Nano openssl-graylog.cnf

in the file created before
DNS.1 = graylog.localhost
IP.1 = ipgraylogserver

nano /etc/default/elasticsearch

put the java_home in elastic search

make the certificates

openssl req -x509 -days 365 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem

openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:secret

put permisions to system

chmod 444 cert.pem
chmod 444 pkcs5-plain.pem
chmod 444 pkcs8-encrypted.pem

nano /etc/graylog/server.conf

put this in server.conf

in HTTPS SETTINGS

http_enable_tls = true

http_tls_cert_file = /etc/ssl/certs/graylog/cert.pem

http_tls_key_file = /etc/ssl/certs/graylog/pkcs8-encrypted.pem

http_tls_key_password = secret

copy the store of java to /etc/ssl/certs/graylog/cacerts.jks

cp -a “${JAVA_HOME}/jre/lib/security/cacerts” /etc/ssl/certs/graylog/cacerts.jks

import our cert to store copied certs before

keytool –importcert –keystore /etc/ssl/certs/graylog/cacerts.jks –storepass changeit –alias graylog-self-signed –file cert.pem

look all is good like this

keytool -keystore /etc/ssl/certs/graylog/cacerts.jks -storepass changeit -list | grep graylog-self-signed -A1

restart graylog-server and elasticserach and mongod

that´s all that I have made…

thanks to everyone.

1 Like

Hi again I am trying your answer and only the last comand goes wrong,

openssl pkcs8 -in pkcs8-encrypted.pem -out client.key -nocrypt

the out is:
Error decrypting key
140476862068032:error:0909006C:PEM routines:get_name:no start line:…/crypto/pem/pem_lib.c:745:Expecting: PRIVATE KEY
I do not know what to do… Please help.
sorry, the client.key appears… I am going to put it in to the client and wait what pass…
Here is the result of ls -l
ls -l
total 172
-rw------- 1 root root 0 Jan 19 15:50 ca.cert
-rw-r–r-- 1 root root 151636 Jan 19 13:10 cacerts
-rw-r–r-- 1 root root 1366 Jan 19 13:11 cacerts.jks
-r–r–r-- 1 root root 1359 Jan 19 13:00 cert.pem
-rw------- 1 root root 0 Jan 19 15:51 client.cert
-rw------- 1 root root 0 Jan 19 16:00 client.key
-rwxrwxrwx 1 root root 593 Jan 19 13:00 openssl-graylog.cnf
-r–r–r-- 1 root root 1704 Jan 19 13:00 pkcs5-plain.pem
-r–r–r-- 1 root root 1874 Jan 19 13:03 pkcs8-encrypted.pem

Thanks again.

Hi chris

I just do it all it put in the document that you send me so… I am lost, very lost…

Keep going in investigation.

best regards.

Hi again Ms gsmith
this is the error that I can see in the frontend when the sodecar is failed:

Exiting: error initializing publisher: 2 errors: no PEM blocks /certificados/client.cert; file is not a certificate adding /certificados/ca.cert to the list of known CAs

I have this error in the client machine in /var/log/graylog-sidecar/ , I have two files one called filebeat_stderr.log with the nest error:

Exiting: error initializing publisher: 2 errors: no PEM blocks /certificados/client.cert; file is not a certificate adding /certificados/ca.cert to the list of known CAs
In the secod file called sidecar.log put the same error:
time="2023-01-19T18:07:24+01:00" level=info msg="Adding process runner for: filebeat"
time="2023-01-19T18:07:31+01:00" level=info msg="[filebeat] Configuration change detected, rewriting configuration file."
time="2023-01-19T18:07:32+01:00" level=error msg="[filebeat] Collector configuration file is not valid, waiting for the next update."
time="2023-01-19T18:07:32+01:00" level=error msg="[filebeat] Validation command output: Exiting: error initializing publisher: 2 errors: no PEM blocks /certificados/client.cert; file is not a certificate adding /certificados/c>
time="2023-01-19T18:10:35+01:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put \"https://192.168.194.130:9000/api/sidecars/8c5fe107-9fbe-432f-ac64-52f3eabf4b5a\": EOF"
time="2023-01-19T18:10:36+01:00" level=info msg="Stopping signal distributor"
time="2023-01-20T11:47:37+01:00" level=info msg="Starting signal distributor"
time="2023-01-20T11:47:48+01:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put \"https://192.168.194.130:9000/api/sidecars/8c5fe107-9fbe-432f-ac64-52f3eabf4b5a\": dial tcp 192.168.194.1>
time="2023-01-20T11:48:26+01:00" level=info msg="Adding process runner for: filebeat"
time="2023-01-20T11:48:27+01:00" level=info msg="[filebeat] Configuration change detected, rewriting configuration file."
time="2023-01-20T11:48:28+01:00" level=error msg="[filebeat] Collector configuration file is not valid, waiting for the next update."
time="2023-01-20T11:48:28+01:00" level=error msg="[filebeat] Validation command output: Exiting: error initializing publisher: 2 errors: no PEM blocks /certificados/client.cert; file is not a certificate adding /certificados/c>

I hope you can help me… Best regards.

Hey @jgutie45

From what I understand from the Log files is:

error initializing publisher: 2 errors: no PEM blocks /certificados/client.cert; file is not a certificate adding /certificados/ca.cert to the list of known CAs

KEYSTORE
Ensure the certificate in the keystore, If so then ensure Graylog can access the keystore. For starters instead of creating a Keystore use JAVA default keystore called “cacerts” and place the certs needed in there.

CERTS
When creating the certificate file make sure the fully qualified domain name is correct.

For example if I had a Graylog server called graylog.domian.com and it ip address is 192.168.1.100 then my file should look like this:

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
# Details about the issuer of the certificate
[req_distinguished_name]
C = US
ST = iowa
L = cedar rapids
O = enseva
OU = admin
CN = graylog.enseva-labs.net
[v3_req]
keyUsage = keyEncipherment, dataEncipherment,nonRepudiation
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
# IP addresses and DNS names the certificate should include
# Use IP.### for IP addresses and DNS.### for DNS names,
# with "###" being a consecutive number.
[alt_names]
IP.1 = 192.168.1.100
DNS.1 = graylog.domain.com

Next insure the keyusage is configured like so

keyUsage = keyEncipherment, dataEncipherment,nonRepudiation

So your end product should look like this from here and this states the following from the documentation.

The resulting PKCS#8 private key (graylog-key.pem) and the X.509 certificate (graylog-certificate.pem) can now be used to enable encrypted connections with Graylog by enabling TLS for the Graylog REST API and the web interface in the Graylog configuration file:

Conclustion:
It maybe the certificate made/used are incorrect. Perhaps the keystore does not have the correct Certs and/or both.

I did a quick search for ya on the matters of HTTPS in the forum, some work some dont mainly because everyone environement is a little different, you can see here

The easiest way knowning the graylog can access those certificate its by putting it into Graylog OWN directory.

Example

[root@graylog graylog_user]# ls /etc/graylog/
total 60K
drwxrwxrwx.   4 graylog graylog 4.0K Oct 10 19:03 .
drwxr-xr-x. 127 root    root    8.0K Jan  5 16:41 ..
drwxr-xr-x.   2 graylog graylog   62 Sep 20 21:43 forwarder
drwxr-xr-x.   2 graylog graylog 4.0K Jan 19 20:48 server
-rwxr-xr-x.   1 graylog graylog 1.4K Dec 16  2020 cert.pem
-rwxr-xr-x.   1 graylog graylog 1.6K Dec 16  2020 graylog-certificate.pem
-rwxr-xr-x.   1 graylog graylog 1.8K Dec 16  2020 graylog-key.pem
-rw-r--r-x.   1 graylog graylog  520 Aug  1 22:16 lookup_dashboards.csv
-rw-r--r--.   1 graylog graylog 6.1K Oct  4 17:54 lookup_streams.csv
-rw-r--r--.   1 graylog graylog  177 Oct  4 18:15 lookup_traffic.csv
-rw-r--r--.   1 graylog graylog  213 Jul 26 23:51 lookup_users.csv
-rw-r--r--.   1 graylog graylog  183 Feb  7  2022 mongodb_ftdc.te
[root@graylog graylog_user]#

The beat input configuration should match where those certs are at.

Hello Mister
First thanks a lot… And then Can I follow the manual in graylog web page documents?
Uploading: image.png…
Uploading: image.png(1)…
I am stopped in Converting an Existing Java Keystore into a Private Key/Certificate Pair
Where is the java keystore?
is it in /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts ?
sorry to be so stupid, this is my new work and it is the first time that in a job they ask to myself about this things, before my jobs are change tv, repair PC, cables, adaptators etcetera but not this… Well anytime should be the first time… ok keep going to this.
I started new configuration, I am stopped in this step, Converting an Existing Java Keystore into a Private Key/Certificate Pair, I am going to follow the manual but the manual is confuse to me so …
I have to do all of this and to work in a sidecar in a client machine with filebeat configuration.
How can I see the configuration of the keys that you tell me?
this errors are in the frontend (browser) with the configuration of graylog-server is done ( in the office work) and try to receive logs from a machine with sidecar an filebeat.
Where can I get a manual with the serial steps, one to one? OH my god?

thanks best regards.

this is the error at home, sme that in the work:


my config filebeat:
image
the files cert.pem pkcs5-plain.pem pkcs8-encrypted.pem pkcs8-plain.pem are in /etc/graylog in the server machine and the same files in the same directory in the client machine.
Maybe could be a java problem

thanks again.

this error:


with this configuration

I am doing something wrong with this instrucctions… please help
Uso de HTTPS (graylog.org)

thanks again.

Hello @jgutie45

Just a reminder, commands and configuration may not work for you because everyone environment is a little different, keep that in mind.

Depends on the System and what JAVA version you have. First, find your version your using.

java -version

Then look for “CACERTS”

root # locate cacerts

Results

[root@graylog graylog_user]# java -version
openjdk version "11.0.17" 2022-10-18 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.17.0.8-2.el7_9) (build 11.0.17+8-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.17.0.8-2.el7_9) (build 11.0.17+8-LTS, mixed mode, sharing)
[root@graylog graylog_user]# locate cacerts
/etc/pki/ca-trust/extracted/java/cacerts
/etc/pki/java/cacerts
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/lib/security/cacerts.backup
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre/lib/security/cacerts
/usr/lib/jvm/java-11-openjdk-11.0.17.0.8-2.el7_9.x86_64/lib/security/cacerts
/usr/share/elasticsearch/jdk/lib/security/cacerts
/usr/share/opensearch/jdk/lib/security/cacerts
[root@graylog graylog_user]#

My default key store is located as shown below

/usr/lib/jvm/java-11-openjdk-11.0.17.0.8-2.el7_9.x86_64/lib/security/cacerts

Example of the steps needed. I remind you this alos depend on your setup you may want to adjust it to your needs.

1. mkdir /etc/ssl/certs/graylog/ && cd /etc/ssl/certs/graylog/
2. openssl req -x509 -days 1095 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem
3. openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem
4. openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:secret
5. openssl req -config openssl-graylog.cnf -out graylog.csr -new -newkey rsa:2048 -nodes -keyout graylog.key
6. openssl req -x509 -sha512 -nodes -days 1095 -newkey rsa:2048 -config openssl-graylog.cnf -keyout graylog.key -out graylog.crt
7. openssl req -config openssl-graylog.cnf -out graylog.csr -key graylog.key -new
8. openssl x509 -x509toreq -in graylog.crt -out graylog.csr -signkey graylog.key
9. openssl pkcs12 -export -in graylog.crt -inkey graylog.key -out keystore.pfx
10. openssl pkcs12 -in keystore.pfx -nokeys -out graylog-certificate.pem
11. openssl pkcs12 -in keystore.pfx -nocerts -out graylog-pkcs5.pem
12. openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem
13. keytool -import -trustcacerts -file graylog.crt -alias graylog.enseva-labs.net -keystore graylog_keystore.jks -storepass secret
14. keytool -list -v -keystore graylog_keystore.jks -alias graylog.enseva-labs.net
15. keytool -importkeystore -srckeystore graylog_keystore.jks -destkeystore keystore.p12 -deststoretype PKCS12
16. openssl pkcs12 -in keystore.p12 -nokeys -out graylog-certificate.pem
17. openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem
18. cp -a “/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/jre/lib/security/cacerts” /etc/ssl/certs/graylog/graylog-key.jks
19. keytool -importcert -keystore graylog.jks -storepass changeit (secret) -alias graylog.domain.com -file cert.pem

OR if you want to use the JAVA default keystore then you would want to use this command,
and again you need to change this for your setup.

keytool -importcert -keystore /usr/lib/jvm/java-11-openjdk-11.0.17.0.8-2.el7_9.x86_64/lib/security/cacerts -storepass changeit (secret) -alias graylog.domain.com -file cert.pem

If your new at creating self-sign certificates and Graylog then make sure your full path to certs are all the same configuration.

EXample:

Graylog configuration file

http_bind_address = graylog.domain.com:9000
http_publish_uri = https://graylog.domain.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/graylog/graylog-certificate.pem
http_tls_key_file = /etc/graylog/graylog-key.pem
http_tls_key_password = secret

Side Note:

In order for the JVM to pick up the new trust store, it has to be started with the JVM parameter -Djavax.net.ssl.trustStore=/path/to/cacerts.jks.

Do you notice it say "NEW " keystore, then if your using the default there is no need for this configuration. you can put in the Full-path to cacerts for testing/troubleshooting.

NOTE: If you do put the passphrase on the certificates , ensure they are correct.

Next is the INPUT.

Once completed restart graylog service to ensure the certs get picked up.

First, There is no need to send a email to community members for this type of issue , please be patient, we do this for free to help others.

Second, when you are configuring HTTPS and/or TCP/TLS look at your logs, see what’s going on. For example, Restart Graylog service and tail -f /var/log/graylog.log file then watch what’s going on, there is probably a clue on what you need to do or google.

Third, there are a lot ( i.e., many) posts here about HTTPS and examples. Talk a look it may help also showing your work will help us help you, the copy & paste probably not a good idea here.

Hope that helps

Thanks Mr gsmith an sorry, all is about my job and in the jobs is to yesterday, you know.

I am going to try all that you send me.

really sorry.

Have a nice day.

Hi again

I was making all configuration and no error but graylog-server doesn´t start up
I put in this file /etc/default/graylog-server this configuration in the # Default Java options for heap and garbage collection:

Path to the java executable.

JAVA=/usr/bin/java

Default Java options for heap and garbage collection.

GRAYLOG_SERVER_JAVA_OPTS=“-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/graylog.jks -Djavax.net.ssl.trustStorePassword=changeit”

Avoid endless loop with some TLSv1.3 implementations.

GRAYLOG_SERVER_JAVA_OPTS=“$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true”

Fix for log4j CVE-2021-44228

GRAYLOG_SERVER_JAVA_OPTS=“$GRAYLOG_SERVER_JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true”

Pass some extra args to graylog-server. (i.e. “-d” to enable debug mode)

GRAYLOG_SERVER_ARGS=“”

Program that will be used to wrap the graylog-server command. Useful to

support programs like authbind.

GRAYLOG_COMMAND_WRAPPER=“”
Sorry to ask again and again
in the server.conf I put
http_enable_tls = true
http_tls_cert_file = /etc/graylog/graylog-certificate.pem
http_tls_key_file = /etc/graylog/graylog-key.pem
http_tls_key_password = secret
If my openssl-graylog.cnf was the netx how would it be? Sorry I can not put the real country and etcetera…

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

Details about the issuer of the certificate

[req_distinguished_name]
C = US
ST = NW
L = NW
O = FMI
OU = Sistems
CN = graylog.localhost

[v3_req]
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

IP addresses and DNS names the certificate should include

Use IP.### for IP addresses and DNS.### for DNS names,

with “###” being a consecutive number.

[alt_names]
IP.1 = 192.168.194.130
DNS.1 = graylog.localhost

THis is very awful, horrible, scary, desperating and bore… Thanks to help me, really means a lot to me.
Graylog manual is a disaster, a very high disaster.

Have a nice day, better than mine I hope so.

I copy the graylog-certificate.pem and graylog-key.pem from /etc/ssl/certs/graylog/ to /etc/graylog/
¿OK?
I am very disappointed with the graylog software.

bye, thanks, have a nice day, best regards.

1 Like

Hello,

If graylog is not starting up then what does the logs show, this should give you an idea whats going on.Then we can focus on that.

This looks good, are you using you own keystore or JAVA default one called cacerts?
And do you have these setting configure?

http_bind_address = graylog.domain.com:9000
http_publish_uri = https://graylog.domain.com:9000/

If you have a dns server are these IP.1 = 192.168.194.130, DNS.1 = graylog.localhost reachable of the network?
If not you can add then to your hosts file on the Graylog server.

To be honest its not the software, it the configurations made. Need to know how all three parts work together first. The HTTPS configuration are pretty hard to get , but once you understand its a lot easier. When I first started it was a mess for weeks.

These posts help me to understand how TCP/TLS works with Graylog and how to create self-signed certificates. perhap looking over them they may help.

Hello… good mooooooooooooooorning
Graylog-server works again… I do not any things clear.
I thing I am using my own store of keys because in the file /etc/default/graylog-server I put this:

Default Java options for heap and garbage collection.

GRAYLOG_SERVER_JAVA_OPTS=“-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/graylog_keystore.jks -Djavax.net.ssl.trustStorePassword=changeit”
Is it correct?

The machine where is installed graylog (192.168.194.130) is seen in all the red and it can go out to internet. the exactly configuration is:

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

Details about the issuer of the certificate

[req_distinguished_name]
C = US
ST = Some-State
L = Some-City
O = My Company
OU = My Division
CN = graylog.example.com

[v3_req]
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

IP addresses and DNS names the certificate should include

Use IP.### for IP addresses and DNS.### for DNS names,

with “###” being a consecutive number.

[alt_names]
IP.1 = 192.168.194.130
DNS.1 = graylog.example.com

DNS.1 is the name of the machice where is installed graylog-server? I am a bit confuse with this.

Now I have that configure the SIDECAR and the FILEBEAT inthe frontend, can you help me with that?

If you need anything more please tellme about that.
thanks again for all. Have a nice day. Best regards.

http_bind_address = graylog.domain.com:9000
http_publish_uri = https://graylog.domain.com:9000/
I don´t have this configured, is it essential?
will be my configuration same?

Thanks again. I am working very hard in it.

Hello

yes it is, @jgutie45 to be honest you really need to read to documentation man

1 Like