Troubleshoot Graylog Open install - Cannot provision certificate

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Hi, I am installing graylog-open on an ubuntu VM. I have got to the stage where I need to provision a certificate on the datanode. I am having difficulties here though as i dont get the option. I ended up using ChatGPT to try fix it.. and I fear I may have made it worse. So I would love some assistance in getting it working.

2. Describe your environment:

  • OS Information:

Distributor ID: Ubuntu
Description: Ubuntu 24.04.2 LTS
Release: 24.04
Codename: noble

  • Package Version:

netgrayadmin@net-graylog:~$ dpkg -l | grep -E “.(elasticsearch|graylog|mongo).”
ii graylog-6.3-repository 1-1 all Package to install Graylog 6.3 GPG key and repository
ii graylog-datanode 6.3.1-1 amd64 Graylog data node
ii graylog-server 6.3.1-1 amd64 Graylog server
ii mongodb-database-tools 100.12.2 amd64 mongodb-database-tools package provides tools for working with the MongoDB server:
ii mongodb-mongosh 2.5.5 amd64 MongoDB Shell CLI REPL Package
hi mongodb-org 7.0.21 amd64 MongoDB open source document-oriented database system (metapackage)
ii mongodb-org-database 7.0.21 amd64 MongoDB open source document-oriented database system (metapackage)
ii mongodb-org-database-tools-extra 7.0.21 amd64 Extra MongoDB database tools
ii mongodb-org-mongos 7.0.21 amd64 MongoDB sharded cluster query router
ii mongodb-org-server 7.0.21 amd64 MongoDB database server
ii mongodb-org-shell 7.0.21 amd64 MongoDB shell client
ii mongodb-org-tools 7.0.21 amd64 MongoDB tools

  • Service logs, configurations, and environment variables:
    sudo journalctl -u graylog-datanode.service --since “10 minutes ago”

Jul 17 11:21:58 net-graylog systemd[1]: graylog-datanode.service: Main process exited, code=exited, status=255/EXCEPTION
Jul 17 11:21:58 net-graylog systemd[1]: graylog-datanode.service: Failed with result ‘exit-code’.
Jul 17 11:21:58 net-graylog systemd[1]: graylog-datanode.service: Consumed 6.243s CPU time.
Jul 17 11:22:08 net-graylog systemd[1]: graylog-datanode.service: Scheduled restart job, restart counter is at 8327.
Jul 17 11:22:08 net-graylog systemd[1]: Started graylog-datanode.service - Graylog data node.
Jul 17 11:22:11 net-graylog systemd[1]: graylog-datanode.service: Main process exited, code=exited, status=255/EXCEPTION
Jul 17 11:22:11 net-graylog systemd[1]: graylog-datanode.service: Failed with result ‘exit-code’.
Jul 17 11:22:11 net-graylog systemd[1]: graylog-datanode.service: Consumed 6.244s CPU time.
##########

netgrayadmin@net-graylog:~$ sudo systemctl status graylog-datanode.service
graylog-datanode.service - Graylog data node
Loaded: loaded (/usr/lib/systemd/system/graylog-datanode.service; enabled; preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2025-07-17 11:22:48 UTC; 6s ago
Docs: http://docs.graylog.org/
Process: 924327 ExecStart=/usr/share/graylog-datanode/bin/graylog-datanode datanode -f /etc/graylog/datanode/datanode.conf (code=exited, status=255/EXCEPTION)
Main PID: 924327 (code=exited, status=255/EXCEPTION)
CPU: 5.486s

netgrayadmin@net-graylog:~$ sudo tail -n 50 /var/log/graylog-datanode/datanode.log
at org.graylog.datanode.configuration.OpensearchConfigurationService.lambda$get$0(OpensearchConfigurationService.java:107) ~[graylog-datanode.jar:?]
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.get(OpensearchConfigurationService.java:108) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.triggerConfigurationChangedEvent(OpensearchConfigurationService.java:120) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.startUp(OpensearchConfigurationService.java:67) ~[graylog-datanode.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:63) ~[guava-33.4.8-jre.jar:?]
at com.google.common.util.concurrent.Callables.lambda$threadRenaming$1(Callables.java:104) ~[guava-33.4.8-jre.jar:?]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source) ~[?:?]
at java.base/sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source) ~[?:?]
at java.base/java.security.KeyStore.load(Unknown Source) ~[?:?]
at org.graylog.datanode.configuration.DatanodeKeystore.loadKeystore(DatanodeKeystore.java:159) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.DatanodeKeystore.hasSignedCertificate(DatanodeKeystore.java:90) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.variants.DatanodeKeystoreOpensearchCertificatesProvider.isConfigured(DatanodeKeystoreOpensearchCertificatesProvider.java:37) ~[graylog-datanode.jar:?]
at org.graylog.datanode.opensearch.configuration.beans.impl.OpensearchSecurityConfigurationBean.lambda$buildConfigurationPart$0(OpensearchSecurityConfigurationBean.java:99) ~[graylog-datanode.jar:?]
at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source) ~[?:?]
at java.base/java.util.Spliterators$ArraySpliterator.tryAdvance(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReferencePipeline.findFirst(Unknown Source) ~[?:?]
at org.graylog.datanode.opensearch.configuration.beans.impl.OpensearchSecurityConfigurationBean.buildConfigurationPart(OpensearchSecurityConfigurationBean.java:100) ~[graylog-datanode.jar:?]
at org.graylog.datanode.opensearch.configuration.beans.impl.OpensearchSecurityConfigurationBean.buildConfigurationPart(OpensearchSecurityConfigurationBean.java:56) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.lambda$get$0(OpensearchConfigurationService.java:107) ~[graylog-datanode.jar:?]
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.get(OpensearchConfigurationService.java:108) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.triggerConfigurationChangedEvent(OpensearchConfigurationService.java:120) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.startUp(OpensearchConfigurationService.java:67) ~[graylog-datanode.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:63) ~[guava-33.4.8-jre.jar:?]
at com.google.common.util.concurrent.Callables.lambda$threadRenaming$1(Callables.java:104) ~[guava-33.4.8-jre.jar:?]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
2025-07-17T11:24:14.689Z INFO [Datanode] SIGNAL received. Shutting down.
2025-07-17T11:24:14.689Z INFO [GracefulShutdown] Graceful shutdown initiated.
2025-07-17T11:24:14.690Z INFO [GracefulShutdown] Goodbye.

3. What steps have you already taken to try and solve the problem?

I confirmed the password_secret is the same for the server and data node.

Tried restarting services, reinstall datanode

4. How can the community help?

I am not an expert in linux in any fashion, but I believe my issue is easlily resolved, not sure how. If someone could steer me that would be great!

Cheers

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hey @anosrep

Either the keystore password is wrong,
Or the keystore file itself is corrupted or mismatched with the password.

sudo systemctl stop graylog-datanode
sudo mv /etc/graylog/datanode/keystore.p12 /etc/graylog/datanode/keystore.p12.bak
sudo graylog-datanode-ctl reconfigure
sudo systemctl start graylog-datanode

what this will do…
Remove the corrupted keystore.
Let Graylog regenerate it.
Use the password in datanode.conf (or default).

2 Likes
3

Yeah, the keystore seems broken. Have you @anosrep managed to solve it by the recommendation from @gsmith?

4

Good morning, thanks for your reply, I will be attempting this shortly and will feedback :slight_smile:

5

Hi mate, this must be my problem!

the keystore.p12 file does not exist, I am guessing I have deleted this during the troubleshooting phase previously by mistake. See below output:

netgrayadmin@net-graylog:~$ sudo systemctl stop graylog-datanode
[sudo] password for netgrayadmin:
netgrayadmin@net-graylog:~$ sudo mv /etc/graylog/datanode/keystore.p12 /etc/graylog/datanode/keystore.p12.bak
mv: cannot stat ‘/etc/graylog/datanode/keystore.p12’: No such file or directory
netgrayadmin@net-graylog:~$ sudo graylog-datanode-ctl reconfigure
sudo: graylog-datanode-ctl: command not found
netgrayadmin@net-graylog:~$

6

This suggests that you have a keystore somewhere, but the key is not matching. This happens if you change password_secret after the initial setup already generated a keystore.

What’s the opensearch_config_location value in your datanode.conf? This path is where datanode stores its keystore.

1 Like
7

Hi Tomas,

Yes i did do that :expressionless:

Ok great it is here:

Root directory of the used opensearch distribution

opensearch_location = /usr/share/graylog-datanode/dist

opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /var/lib/graylog-datanode/opensearch/data
opensearch_logs_location = /var/log/graylog-datanode/opensearch
opensearch_heap = 4GB

netgrayadmin@net-graylog:~$ ls -ltr /var/lib/graylog-datanode/opensearch/config
total 16
-rw-r–r-- 1 graylog-datanode graylog-datanode 36 Jul 15 09:58 datanode.lock
-rw-r–r-- 1 graylog-datanode graylog-datanode 5628 Jul 15 11:18 keystore.jks
drwxr-xr-x 2 graylog-datanode graylog-datanode 4096 Jul 18 10:08 native_libs

I tried to open it with nano the keystore.jks file but it is blank.

8

Ok, then I suggest stopping both datanode and graylog server, deleting /var/lib/graylog-datanode/opensearch/config/keystore.jks and if you changed the password during the setup process, you’ll need to delete the content of graylog database in mongodb, as some of the entries there are encrypted with the old password and not readable anymore.

Afterwards you can start both services and start again with the preflight setup.

1 Like
9

Hi Tomas,

Thanks for this, is this the correct way to do this within mongosh:

test> use graylog db.dropDatabase()
switched to db graylog
graylog> exit

?

10

Hi All,

I dropped the database after stopping services:

test> 

test> use graylog
switched to db graylog
graylog> 

graylog> db.dropDatabase()
{ ok: 1, dropped: 'graylog' }
graylog> 

graylog> 

graylog> exit

Then I started the services however, still getting the cert provisioning error in the GUI:

image
image1689×775 43.9 KB

When I check the services, the datanode still seems to have a problem:

netgrayadmin@net-graylog:~$ sudo systemctl status graylog-datanode.service 
● graylog-datanode.service - Graylog data node
     Loaded: loaded (/usr/lib/systemd/system/graylog-datanode.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2025-07-18 10:38:43 UTC; 182ms ago
       Docs: http://docs.graylog.org/
    Process: 27496 ExecStart=/usr/share/graylog-datanode/bin/graylog-datanode datanode -f /etc/graylog/datanode/datanode.conf (code=exited, status=1/FAILURE)
   Main PID: 27496 (code=exited, status=1/FAILURE)
        CPU: 4.586s

Seeing alot of this within the logs:

Caused by: java.io.IOException: keystore password was incorrect
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source) ~[?:?]
        at java.base/sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source) ~[?:?]
        at java.base/java.security.KeyStore.load(Unknown Source) ~[?:?]
        at org.graylog.datanode.configuration.DatanodeKeystore.loadKeystore(DatanodeKeystore.java:159) ~[graylog-datanode.jar:?]
        at org.graylog.datanode.configuration.DatanodeKeystore.hasSignedCertificate(DatanodeKeystore.java:90) ~[graylog-datanode.jar:?]
        at org.graylog.datanode.bootstrap.preflight.DatanodeKeystoreCheck.checkCertificateRenewal(DatanodeKeystoreCheck.java:105) ~[graylog-datanode.jar:?]
        ... 8 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source) ~[?:?]
        at java.base/sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source) ~[?:?]
        at java.base/java.security.KeyStore.load(Unknown Source) ~[?:?]
        at org.graylog.datanode.configuration.DatanodeKeystore.loadKeystore(DatanodeKeystore.java:159) ~[graylog-datanode.jar:?]
        at org.graylog.datanode.configuration.DatanodeKeystore.hasSignedCertificate(DatanodeKeystore.java:90) ~[graylog-datanode.jar:?]
        at org.graylog.datanode.bootstrap.preflight.DatanodeKeystoreCheck.checkCertificateRenewal(DatanodeKeystoreCheck.java:105) ~[graylog-datanode.jar:?]
        ... 8 more

Maybe I need to start fresh?

11

Have you also deleted the keystore, together with the mongodb content?

1 Like
12

Ahh I did forget to do that, I have done that now and made some progress! thanks.

Its sat at this stage now for around 5minutes:

image
image1367×408 25.8 KB

Is there anything else I need to do in the backend to get this certificate working?
I can ping the hostname of the server ok from itself, set a dns entry in the hosts file.
I checked hostname in datanode.conf file looks ok.

13

Any exceptions in the datanode log?

14

Hi Tomas,

2025-07-18T12:54:32.606Z ERROR [ProcessStateMachine] Failed to fire event PROCESS_STARTED
java.lang.RuntimeException: Failed to execute opensearch cli/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/bin/opensearch-keystore
STDOUT/STDERR: Invalid initial heap size: -Xms4GB
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

I have 8GB of ram so increased heap size to:

# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms4g -Xmx4g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow"

Then i stopped and started the 2 services and getting this now:

2025-07-18T13:02:22.257Z INFO  [OpensearchSecurityConfigurationBean] Opensearch Transport certificate has following serial number: 1752843740310
2025-07-18T13:02:22.258Z INFO  [OpensearchSecurityConfigurationBean] Opensearch Transport certificate has following validity: 2025-07-18T13:02:20.000+0000 - 2030-07-17T13:02:20.000+0000
2025-07-18T13:02:22.262Z INFO  [JerseyService] Server configuration changed, restarting Data node REST API to apply security changes
2025-07-18T13:02:22.263Z INFO  [OpensearchProcessService] OpenSearch starting up
2025-07-18T13:02:22.264Z INFO  [JerseyService] Shutting down HTTP listener at <10.157.255.250:8999>
2025-07-18T13:02:22.264Z INFO  [OpensearchCommandLineProcess] Creating opensearch keystore
2025-07-18T13:02:22.268Z INFO  [NetworkListener] Stopped listener bound to [10.157.255.250:8999]
2025-07-18T13:02:22.326Z INFO  [NetworkListener] Started listener bound to [10.157.255.250:8999]
2025-07-18T13:02:22.327Z INFO  [HttpServer] [HttpServer-2] Started.
2025-07-18T13:02:22.328Z INFO  [JerseyService] Started REST API at <10.157.255.250:8999>
2025-07-18T13:02:22.468Z ERROR [ProcessStateMachine] Failed to fire event PROCESS_STARTED
java.lang.RuntimeException: Failed to execute opensearch cli/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/bin/opensearch-keystore
STDOUT/STDERR: Invalid initial heap size: -Xms4GB
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
15

Try changing the opensearch_heap setting to 4g and restart the datanode service.

16

Hi Tomas,

Is this within the datanode.conf file?

I just checked and it is set to that already:

#### OpenSearch folders
#
# set these if you need OpenSearch to be located in a special place or want to include an existing version
#
# Root directory of the used opensearch distribution
opensearch_location = /usr/share/graylog-datanode/dist

opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /var/lib/graylog-datanode/opensearch/data
opensearch_logs_location = /var/log/graylog-datanode/opensearch
opensearch_heap = 4GB
17

Yes, correct location, but the last line has to be

opensearch_heap = 4g

in order to configure 4GB. Java accepts only <size>[g|G|m|M|k|K] format, so the additional B makes the configuration value invalid and crashes the managed opensearch process. That’s why your preflight never finishes.

2 Likes
18

Hi Tomas,

You nailed it!

Thank you so much, that’s it operational now :smiley:

image
image702×408 18.6 KB

appreciate your speedy support.

Many thanks!

1 Like
19

Glad I could help! Enjoy your new setup :partying_face: