Certificate issue

[docgyan (Girish Mayigowda)]

**1. Issue
Hello, Graylog was working fine until I replaced the IP address with the fully qualified domain and also added the new certificate for the FQDN
Below is the error:

2025-06-17T11:39:13.461Z WARN  [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 9 ms)
2025-06-17T11:39:14.193Z WARN  [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 5 ms)
2025-06-17T11:39:15.031Z WARN  [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 5 ms)

2. Describe your environment:

• OS Information:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"

• Package Version:

graylog-datanode                         6.1.10-1                                amd64        Graylog data node
graylog-server                           6.1.10-1                                amd64        Graylog server
mongodb-org-server                       7.0.18                                  amd64        MongoDB database server

Graylog.conf:

is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = XXXXXXXXXLSsSNDpOQzzc1gP8e44FRHa2ITzQsNxg48YXieWVEAzWo5jqGkH5MRXAg1B67Ixla9pDgKZHLE-ekAgXXXXXXXx
root_password_sha2 = fc9b63a79bfd52787b6e93de9befd76acd8971384e3e222222221800bedfef81
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.5.X.X:9000
http_publish_uri = http://xyz.com:9000/
http_external_uri = http://xyz.com9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/graylog/ssl/wildcard.pem
http_tls_key_file = /etc/graylog/ssl/wildcard.key
stream_aware_field_types=false
disabled_retention_strategies = none,close
allow_leading_wildcard_searches = false
allow_highlighting = false
field_value_suggestion_mode = on
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 72h
message_journal_max_size =100gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
integrations_scripts_dir = /usr/share/graylog-server/scripts

datanode.conf:

node_id_file = /etc/graylog/datanode/node-id
config_location = /etc/graylog/datanode
password_secret = XXXXXXXXXLSsSNDpOQzzc1gP8e44FRHa2ITzQsNxg48YXieWVEAzWo5jqGkH5MRXAg1B67Ixla9pDgKZHLE-ekAgUXXXXXXX
root_password_sha2 =
mongodb_uri = mongodb://localhost/graylog
opensearch_location = /usr/share/graylog-datanode/dist
opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /var/lib/graylog-datanode/opensearch/data
opensearch_logs_location = /var/log/graylog-datanode/opensearch
opensearch_heap = 8g

3. What steps have you already taken to try and solve the problem?

As the error says that the issue is with the trust manager i tried to search for it but failed.

4. How can the community help?

Please help me to resolve this certificate issue.

[Tdvorak (Tomas Dvorak)]

Hey @docgyan
I not quite sure what’s happening with your setup, so I’ll try to ask some questions and maybe we’ll figure it out together.

Your setup consists of one server and one data node? You are managing your server certificate externally and you have changed the certificate recently? The error log you are posting is from server.log? Is the datanode starting correctly and running fine or is there anything problematic on data node side? Are there any other exceptions in the server log?

Thanks,
Tomas

[docgyan (Girish Mayigowda)]

@Tdvorak

Your setup consists of one server and one data node? You are managing your server certificate externally and you have changed the certificate recently? The error log you are posting is from server.log? Is the datanode starting correctly and running fine or is there anything problematic on data node side? Are there any other exceptions in the server log?

Hello @Tdvorak Thank you for responding.

Below are the answers for your questions:
Your setup consists of one server and one data node?
I have configured Graylog server and datanode both in a same server

You are managing your server certificate externally and you have changed the certificate recently?

Yes, I have added a new external certificate.
Earlier i was accessing Graylog with the IP address and now i am trying to access it via URL and it works as well but data is not loading.

The error log you are posting is from server.log?
Yes and below are the recent logs

Server.log:

2025-06-18T02:01:26.876Z INFO  [Message] Ignoring invalid or reserved key filebeat_kubernetes_labels_app_kubernetes_io/component for message 24b409c6-4be8-11f0-ab61-0022489fXXXX
2025-06-18T02:01:26.876Z INFO  [Message] Ignoring invalid or reserved key filebeat_kubernetes_labels_app_kubernetes_io/instance for message 24b409c6-4be8-11f0-ab61-0022489fXXXX
2025-06-18T02:01:27.214Z WARN  [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 4 ms)
2025-06-18T02:01:29.225Z WARN  [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 5 ms)
2025-06-18T02:01:31.254Z WARN  [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 31 ms)

datanode.log:
Below is the only errror i am seeing and rest all are info logs:

2025-06-18T00:00:03.311Z INFO  [OpensearchProcessImpl] ERROR StatusConsoleListener Could not define attribute view on path "/var/log/graylog-datanode/opensearch/datanode-cluster_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
2025-06-18T00:00:03.317Z INFO  [OpensearchProcessImpl]  java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation")
2025-06-18T00:00:03.317Z INFO  [OpensearchProcessImpl]  at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
2025-06-18T00:00:03.317Z INFO  [OpensearchProcessImpl]  at java.base/java.security.AccessController.checkPermission(AccessController.java:1071)
2025-06-18T00:00:03.317Z INFO  [OpensearchProcessImpl]  at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
2025-06-18T00:00:03.318Z INFO  [OpensearchProcessImpl]  at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.checkWriteExtended(UnixFileAttributeViews.java:195)
2025-06-18T00:00:03.318Z INFO  [OpensearchProcessImpl]  at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:264)
2025-06-18T00:00:03.318Z INFO  [OpensearchProcessImpl]  at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:299)
2025-06-18T00:00:03.318Z INFO  [OpensearchProcessImpl]  at org.apache.logging.log4j.core.util.FileUtils.defineFilePosixAttributeView(FileUtils.java:181)
2025-06-18T00:00:03.318Z INFO  [OpensearchProcessImpl]  at org.apache.logging.log4j.core.appender.FileManager.defineAttributeView(FileManager.java:216)
2025-06-18T00:00:03.318Z INFO  [OpensearchProcessImpl]  at org.apache.logging.log4j.core.appender.FileManager.createOutputStream(FileManager.java:203)
2025-06-18T00:00:03.318Z INFO  [OpensearchProcessImpl]  at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)

Is the datanode starting correctly and running fine or is there anything problematic on data node side?
It is up and running and below is the status of the datanode and graylog:

● graylog-datanode.service - Graylog data node
     Loaded: loaded (/usr/lib/systemd/system/graylog-datanode.service; enabled; preset: enabled)
     Active: active (running) since Tue 2025-06-17 09:34:40 UTC; 16h ago
       Docs: http://docs.graylog.org/
   Main PID: 2026454 (java)
      Tasks: 188 (limit: 38488)
     Memory: 16.8G (peak: 18.9G)
        CPU: 15h 10min 10.458s

● graylog-server.service - Graylog server
     Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; preset: enabled)
     Active: active (running) since Tue 2025-06-17 10:21:58 UTC; 15h ago
       Docs: http://docs.graylog.org/
   Main PID: 2054799 (graylog-server)
      Tasks: 285 (limit: 38488)
     Memory: 13.5G (peak: 16.2G)
        CPU: 1d 9h 53min 8.347s

Are there any other exceptions in the server log?
Apart from the complains about the Journal filling up 95% I don't see any other exceptions.

[Tdvorak (Tomas Dvorak)]

OK, thank you! So far I haven’t seen anything suspicious. Let’s talk about the certificate itself. Is it selfsigned? Is the chain correct and complete? Does your browser trust it by default?

Have you added the cert or CA to the JVM truststore?

Is there any error in browser console, when you try to load search results?

[docgyan (Girish Mayigowda)]

Hello @Tdvorak , Its not self signed but its issues by “COMODO CA Limited” and my browser trust it by default.

Common Name (CN)	*.xyz.xyz.com
Organization (O)	XYZ
Organizational Unit (OU)
Common Name (CN)	COMODO RSA Organization Validation Secure Server CA
Organization (O)	COMODO CA Limited
Organizational Unit (OU)
Issued On	Tuesday, August 6, 2024 at 5:30:00 AM
Expires On	Thursday, August 7, 2025 at 5:29:59 AM

I have not added cert to JVM truststore, If I have to then please let know how can I achieve it.
I don’t see any error in the browser console when I try to load search results.

Thanks.

[Tdvorak (Tomas Dvorak)]

I’d point you to https://graylog.org/post/how-to-guide-securing-graylog-with-tls/, look for the Java Key Store section.

Generally, an up-to-date cert is probably issued by a CA that’s included in the JVM truststore. But it may be that you are using an outdated JVM distribution, where the CA cert is already expired or not present. In that situation, it would make sense to add your cert (or better, the CA public cert) to the truststore.

[docgyan (Girish Mayigowda)]

Hello @Tdvorak I did follow the steps mentioned in the link and i see the new error in server.log

2025-06-19T04:54:10.336Z ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:772) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:584) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:298) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:381) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:358) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:270) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:55) [graylog.jar:?]
        Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
        Caused by: java.security.GeneralSecurityException: org.bouncycastle.pkcs.PKCSIOException: malformed data: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:88) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:379) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:196) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:162) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:64) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[graylog.jar:?]
                at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
        Caused by: org.bouncycastle.pkcs.PKCSIOException: malformed data: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.parseBytes(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:66) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.doBuildKeyStore(PemKeyStore.java:99) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:85) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:379) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:196) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:162) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:64) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[graylog.jar:?]
                at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
        Caused by: java.lang.IllegalArgumentException: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
                at org.bouncycastle.asn1.ASN1Sequence.getInstance(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.asn1.x509.AlgorithmIdentifier.getInstance(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.asn1.pkcs.EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.asn1.pkcs.EncryptedPrivateKeyInfo.getInstance(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.parseBytes(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:66) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.doBuildKeyStore(PemKeyStore.java:99) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:85) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:379) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:196) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:162) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:64) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[graylog.jar:?]
                at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
2025-06-19T04:54:10.340Z INFO  [Server] SIGNAL received. Shutting down.
2025-06-19T04:54:10.344Z INFO  [GracefulShutdown] Graceful shutdown initiated.
2025-06-19T04:54:10.344Z INFO  [GracefulShutdown] Node status: [Override lb:DEAD [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2025-06-19T04:54:13.346Z INFO  [GracefulShutdown] Goodbye.

[docgyan (Girish Mayigowda)]

Hello @Tdvorak ,

I did few other changes and now i am getting below error:

ERROR [CmdLineTool] Invalid configuration
com.github.joschi.jadconfig.ValidationException: Unreadable or missing HTTP private key: /etc/graylog/ssl/privkey.key

I followed the below mentioned thread but the solution was for self singed certificates and the vendor provided certificate.

Setting up HTTPS connection error - Unreadable or missing HTTP private key

@gsmith @ttsandrew I have found the solution and now successfully completed the HTTPS setup. The browser is showing ‘Not secure’ because its a self-signed certificate and new versions of browsers will show it like this. [image] I have made below change in ‘openssl-graylog.cnf’ file. Without this change browsers were throwing SSL key errors as shown in earlier messages. [image] I would suggest making change in Graylog official documentation and update this with above change. Thanks a l…

Could you please help if i have missed something.

Thanks,
Girish

[docgyan (Girish Mayigowda)]

Please ignore it, After giving the full permission of the folder and also the certificates files it started to work.

Thanks a lot for the initial help @Tdvorak