**1. Issue Hello, Graylog was working fine until I replaced the IP address with the fully qualified domain and also added the new certificate for the FQDN
Below is the error:
2025-06-17T11:39:13.461Z WARN [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 9 ms)
2025-06-17T11:39:14.193Z WARN [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 5 ms)
2025-06-17T11:39:15.031Z WARN [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 5 ms)
Hey @docgyan
I not quite sure what’s happening with your setup, so I’ll try to ask some questions and maybe we’ll figure it out together.
Your setup consists of one server and one data node? You are managing your server certificate externally and you have changed the certificate recently? The error log you are posting is from server.log? Is the datanode starting correctly and running fine or is there anything problematic on data node side? Are there any other exceptions in the server log?
Below are the answers for your questions:
Your setup consists of one server and one data node? I have configured Graylog server and datanode both in a same server
You are managing your server certificate externally and you have changed the certificate recently?
Yes, I have added a new external certificate.
Earlier i was accessing Graylog with the IP address and now i am trying to access it via URL and it works as well but data is not loading.
The error log you are posting is from server.log? Yes and below are the recent logs
Server.log:
2025-06-18T02:01:26.876Z INFO [Message] Ignoring invalid or reserved key filebeat_kubernetes_labels_app_kubernetes_io/component for message 24b409c6-4be8-11f0-ab61-0022489fXXXX
2025-06-18T02:01:26.876Z INFO [Message] Ignoring invalid or reserved key filebeat_kubernetes_labels_app_kubernetes_io/instance for message 24b409c6-4be8-11f0-ab61-0022489fXXXX
2025-06-18T02:01:27.214Z WARN [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 4 ms)
2025-06-18T02:01:29.225Z WARN [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 5 ms)
2025-06-18T02:01:31.254Z WARN [ProxiedResource] Failed to call API on node <1bb277fc-06c4-4bcf-8359-5XXXXXe97320>, cause: None of the TrustManagers trust this certificate chain. (duration: 31 ms)
datanode.log:
Below is the only errror i am seeing and rest all are info logs:
2025-06-18T00:00:03.311Z INFO [OpensearchProcessImpl] ERROR StatusConsoleListener Could not define attribute view on path "/var/log/graylog-datanode/opensearch/datanode-cluster_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
2025-06-18T00:00:03.317Z INFO [OpensearchProcessImpl] java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation")
2025-06-18T00:00:03.317Z INFO [OpensearchProcessImpl] at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
2025-06-18T00:00:03.317Z INFO [OpensearchProcessImpl] at java.base/java.security.AccessController.checkPermission(AccessController.java:1071)
2025-06-18T00:00:03.317Z INFO [OpensearchProcessImpl] at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
2025-06-18T00:00:03.318Z INFO [OpensearchProcessImpl] at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.checkWriteExtended(UnixFileAttributeViews.java:195)
2025-06-18T00:00:03.318Z INFO [OpensearchProcessImpl] at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:264)
2025-06-18T00:00:03.318Z INFO [OpensearchProcessImpl] at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:299)
2025-06-18T00:00:03.318Z INFO [OpensearchProcessImpl] at org.apache.logging.log4j.core.util.FileUtils.defineFilePosixAttributeView(FileUtils.java:181)
2025-06-18T00:00:03.318Z INFO [OpensearchProcessImpl] at org.apache.logging.log4j.core.appender.FileManager.defineAttributeView(FileManager.java:216)
2025-06-18T00:00:03.318Z INFO [OpensearchProcessImpl] at org.apache.logging.log4j.core.appender.FileManager.createOutputStream(FileManager.java:203)
2025-06-18T00:00:03.318Z INFO [OpensearchProcessImpl] at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Is the datanode starting correctly and running fine or is there anything problematic on data node side? It is up and running and below is the status of the datanode and graylog:
● graylog-datanode.service - Graylog data node
Loaded: loaded (/usr/lib/systemd/system/graylog-datanode.service; enabled; preset: enabled)
Active: active (running) since Tue 2025-06-17 09:34:40 UTC; 16h ago
Docs: http://docs.graylog.org/
Main PID: 2026454 (java)
Tasks: 188 (limit: 38488)
Memory: 16.8G (peak: 18.9G)
CPU: 15h 10min 10.458s
● graylog-server.service - Graylog server
Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; preset: enabled)
Active: active (running) since Tue 2025-06-17 10:21:58 UTC; 15h ago
Docs: http://docs.graylog.org/
Main PID: 2054799 (graylog-server)
Tasks: 285 (limit: 38488)
Memory: 13.5G (peak: 16.2G)
CPU: 1d 9h 53min 8.347s
Are there any other exceptions in the server log? Apart from the complains about the Journal filling up 95% I don't see any other exceptions.
OK, thank you! So far I haven’t seen anything suspicious. Let’s talk about the certificate itself. Is it selfsigned? Is the chain correct and complete? Does your browser trust it by default?
Have you added the cert or CA to the JVM truststore?
Is there any error in browser console, when you try to load search results?
Hello @Tdvorak , Its not self signed but its issues by “COMODO CA Limited” and my browser trust it by default.
Common Name (CN)
*.xyz.xyz.com
Organization (O)
XYZ
Organizational Unit (OU)
Common Name (CN)
COMODO RSA Organization Validation Secure Server CA
Organization (O)
COMODO CA Limited
Organizational Unit (OU)
Issued On
Tuesday, August 6, 2024 at 5:30:00 AM
Expires On
Thursday, August 7, 2025 at 5:29:59 AM
I have not added cert to JVM truststore, If I have to then please let know how can I achieve it.
I don’t see any error in the browser console when I try to load search results.
Generally, an up-to-date cert is probably issued by a CA that’s included in the JVM truststore. But it may be that you are using an outdated JVM distribution, where the CA cert is already expired or not present. In that situation, it would make sense to add your cert (or better, the CA public cert) to the truststore.
Hello @Tdvorak I did follow the steps mentioned in the link and i see the new error in server.log
2025-06-19T04:54:10.336Z ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:772) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:584) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:298) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:381) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:358) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:270) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:55) [graylog.jar:?]
Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
Caused by: java.security.GeneralSecurityException: org.bouncycastle.pkcs.PKCSIOException: malformed data: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:88) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:379) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:196) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:162) ~[graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:64) ~[graylog.jar:?]
at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[graylog.jar:?]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.bouncycastle.pkcs.PKCSIOException: malformed data: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.parseBytes(Unknown Source) ~[graylog.jar:?]
at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:66) ~[graylog.jar:?]
at org.graylog2.shared.security.tls.PemKeyStore.doBuildKeyStore(PemKeyStore.java:99) ~[graylog.jar:?]
at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:85) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:379) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:196) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:162) ~[graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:64) ~[graylog.jar:?]
at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[graylog.jar:?]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.lang.IllegalArgumentException: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
at org.bouncycastle.asn1.ASN1Sequence.getInstance(Unknown Source) ~[graylog.jar:?]
at org.bouncycastle.asn1.x509.AlgorithmIdentifier.getInstance(Unknown Source) ~[graylog.jar:?]
at org.bouncycastle.asn1.pkcs.EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
at org.bouncycastle.asn1.pkcs.EncryptedPrivateKeyInfo.getInstance(Unknown Source) ~[graylog.jar:?]
at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.parseBytes(Unknown Source) ~[graylog.jar:?]
at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:66) ~[graylog.jar:?]
at org.graylog2.shared.security.tls.PemKeyStore.doBuildKeyStore(PemKeyStore.java:99) ~[graylog.jar:?]
at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:85) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:379) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:196) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:162) ~[graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:64) ~[graylog.jar:?]
at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[graylog.jar:?]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
2025-06-19T04:54:10.340Z INFO [Server] SIGNAL received. Shutting down.
2025-06-19T04:54:10.344Z INFO [GracefulShutdown] Graceful shutdown initiated.
2025-06-19T04:54:10.344Z INFO [GracefulShutdown] Node status: [Override lb:DEAD [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2025-06-19T04:54:13.346Z INFO [GracefulShutdown] Goodbye.