Graylog wont start after Update

Graylog Central (peer support)
1

Hi dear community,

i’v tried to update our graylog v6.0.14 to 6.1 with no sucess.

The graylog.service exits with an NullPointerException at CustomCAX509TrustManager

com.google.inject.CreationException: Unable to create injector, see the following errors:

at CustomCAX509TrustManager.(CustomCAX509TrustManager.java:42)
while locating CustomCAX509TrustManager
at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:44)
while locating X509TrustManager

Caused by: NullPointerException
at java.base/Hashtable.put(Unknown Source)
at PKCS12KeyStoreSpi$IgnoresCaseHashtable.put(Unknown Source)
at PKCS12KeyStoreSpi.engineSetCertificateEntry(Unknown Source)
at AdaptingKeyStoreSpi.engineSetCertificateEntry(Unknown Source)
at java.base/KeyStore.setCertificateEntry(Unknown Source)
at CaTruststoreImpl.filterOutPrivateKey(CaTruststoreImpl.java:74)
at java.base/Optional.map(Unknown Source)
at CaTruststoreImpl.getTrustStore(CaTruststoreImpl.java:57)
at CustomCAX509TrustManager.refresh(CustomCAX509TrustManager.java:58)
at CustomCAX509TrustManager.(CustomCAX509TrustManager.java:45)
at CustomCAX509TrustManager$$FastClassByGuice$$1947d48.GUICE$TRAMPOLINE()
at CustomCAX509TrustManager$$FastClassByGuice$$1947d48.apply()
at DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
at ConstructorInjector.provision(ConstructorInjector.java:114)
at ConstructorInjector.construct(ConstructorInjector.java:91)
at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
at FactoryProxy.get(FactoryProxy.java:60)
at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at SingletonScope$1.get(SingletonScope.java:169)
at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
at InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
at InternalInjectorCreator.build(InternalInjectorCreator.java:113)
at Guice.createInjector(Guice.java:87)
at Guice.createInjector(Guice.java:69)
at Guice.createInjector(Guice.java:59)
at ServerBootstrap.getPreflightInjector(ServerBootstrap.java:287)
at ServerBootstrap.runPreflightWeb(ServerBootstrap.java:193)
at ServerBootstrap.runPreFlightChecks(ServerBootstrap.java:179)
at ServerBootstrap.beforeInjectorCreation(ServerBootstrap.java:152)
at CmdLineTool.doRun(CmdLineTool.java:338)
at CmdLineTool.run(CmdLineTool.java:270)
at Main.main(Main.java:55)

at CustomCAX509TrustManager.(CustomCAX509TrustManager.java:42)
while locating CustomCAX509TrustManager
at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:44)
at TrustManagerAndSocketFactoryProvider.(TrustManagerAndSocketFactoryProvider.java:37)
_ for 1st parameter
at TrustManagerAndSocketFactoryProvider.class(TrustManagerAndSocketFactoryProvider.java:37)
at OkHttpClientProvider.(OkHttpClientProvider.java:75)
_ for 5th parameter
at OkHttpClientProvider.class(OkHttpClientProvider.java:59)
while locating OkHttpClientProvider
at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:46)
while locating okhttp3.OkHttpClient

Caused by: NullPointerException
at java.base/Hashtable.put(Unknown Source)
at PKCS12KeyStoreSpi$IgnoresCaseHashtable.put(Unknown Source)
at PKCS12KeyStoreSpi.engineSetCertificateEntry(Unknown Source)
at AdaptingKeyStoreSpi.engineSetCertificateEntry(Unknown Source)
at java.base/KeyStore.setCertificateEntry(Unknown Source)
at CaTruststoreImpl.filterOutPrivateKey(CaTruststoreImpl.java:74)
at java.base/Optional.map(Unknown Source)
at CaTruststoreImpl.getTrustStore(CaTruststoreImpl.java:57)
at CustomCAX509TrustManager.refresh(CustomCAX509TrustManager.java:58)
at CustomCAX509TrustManager.(CustomCAX509TrustManager.java:45)
at CustomCAX509TrustManager$$FastClassByGuice$$1947d48.GUICE$TRAMPOLINE()
at CustomCAX509TrustManager$$FastClassByGuice$$1947d48.apply()
at DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
at ConstructorInjector.provision(ConstructorInjector.java:114)
at ConstructorInjector.construct(ConstructorInjector.java:91)
at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
at FactoryProxy.get(FactoryProxy.java:60)
at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at SingletonScope$1.get(SingletonScope.java:169)
at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at SingleParameterInjector.inject(SingleParameterInjector.java:40)
at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
at ConstructorInjector.provision(ConstructorInjector.java:113)
at ConstructorInjector.construct(ConstructorInjector.java:91)
at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at SingletonScope$1.get(SingletonScope.java:169)
at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at SingleParameterInjector.inject(SingleParameterInjector.java:40)
at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
at ConstructorInjector.provision(ConstructorInjector.java:113)
at ConstructorInjector.construct(ConstructorInjector.java:91)
at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at SingletonScope$1.get(SingletonScope.java:169)
at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at BoundProviderFactory.get(BoundProviderFactory.java:58)
at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at SingletonScope$1.get(SingletonScope.java:169)
at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
at InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
at InternalInjectorCreator.build(InternalInjectorCreator.java:113)
at Guice.createInjector(Guice.java:87)
at Guice.createInjector(Guice.java:69)
at Guice.createInjector(Guice.java:59)
at ServerBootstrap.getPreflightInjector(ServerBootstrap.java:287)
at ServerBootstrap.runPreflightWeb(ServerBootstrap.java:193)
at ServerBootstrap.runPreFlightChecks(ServerBootstrap.java:179)
at ServerBootstrap.beforeInjectorCreation(ServerBootstrap.java:152)
at CmdLineTool.doRun(CmdLineTool.java:338)
at CmdLineTool.run(CmdLineTool.java:270)
at Main.main(Main.java:55)

at CustomCAX509TrustManager.(CustomCAX509TrustManager.java:42)
while locating CustomCAX509TrustManager
at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:44)
at TrustManagerAndSocketFactoryProvider.(TrustManagerAndSocketFactoryProvider.java:37)
_ for 1st parameter
at TrustManagerAndSocketFactoryProvider.class(TrustManagerAndSocketFactoryProvider.java:37)
at OkHttpClientProvider.(OkHttpClientProvider.java:75)
_ for 5th parameter
at OkHttpClientProvider.class(OkHttpClientProvider.java:59)
while locating OkHttpClientProvider
at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:46)
at VersionProbe.(VersionProbe.java:74)
_ for 2nd parameter
at ElasticsearchVersionProvider.(ElasticsearchVersionProvider.java:53)
_ for 3rd parameter
at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:47)
while locating ElasticsearchVersionProvider

Caused by: NullPointerException
at java.base/Hashtable.put(Unknown Source)
at PKCS12KeyStoreSpi$IgnoresCaseHashtable.put(Unknown Source)
at PKCS12KeyStoreSpi.engineSetCertificateEntry(Unknown Source)
at AdaptingKeyStoreSpi.engineSetCertificateEntry(Unknown Source)
at java.base/KeyStore.setCertificateEntry(Unknown Source)
at CaTruststoreImpl.filterOutPrivateKey(CaTruststoreImpl.java:74)
at java.base/Optional.map(Unknown Source)
at CaTruststoreImpl.getTrustStore(CaTruststoreImpl.java:57)
at CustomCAX509TrustManager.refresh(CustomCAX509TrustManager.java:58)
at CustomCAX509TrustManager.(CustomCAX509TrustManager.java:45)
at CustomCAX509TrustManager$$FastClassByGuice$$1947d48.GUICE$TRAMPOLINE()
at CustomCAX509TrustManager$$FastClassByGuice$$1947d48.apply()
at DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
at ConstructorInjector.provision(ConstructorInjector.java:114)
at ConstructorInjector.construct(ConstructorInjector.java:91)
at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
at FactoryProxy.get(FactoryProxy.java:60)
at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at SingletonScope$1.get(SingletonScope.java:169)
at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at SingleParameterInjector.inject(SingleParameterInjector.java:40)
at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
at ConstructorInjector.provision(ConstructorInjector.java:113)
at ConstructorInjector.construct(ConstructorInjector.java:91)
at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at SingletonScope$1.get(SingletonScope.java:169)
at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at SingleParameterInjector.inject(SingleParameterInjector.java:40)
at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
at ConstructorInjector.provision(ConstructorInjector.java:113)
at ConstructorInjector.construct(ConstructorInjector.java:91)
at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at SingletonScope$1.get(SingletonScope.java:169)
at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at BoundProviderFactory.get(BoundProviderFactory.java:58)
at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at SingletonScope$1.get(SingletonScope.java:169)
at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at SingleParameterInjector.inject(SingleParameterInjector.java:40)
at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
at ConstructorInjector.provision(ConstructorInjector.java:113)
at ConstructorInjector.construct(ConstructorInjector.java:91)
at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
at SingleParameterInjector.inject(SingleParameterInjector.java:40)
at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
at ConstructorInjector.provision(ConstructorInjector.java:113)
at ConstructorInjector.construct(ConstructorInjector.java:91)
at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at SingletonScope$1.get(SingletonScope.java:169)
at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
at InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
at InternalInjectorCreator.build(InternalInjectorCreator.java:113)
at Guice.createInjector(Guice.java:87)
at Guice.createInjector(Guice.java:69)
at Guice.createInjector(Guice.java:59)
at ServerBootstrap.getPreflightInjector(ServerBootstrap.java:287)
at ServerBootstrap.runPreflightWeb(ServerBootstrap.java:193)
at ServerBootstrap.runPreFlightChecks(ServerBootstrap.java:179)
at ServerBootstrap.beforeInjectorCreation(ServerBootstrap.java:152)
at CmdLineTool.doRun(CmdLineTool.java:338)
at CmdLineTool.run(CmdLineTool.java:270)
at Main.main(Main.java:55)

3 errors

======================
Full classname legend:

AdaptingKeyStoreSpi: “org.bouncycastle.jcajce.provider.keystore.util.AdaptingKeyStoreSpi”
BoundProviderFactory: “com.google.inject.internal.BoundProviderFactory”
CaTruststoreImpl: “org.graylog.security.certutil.CaTruststoreImpl”
CmdLineTool: “org.graylog2.bootstrap.CmdLineTool”
ConstructorBindingImpl$Factory: “com.google.inject.internal.ConstructorBindingImpl$Factory”
ConstructorInjector: “com.google.inject.internal.ConstructorInjector”
CustomCAX509TrustManager: “org.graylog2.security.CustomCAX509TrustManager”
CustomCAX509TrustManager$$FastClassByGuice$$1947d48: “org.graylog2.security.CustomCAX509TrustManager$$FastClassByGuice$$1947d48”
DefaultConstructionProxyFactory$FastClassProxy: “com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy”
ElasticsearchVersionProvider: “org.graylog2.storage.providers.ElasticsearchVersionProvider”
FactoryProxy: “com.google.inject.internal.FactoryProxy”
Guice: “com.google.inject.Guice”
InternalFactoryToProviderAdapter: “com.google.inject.internal.InternalFactoryToProviderAdapter”
InternalInjectorCreator: “com.google.inject.internal.InternalInjectorCreator”
KeyStore: “java.security.KeyStore”
Main: “org.graylog2.bootstrap.Main”
OkHttpClientProvider: “org.graylog2.shared.bindings.providers.OkHttpClientProvider”
PKCS12KeyStoreSpi: “org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi”
PKCS12KeyStoreSpi$IgnoresCaseHashtable: “org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi$IgnoresCaseHashtable”
ProviderToInternalFactoryAdapter: “com.google.inject.internal.ProviderToInternalFactoryAdapter”
ServerBootstrap: “org.graylog2.bootstrap.ServerBootstrap”
ServerPreflightChecksModule: “org.graylog2.bootstrap.preflight.ServerPreflightChecksModule”
SingleParameterInjector: “com.google.inject.internal.SingleParameterInjector”
SingletonScope$1: “com.google.inject.internal.SingletonScope$1”
TrustManagerAndSocketFactoryProvider: “org.graylog2.security.TrustManagerAndSocketFactoryProvider”
VersionProbe: “org.graylog2.storage.versionprobe.VersionProbe”
X509TrustManager: “javax.net.ssl.X509TrustManager”

End of classname legend:

    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:589) ~[graylog.jar:?]
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:190) ~[graylog.jar:?]
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113) ~[graylog.jar:?]
    at com.google.inject.Guice.createInjector(Guice.java:87) ~[graylog.jar:?]
    at com.google.inject.Guice.createInjector(Guice.java:69) ~[graylog.jar:?]
    at com.google.inject.Guice.createInjector(Guice.java:59) ~[graylog.jar:?]
    at org.graylog2.bootstrap.ServerBootstrap.getPreflightInjector(ServerBootstrap.java:287) ~[graylog.jar:?]
    at org.graylog2.bootstrap.ServerBootstrap.runPreflightWeb(ServerBootstrap.java:193) ~[graylog.jar:?]
    at org.graylog2.bootstrap.ServerBootstrap.runPreFlightChecks(ServerBootstrap.java:179) ~[graylog.jar:?]
    at org.graylog2.bootstrap.ServerBootstrap.beforeInjectorCreation(ServerBootstrap.java:152) ~[graylog.jar:?]
    at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:338) ~[graylog.jar:?]
    at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:270) [graylog.jar:?]
    at org.graylog2.bootstrap.Main.main(Main.java:55) [graylog.jar:?]

2. Describe your environment:

The service runs on a single VM with Debian 12.12

output from apt list --installed | grep ‘mongo|opensearch|graylog’

graylog-6.1-repository/stable,now 1-1 all [installed]
graylog-server/stable,now 6.1.15-1 amd64 [installed]
mongodb-database-tools/bookworm/mongodb-org/7.0,now 100.13.0 amd64 [installed,automatic]
mongodb-mongosh/bookworm/mongodb-org/7.0,now 2.5.8 amd64 [installed,automatic]
mongodb-org-database-tools-extra/bookworm/mongodb-org/7.0,now 7.0.24 amd64 [installed,automatic]
mongodb-org-database/bookworm/mongodb-org/7.0,now 7.0.24 amd64 [installed,automatic]
mongodb-org-mongos/bookworm/mongodb-org/7.0,now 7.0.24 amd64 [installed,automatic]
mongodb-org-server/bookworm/mongodb-org/7.0,now 7.0.24 amd64 [installed,automatic]
mongodb-org-shell/bookworm/mongodb-org/7.0,now 7.0.24 amd64 [installed,automatic]
mongodb-org-tools/bookworm/mongodb-org/7.0,now 7.0.24 amd64 [installed,automatic]
mongodb-org/bookworm/mongodb-org/7.0,now 7.0.24 amd64 [installed]
opensearch/stable,now 2.15.0 amd64 [installed,upgradable to: 2.19.3]

3. What steps have you already taken to try and solve the problem?

I have verified all the certificates, renewed the graylog.jks, checked the permissions on those certificates, checked the jvm-settings.

4. How can the community help?

Every point in the right direction would be helpful.

If more information is needed, let me know!

Many thanks!

2

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.