Unable to start Graylog after update from 6.0.9 to 6.1.x - "IllegalArgumentException: Illegal base64 character 3f"

Graylog Central (peer support)
1

1. Describe your incident:
Hello! After updating Graylog Server 6.0.9 to any version of 6.1.x on my Ubuntu, I cannot start graylog service again. I only updated Graylog, other components are “unharmed”.

2. Describe your environment:

  • OS Information: Ubuntu 22.04.5 LTS

  • Package Version:

  • graylog-server 6.1.4-2 amd64

  • elasticsearch 7.17.18 amd64 (I know it’s above the recommended version but it never caused any problem)

  • mongodb-org 7.0.16 amd64

  • Service logs, configurations, and environment variables:

I’m using HTTPS just for Graylog with cert signed by our Windows CA server.

2025-01-08T12:30:05.228+01:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {default properties file=[cloud_inputs=on, investigation_report_by_ai=on, data_tiering_cloud=off, composable_index_templates=off, preflight_web=on, data_node_migration=on, remote_reindex_migration=off, instant_archiving=off, configurable_value_units=on, new_report_creation=on, threat_coverage=on]}
2025-01-08T12:30:05.556+01:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 6.1.4+7528370 [org.graylog.aws.AWSPlugin]
2025-01-08T12:30:05.556+01:00 INFO  [CmdLineTool] Loaded plugin: Integrations 6.1.4+7528370 [org.graylog.integrations.IntegrationsPlugin]
2025-01-08T12:30:05.556+01:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 6.1.4+7528370 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2025-01-08T12:30:05.557+01:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 6.1.4+7528370 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2025-01-08T12:30:05.557+01:00 INFO  [CmdLineTool] Loaded plugin: OpenSearch 2 Support 6.1.4+7528370 [org.graylog.storage.opensearch2.OpenSearch2Plugin]
2025-01-08T12:30:05.568+01:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Dgraylog2.installation_source=deb
2025-01-08T12:30:05.677+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.15.0-130-generic"}, "platform": "Java/Eclipse Adoptium/17.0.13+11"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3c3a0032, com.mongodb.Jep395RecordCodecProvider@7ceb4478, com.mongodb.KotlinCodecProvider@7fdab70c]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-01-08T12:30:05.679+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.15.0-130-generic"}, "platform": "Java/Eclipse Adoptium/17.0.13+11"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3c3a0032, com.mongodb.Jep395RecordCodecProvider@7ceb4478, com.mongodb.KotlinCodecProvider@7fdab70c]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-01-08T12:30:05.692+01:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=12318647, minRoundTripTimeNanos=0}
2025-01-08T12:30:05.737+01:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 7.0.16
2025-01-08T12:30:05.991+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.15.0-130-generic"}, "platform": "Java/Eclipse Adoptium/17.0.13+11"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3c3a0032, com.mongodb.Jep395RecordCodecProvider@7ceb4478, com.mongodb.KotlinCodecProvider@7fdab70c]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-01-08T12:30:05.992+01:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=917510, minRoundTripTimeNanos=0}
2025-01-08T12:30:05.993+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.15.0-130-generic"}, "platform": "Java/Eclipse Adoptium/17.0.13+11"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3c3a0032, com.mongodb.Jep395RecordCodecProvider@7ceb4478, com.mongodb.KotlinCodecProvider@7fdab70c]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-01-08T12:30:06.092+01:00 INFO  [IndexerDiscoveryProvider] No indexer hosts configured, using fallback http://127.0.0.1:9200
2025-01-08T12:30:06.274+01:00 ERROR [CmdLineTool] Startup error:
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) [Guice/ErrorInjectingConstructor]: IllegalArgumentException: Illegal base64 character 3f
  at CustomCAX509TrustManager.<init>(CustomCAX509TrustManager.java:42)
  while locating CustomCAX509TrustManager
  at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:44)
  while locating X509TrustManager

Learn more:
  https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR
Caused by: IllegalArgumentException: Illegal base64 character 3f
        at java.base/Base64$Decoder.decode0(Unknown Source)
        at java.base/Base64$Decoder.decode(Unknown Source)
        at java.base/Base64$Decoder.decode(Unknown Source)
        at java.base/Optional.map(Unknown Source)
        at CaPersistenceService.readFromDatabase(CaPersistenceService.java:205)
        at CaPersistenceService.loadKeyStore(CaPersistenceService.java:187)
        at CaTruststoreImpl.getTrustStore(CaTruststoreImpl.java:55)
        at CustomCAX509TrustManager.refresh(CustomCAX509TrustManager.java:58)
        at CustomCAX509TrustManager.<init>(CustomCAX509TrustManager.java:45)
        at CustomCAX509TrustManager$$FastClassByGuice$$19b96a4.GUICE$TRAMPOLINE(<generated>)
        at CustomCAX509TrustManager$$FastClassByGuice$$19b96a4.apply(<generated>)
        at DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
        at ConstructorInjector.provision(ConstructorInjector.java:114)
        at ConstructorInjector.construct(ConstructorInjector.java:91)
        at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
        at FactoryProxy.get(FactoryProxy.java:60)
        at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at SingletonScope$1.get(SingletonScope.java:169)
        at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
        at InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
        at InternalInjectorCreator.build(InternalInjectorCreator.java:113)
        at Guice.createInjector(Guice.java:87)
        at Guice.createInjector(Guice.java:69)
        at Guice.createInjector(Guice.java:59)
        at ServerBootstrap.getPreflightInjector(ServerBootstrap.java:287)
        at ServerBootstrap.runPreflightWeb(ServerBootstrap.java:193)
        at ServerBootstrap.runPreFlightChecks(ServerBootstrap.java:179)
        at ServerBootstrap.beforeInjectorCreation(ServerBootstrap.java:152)
        at CmdLineTool.doRun(CmdLineTool.java:338)
        at CmdLineTool.run(CmdLineTool.java:270)
        at Main.main(Main.java:55)

2) [Guice/ErrorInjectingConstructor]: IllegalArgumentException: Illegal base64 character 3f
  at CustomCAX509TrustManager.<init>(CustomCAX509TrustManager.java:42)
  while locating CustomCAX509TrustManager
  at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:44)
  at TrustManagerAndSocketFactoryProvider.<init>(TrustManagerAndSocketFactoryProvider.java:37)
      \_ for 1st parameter
  at TrustManagerAndSocketFactoryProvider.class(TrustManagerAndSocketFactoryProvider.java:37)
  at OkHttpClientProvider.<init>(OkHttpClientProvider.java:75)
      \_ for 5th parameter
  at OkHttpClientProvider.class(OkHttpClientProvider.java:59)
  while locating OkHttpClientProvider
  at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:46)
  while locating okhttp3.OkHttpClient

Learn more:
  https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR
Caused by: IllegalArgumentException: Illegal base64 character 3f
        at java.base/Base64$Decoder.decode0(Unknown Source)
        at java.base/Base64$Decoder.decode(Unknown Source)
        at java.base/Base64$Decoder.decode(Unknown Source)
        at java.base/Optional.map(Unknown Source)
        at CaPersistenceService.readFromDatabase(CaPersistenceService.java:205)
        at CaPersistenceService.loadKeyStore(CaPersistenceService.java:187)
        at CaTruststoreImpl.getTrustStore(CaTruststoreImpl.java:55)
        at CustomCAX509TrustManager.refresh(CustomCAX509TrustManager.java:58)
        at CustomCAX509TrustManager.<init>(CustomCAX509TrustManager.java:45)
        at CustomCAX509TrustManager$$FastClassByGuice$$19b96a4.GUICE$TRAMPOLINE(<generated>)
        at CustomCAX509TrustManager$$FastClassByGuice$$19b96a4.apply(<generated>)
        at DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
        at ConstructorInjector.provision(ConstructorInjector.java:114)
        at ConstructorInjector.construct(ConstructorInjector.java:91)
        at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
        at FactoryProxy.get(FactoryProxy.java:60)
        at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at SingletonScope$1.get(SingletonScope.java:169)
        at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at SingleParameterInjector.inject(SingleParameterInjector.java:40)
        at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
        at ConstructorInjector.provision(ConstructorInjector.java:113)
        at ConstructorInjector.construct(ConstructorInjector.java:91)
        at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
        at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at SingletonScope$1.get(SingletonScope.java:169)
        at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at SingleParameterInjector.inject(SingleParameterInjector.java:40)
        at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
        at ConstructorInjector.provision(ConstructorInjector.java:113)
        at ConstructorInjector.construct(ConstructorInjector.java:91)
        at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
        at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at SingletonScope$1.get(SingletonScope.java:169)
        at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at BoundProviderFactory.get(BoundProviderFactory.java:58)
        at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at SingletonScope$1.get(SingletonScope.java:169)
        at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
        at InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
        at InternalInjectorCreator.build(InternalInjectorCreator.java:113)
        at Guice.createInjector(Guice.java:87)
        at Guice.createInjector(Guice.java:69)
        at Guice.createInjector(Guice.java:59)
        at ServerBootstrap.getPreflightInjector(ServerBootstrap.java:287)
        at ServerBootstrap.runPreflightWeb(ServerBootstrap.java:193)
        at ServerBootstrap.runPreFlightChecks(ServerBootstrap.java:179)
        at ServerBootstrap.beforeInjectorCreation(ServerBootstrap.java:152)
        at CmdLineTool.doRun(CmdLineTool.java:338)
        at CmdLineTool.run(CmdLineTool.java:270)
        at Main.main(Main.java:55)

3) [Guice/ErrorInjectingConstructor]: IllegalArgumentException: Illegal base64 character 3f
  at CustomCAX509TrustManager.<init>(CustomCAX509TrustManager.java:42)
  while locating CustomCAX509TrustManager
  at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:44)
  at TrustManagerAndSocketFactoryProvider.<init>(TrustManagerAndSocketFactoryProvider.java:37)
      \_ for 1st parameter
  at TrustManagerAndSocketFactoryProvider.class(TrustManagerAndSocketFactoryProvider.java:37)
  at OkHttpClientProvider.<init>(OkHttpClientProvider.java:75)
      \_ for 5th parameter
  at OkHttpClientProvider.class(OkHttpClientProvider.java:59)
  while locating OkHttpClientProvider
  at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:46)
  at VersionProbe.<init>(VersionProbe.java:74)
      \_ for 2nd parameter
  at ElasticsearchVersionProvider.<init>(ElasticsearchVersionProvider.java:53)
      \_ for 3rd parameter
  at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:47)
  while locating ElasticsearchVersionProvider

Learn more:
  https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR
Caused by: IllegalArgumentException: Illegal base64 character 3f
        at java.base/Base64$Decoder.decode0(Unknown Source)
        at java.base/Base64$Decoder.decode(Unknown Source)
        at java.base/Base64$Decoder.decode(Unknown Source)
        at java.base/Optional.map(Unknown Source)
        at CaPersistenceService.readFromDatabase(CaPersistenceService.java:205)
        at CaPersistenceService.loadKeyStore(CaPersistenceService.java:187)
        at CaTruststoreImpl.getTrustStore(CaTruststoreImpl.java:55)
        at CustomCAX509TrustManager.refresh(CustomCAX509TrustManager.java:58)
        at CustomCAX509TrustManager.<init>(CustomCAX509TrustManager.java:45)
        at CustomCAX509TrustManager$$FastClassByGuice$$19b96a4.GUICE$TRAMPOLINE(<generated>)
        at CustomCAX509TrustManager$$FastClassByGuice$$19b96a4.apply(<generated>)
        at DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
        at ConstructorInjector.provision(ConstructorInjector.java:114)
        at ConstructorInjector.construct(ConstructorInjector.java:91)
        at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
        at FactoryProxy.get(FactoryProxy.java:60)
        at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at SingletonScope$1.get(SingletonScope.java:169)
        at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at SingleParameterInjector.inject(SingleParameterInjector.java:40)
        at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
        at ConstructorInjector.provision(ConstructorInjector.java:113)
        at ConstructorInjector.construct(ConstructorInjector.java:91)
        at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
        at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at SingletonScope$1.get(SingletonScope.java:169)
        at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at SingleParameterInjector.inject(SingleParameterInjector.java:40)
        at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
        at ConstructorInjector.provision(ConstructorInjector.java:113)
        at ConstructorInjector.construct(ConstructorInjector.java:91)
        at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
        at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at SingletonScope$1.get(SingletonScope.java:169)
        at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at BoundProviderFactory.get(BoundProviderFactory.java:58)
        at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at SingletonScope$1.get(SingletonScope.java:169)
        at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at SingleParameterInjector.inject(SingleParameterInjector.java:40)
        at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
        at ConstructorInjector.provision(ConstructorInjector.java:113)
        at ConstructorInjector.construct(ConstructorInjector.java:91)
        at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
        at SingleParameterInjector.inject(SingleParameterInjector.java:40)
        at SingleParameterInjector.getAll(SingleParameterInjector.java:60)
        at ConstructorInjector.provision(ConstructorInjector.java:113)
        at ConstructorInjector.construct(ConstructorInjector.java:91)
        at ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
        at ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at SingletonScope$1.get(SingletonScope.java:169)
        at InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
        at InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
        at InternalInjectorCreator.build(InternalInjectorCreator.java:113)
        at Guice.createInjector(Guice.java:87)
        at Guice.createInjector(Guice.java:69)
        at Guice.createInjector(Guice.java:59)
        at ServerBootstrap.getPreflightInjector(ServerBootstrap.java:287)
        at ServerBootstrap.runPreflightWeb(ServerBootstrap.java:193)
        at ServerBootstrap.runPreFlightChecks(ServerBootstrap.java:179)
        at ServerBootstrap.beforeInjectorCreation(ServerBootstrap.java:152)
        at CmdLineTool.doRun(CmdLineTool.java:338)
        at CmdLineTool.run(CmdLineTool.java:270)
        at Main.main(Main.java:55)

3 errors

======================
Full classname legend:
======================
BoundProviderFactory:                                "com.google.inject.internal.BoundProviderFactory"
CaPersistenceService:                                "org.graylog.security.certutil.CaPersistenceService"
CaTruststoreImpl:                                    "org.graylog.security.certutil.CaTruststoreImpl"
CmdLineTool:                                         "org.graylog2.bootstrap.CmdLineTool"
ConstructorBindingImpl$Factory:                      "com.google.inject.internal.ConstructorBindingImpl$Factory"
ConstructorInjector:                                 "com.google.inject.internal.ConstructorInjector"
CustomCAX509TrustManager:                            "org.graylog2.security.CustomCAX509TrustManager"
CustomCAX509TrustManager$$FastClassByGuice$$19b96a4: "org.graylog2.security.CustomCAX509TrustManager$$FastClassByGuice$$19b96a4"
DefaultConstructionProxyFactory$FastClassProxy:      "com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy"
ElasticsearchVersionProvider:                        "org.graylog2.storage.providers.ElasticsearchVersionProvider"
FactoryProxy:                                        "com.google.inject.internal.FactoryProxy"
Guice:                                               "com.google.inject.Guice"
InternalFactoryToProviderAdapter:                    "com.google.inject.internal.InternalFactoryToProviderAdapter"
InternalInjectorCreator:                             "com.google.inject.internal.InternalInjectorCreator"
Main:                                                "org.graylog2.bootstrap.Main"
OkHttpClientProvider:                                "org.graylog2.shared.bindings.providers.OkHttpClientProvider"
ProviderToInternalFactoryAdapter:                    "com.google.inject.internal.ProviderToInternalFactoryAdapter"
ServerBootstrap:                                     "org.graylog2.bootstrap.ServerBootstrap"
ServerPreflightChecksModule:                         "org.graylog2.bootstrap.preflight.ServerPreflightChecksModule"
SingleParameterInjector:                             "com.google.inject.internal.SingleParameterInjector"
SingletonScope$1:                                    "com.google.inject.internal.SingletonScope$1"
TrustManagerAndSocketFactoryProvider:                "org.graylog2.security.TrustManagerAndSocketFactoryProvider"
VersionProbe:                                        "org.graylog2.storage.versionprobe.VersionProbe"
X509TrustManager:                                    "javax.net.ssl.X509TrustManager"
========================
End of classname legend:
========================

        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:589) ~[graylog.jar:?]
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:190) ~[graylog.jar:?]
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113) ~[graylog.jar:?]
        at com.google.inject.Guice.createInjector(Guice.java:87) ~[graylog.jar:?]
        at com.google.inject.Guice.createInjector(Guice.java:69) ~[graylog.jar:?]
        at com.google.inject.Guice.createInjector(Guice.java:59) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.getPreflightInjector(ServerBootstrap.java:287) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.runPreflightWeb(ServerBootstrap.java:193) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.runPreFlightChecks(ServerBootstrap.java:179) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.beforeInjectorCreation(ServerBootstrap.java:152) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:338) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:270) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:55) [graylog.jar:?]

3. What steps have you already taken to try and solve the problem?
Tried to install various 6.1.x versions, checked elastic,mongo and graylog logs but cannot find anything just the log pasted above. Searched on Google and found the same problem but wasn’t any solution (but that topic is closed) https://community.graylog.org/t/unable-to-start-datanode-or-server-service-on-standalone-install-illegalargumentexception-illegal-base64-character-3f/34031

4. How can the community help?
I’m not a pro with Graylog so I appreciate any tips where to start debugging. None of the previous updates caused a similar error. Thanks.

2

Looking at the failing code line CaPersistenceService.readFromDatabase, it is getting an invalid CA keystore string from the DB. So either the DB entry is wrong, or the decryption key password_secret is wrong.
Check the password_secret entry in your .conf file.

3

Thanks. It looks like there is the problem, BUT…

When I freshly installed and configured first time this Graylog last year I made a backup copy of the config file.
A bit later I changed the password_secret (and sha2 password also) and started to build and setup the working enviroment after.

Now If I set the the password_secret from the first backup conf file the service starts. But in this case I can only log in with the local admin account (no AD in this case), I can see dashboards, indices etc, but I cannot access the data stored in inputs.
When I set the password_secret back to the original (which has not changed and used in 6.0.9) I cannot start the graylog service again and I get the errors shown above.
It seems something is corrupted here…or something changed in this version and cannot read this hash…strange.

When you say “getting an invalid CA keystore string from the DB” you mean mongodb? Or it stores it somewhere else? I tried to find ca related things in mongodb but haven’t find - yet.
Thanks.

4

Update: I changed the password_secret on my working 6.0.9 to the one I found in my old backup conf file. After this I restarted the graylog service without any error and I could log in with local admin credentials. After a quick authentication reconfiguration in Graylog (I just had to type in the password again for AD service) i could use the AD login again. At this point I did the update to 6.1.5 and it’s working fine!
So it’s ok now but I really don’t understand whats going on.