Unable to start datanode or server service on standalone install - "IllegalArgumentException: Illegal base64 character 3f"

[calebm76 (Caleb)]

1. Describe your incident:
I’m unable to start the the datanode or server service on a clean install on a Rocky 8.7 machine. I’m using graylog-datanode/server 6.1.1. The error that seems to be cropping up in both server and datanode logs:

) [Guice/ErrorInjectingConstructor]: IllegalArgumentException: Illegal base64 character 3f
  at CustomCAX509TrustManager.<init>(CustomCAX509TrustManager.java:42)
  while locating CustomCAX509TrustManager
  at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:44)
  at TrustManagerAndSocketFactoryProvider.<init>(TrustManagerAndSocketFactoryProvider.java:37)
      \_ for 1st parameter
  at TrustManagerAndSocketFactoryProvider.class(TrustManagerAndSocketFactoryProvider.java:37)
  at OkHttpClientProvider.<init>(OkHttpClientProvider.java:75)
      \_ for 5th parameter
  at OkHttpClientProvider.class(OkHttpClientProvider.java:59)
  while locating OkHttpClientProvider
  at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:46)
  at VersionProbe.<init>(VersionProbe.java:74)
      \_ for 2nd parameter
  at ElasticsearchVersionProvider.<init>(ElasticsearchVersionProvider.java:53)
      \_ for 3rd parameter
  at ServerPreflightChecksModule.configure(ServerPreflightChecksModule.java:47)
  while locating ElasticsearchVersionProvider

2. Describe your environment:

• OS Information: Rocky 8.7, kernel 4.18.0-553.22.1.el8_10.x86_64

• Package Version: graylog-datanode-6.1.1-1.x86_64 & graylog-server-6.1.1-1.x86_64

• Service logs, configurations, and environment variables: (wouldn’t let me upload my log files)

3. What steps have you already taken to try and solve the problem?
I’ve tried removing graylog-datanode & graylog-server and installing again, multiple times. Tried installing java openjdk 17 (even though docs don’t say to) but this hasn’t helped.

4. How can the community help?
If anyone else has seen this error before let me know, although I have searched Google and the forums a lot and can’t see anything.

[Wine_Merchant]

Hey @calebm76,

Sounds like the keystore could be corrupted in some way but Datanode/Graylog don’t share that.

When reinstalling Graylod/Datanode did you drop the collections from Mongo related to Datanode certs certificate_exchange and data_node_certificates?

[matthias_gl (Matthias)]

Hi @calebm76 and welcome to the community,
do you see this error immediately when starting up Graylog? Are you using a MongoDB that has previously been used by Graylog? If so, have you used data node there?