Not able to upgrade beyond 6.3.11 - Couldn't deserialize value (encrypted_value and salt must be strings and cannot be missing)

Graylog Central (peer support)
1

1. Describe your incident:

When starting a processing node on 6.3.12 or later, the node logs errors that a “decrypted_value and salt must be strings and cannot be missing” and then restarts. The cluster runs on version 6.3.11 without issue.

I believe it’s related to this change, but we have no secrets stored for any inputs: Encrypt AWS and TCP input secrets (6.3) by danotorrey · Pull Request #25829 · Graylog2/graylog2-server · GitHub

2. Describe your environment:

  • OS Information: AlmaLinux 9.7

  • Package Version:

    • Graylog: 6.3.11
    • Graylog datanode: 6.3.14
    • MongoDB: 6.0.27

  • Service logs, configurations, and environment variables:

2026-07-14 11:51:20,801 ERROR: org.graylog2.filters.ExtractorFilter - Unable to load extractors for all inputs
java.lang.IllegalArgumentException: Couldn't deserialize value (encrypted_value and salt must be strings and cannot be missing)
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
        at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4644)
        at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:4575)
        at org.graylog2.inputs.InputServiceImpl.lambda$createFromDbObject$2(InputServiceImpl.java:527)
        at java.base/java.lang.Iterable.forEach(Unknown Source)
        at org.graylog2.inputs.InputServiceImpl.createFromDbObject(InputServiceImpl.java:526)
        at org.graylog2.inputs.InputServiceImpl.all(InputServiceImpl.java:108)
        at org.graylog2.filters.ExtractorFilter.loadAllExtractors(ExtractorFilter.java:148)
        at org.graylog2.filters.ExtractorFilter.lifecycleChanged(ExtractorFilter.java:142)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:84)
        at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:141)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:70)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:259)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Couldn't deserialize value (encrypted_value and salt must be strings and cannot be missing)
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
        at org.graylog2.security.encryption.EncryptedValueDeserializer.parseFromDatabase(EncryptedValueDeserializer.java:79)
        at org.graylog2.security.encryption.EncryptedValueDeserializer.deserialize(EncryptedValueDeserializer.java:51)
        at org.graylog2.security.encryption.EncryptedValueDeserializer.deserialize(EncryptedValueDeserializer.java:31)
        at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4639)
        ... 18 more
2026-07-14 11:51:20,801 ERROR: org.graylog2.filters.StaticFieldFilter - Unable to load static fields for all inputs
java.lang.IllegalArgumentException: Couldn't deserialize value (encrypted_value and salt must be strings and cannot be missing)
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
        at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4644)
        at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:4575)
        at org.graylog2.inputs.InputServiceImpl.lambda$createFromDbObject$2(InputServiceImpl.java:527)
        at java.base/java.lang.Iterable.forEach(Unknown Source)
        at org.graylog2.inputs.InputServiceImpl.createFromDbObject(InputServiceImpl.java:526)
        at org.graylog2.inputs.InputServiceImpl.all(InputServiceImpl.java:108)
        at org.graylog2.filters.StaticFieldFilter.loadAllStaticFields(StaticFieldFilter.java:116)
        at org.graylog2.filters.StaticFieldFilter.lifecycleChanged(StaticFieldFilter.java:110)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:84)
        at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:141)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:70)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:259)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Couldn't deserialize value (encrypted_value and salt must be strings and cannot be missing)
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
        at org.graylog2.security.encryption.EncryptedValueDeserializer.parseFromDatabase(EncryptedValueDeserializer.java:79)
        at org.graylog2.security.encryption.EncryptedValueDeserializer.deserialize(EncryptedValueDeserializer.java:51)
        at org.graylog2.security.encryption.EncryptedValueDeserializer.deserialize(EncryptedValueDeserializer.java:31)
        at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4639)
        ... 18 more

3. What steps have you already taken to try and solve the problem?

Tested combinations of Graylog and datanode starting at 6.3.12. Datanodes are able to run later versions but Graylog itself cannot.

4. How can the community help?

How can I either create the encrypted_value and salt before upgrade or convince Graylog that my inputs don’t have secrets?

2

Hey @ddean,

Do you have any AWS Kinesis/Cloudwatch inputs currently running, if possible could you edit the input config, alter nothing but hit save. Then test again.

3

Hi @Wine_Merchant - thank you for the advice! We don’t have any AWS Kinesis or Cloudwatch inputs. I took your advice and tried editing the config and saving without changes on all our inputs, but I’m still not able to start the cluster on 6.3.12 or later.

4

Thanks for confirming @ddean, I wonder if this is an issue that will occur during a rolling upgrade where mixed versions exist in the cluster.

Nodes on 6.3.11 don’t encrypt values but the 6.3.12 nodes are expecting encrypted values, so for as long as the 6.3.11 nodes are present this will fail. I would recommend a backup of Mongo and attempt an upgrade that isn’t rolling, ensuring the leader node is upgraded first.