Graylog Pod Failing to connect to Opensearch due to TLS error

1. Describe your incident:

I am setting up Graylog and Opensearch using Helm on a kubernetes cluster. Opensearch is running with TLS configured. I have generated a JAVA jks which i have imported into the graylog pod so as to established a trusted source, but i am still getting error “java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty”

2. Describe your environment:

  • OS Information:
    Nodes OS: Centos 8
    Platform Kubernetes v1.24.8
    Helm version v3.12.2

  • Package Version:
    Graylog 5.0.3

  • Service logs, configurations, and environment variables:

Graylog Pod Logs:

javax.net.ssl|ERROR|10|main|2024-01-23 05:11:01.531 UTC|null:-1|Fatal (INTERNAL_ERROR): Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty (
“throwable” : {
java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
at java.base/sun.security.validator.PKIXValidator.(Unknown Source)
at java.base/sun.security.validator.Validator.getInstance(Unknown Source)
at java.base/sun.security.ssl.X509TrustManagerImpl.getValidator(Unknown Source)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(Unknown Source)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(Unknown Source)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(Unknown Source)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:336)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at org.graylog2.storage.versionprobe.VersionProbe.lambda$addAuthenticationIfPresent$3(VersionProbe.java:154)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
at okhttp3.RealCall.execute(RealCall.java:81)
at retrofit2.OkHttpCall.execute(OkHttpCall.java:204)
at org.graylog2.storage.versionprobe.VersionProbe.rootResponse(VersionProbe.java:174)
at org.graylog2.storage.versionprobe.VersionProbe.probeSingleHost(VersionProbe.java:137)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.findFirst(Unknown Source)
at org.graylog2.storage.versionprobe.VersionProbe.probeAllHosts(VersionProbe.java:107)
at org.graylog2.storage.versionprobe.VersionProbe.lambda$probe$1(VersionProbe.java:95)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at org.graylog2.storage.versionprobe.VersionProbe.probe(VersionProbe.java:95)
at org.graylog2.bootstrap.preflight.SearchDbPreflightCheck.runCheck(SearchDbPreflightCheck.java:50)
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
at org.graylog2.bootstrap.preflight.PreflightCheckService.runChecks(PreflightCheckService.java:51)
at org.graylog2.bootstrap.ServerBootstrap.runPreFlightChecks(ServerBootstrap.java:153)
at org.graylog2.bootstrap.ServerBootstrap.beforeInjectorCreation(ServerBootstrap.java:138)
at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:304)
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:260)
at org.graylog2.bootstrap.Main.main(Main.java:45)
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
at java.base/java.security.cert.PKIXParameters.setTrustAnchors(Unknown Source)
at java.base/java.security.cert.PKIXParameters.(Unknown Source)
at java.base/java.security.cert.PKIXBuilderParameters.(Unknown Source)

Graylog Values YAML:

3. What steps have you already taken to try and solve the problem?
I have tried to run an init Container so as to change the ownership of the trustore file to use graylog user as below but cant make a breakthrough. When i ssh into pod he file is still owned by root user instead of graylog user
init:
## Init Container image
##
#image:
# repository: “alpine”
# pullPolicy: “IfNotPresent”
#command:
#- chown
#- -R
#- graylog:graylog
#- /usr/share/graylog/os-cacerts.jks

4. How can the community help?

Kindly assist on how i can solve these issues, how can I successfuly connect Graylog Pod to Openserarch using helm charts

Default values for Graylog.

This is a YAML-formatted file.

Declare variables to be passed into your templates.

rbac:

Specifies whether RBAC resources should be created

create: true

resources:
- pods
- secrets

serviceAccount:

Specifies whether a ServiceAccount should be created

create: true

The name of the ServiceAccount to use.

If not set and create is true, a name is generated using the fullname template

name:

Service Account annotations

annotations: {}

tags:

If true, this chart will install opensearch from requirement dependencies

install-opensearch: false

If true, this chart will install MongoDB replicaset from requirement dependencies

install-mongodb: true

Enable only if your current release was migrated from helm2 (using 2to3 plugin).

Because Kubernetes considers some StatefulSets fields/labels immutable,

this flag preserves the values rendered by helm2. This allows helm3

to upgrade the current release without a complete purge/reinstall.

helm2Compatibility: true

Specify image pull secrets used in the deployment

imagePullSecrets:

imagePullSecrets:

- name: some-registry

- name: another-registry

graylog:

Graylog image version

Important note: Official Graylog Docker image may replace the existing Docker image tags and cause some corrupt when starting the pod.

Make sure you strict with the x version of Graylog where x is ${version}-${x}

image:
repository: “graylog/graylog”
tag: “5.0.3”
pullPolicy: “IfNotPresent”

Graylog default Java option

javaOpts: “-Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:-OmitStackTraceInFastThrow -XX:+UseG1GC -server”

javaOpts: “-Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:-OmitStackTraceInFastThrow -XX:+UseG1GC -Djavax.net.debug=all -Djavax.net.ssl.trustStore=/usr/share/graylog/os-cacerts.jks -server”
#javaOpts: “-Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:-OmitStackTraceInFastThrow -XX:+UseG1GC -server”

Number of Graylog instance

replicas: 1

Additional environment variables to be added to Graylog pods

env:
#- name: SKIP_PREFLIGHT_CHECK
#value: true

Additional labels to be added to Graylog pods

customLabels: {}

Additional environment variables in raw yaml format

- name: POD_IP

valueFrom:

fieldRef:

fieldPath: status.podIP

- name: SERVICE_8000_NAME

value: servicename

envRaw: {}

Pod affinity

affinity: {}

Node tolerations for node-exporter scheduling to nodes with taints

tolerations:

# - key: “key”
# operator: “Equal|Exists”
# value: “value”
# effect: “NoSchedule|PreferNoSchedule|NoExecute(1.6 only)”

Node labels for node-exporter pod assignment

nodeSelector: {}

Annotations to be added to Graylog pods

podAnnotations: {}

If specified, indicates the pod’s priority.

priorityClassName:

Set security context for defining privilege and accessing control settings entire Pod

podSecurityContext:
{}
# runAsUser: 10001
# fsGroup: 65534

Set security context for defining privilege and accessing control settings for Graylog container

securityContext:
privileged: false
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000

persistence:
## If true, Graylog will create/use a Persistent Volume Claim
## If false, use emptyDir
##
enabled: true
## Graylog data Persistent Volume access modes
## Must match those of existing PV or dynamic provisioner
##
##
accessMode: ReadWriteOnce
## Graylog data Persistent Volume size
##
size: “20Gi”
## Graylog data Persistent Volume Storage Class
## If defined, storageClassName:
## If set to “-”, storageClassName: “”, which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
# storageClass: “ssd”

Additional plugins you need to install on Graylog.

plugins:
## If true, set proxy server to retrieve the plugins
##
proxy:
enabled: false
host:

locations: []

Sidecar containers

sidecarContainers:

Additional init containers

extraInitContainers:
#- name: volume-permissions
#image: busybox
#command: [“sh”, “-c”, “chown -R 1100:1100 /usr/share/graylog/os-cacerts.jks”]
#volumeMounts:
#- name: my-config-volume
# mountPath: /usr/share/graylog/

Additional volume mounts

extraVolumeMounts:
- name: my-config-volume
mountPath: /usr/share/graylog/os-cacerts.jks
subPath: os-cacerts.jks
readOnly: false

Additional volumes

extraVolumes:
- name: my-config-volume
configMap:
name: java-ca

A service for Graylog web interface

service:
type: NodePort
port: 9000
## Add additional ports for the service/statefulset to expose
##
ports:

headless:
  ## Add suffix to headless service name
  ##
  suffix: ""

master:
  ## Enable Graylog master service
  ##
  enabled: true
  ## Graylog master service Ingress annotations
  ##
  annotations: {}
  ## Graylog master service port.
  ##
  port: 9000
  ## Graylog master service type.
  ##
  type: ClusterIP

startupProbe:
periodSeconds: 60
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 5

livenessProbe:
initialDelaySeconds: 0
periodSeconds: 30
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 5

readinessProbe:
initialDelaySeconds: 0
periodSeconds: 10
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 5

Additional input ports for receiving logs from servers

Note: Name must be in IANA_SVC_NAME (at most 15 characters, matching regex a-z0-9* and it must contains at least one letter [a-z], hyphens cannot be adjacent to other hyphens)

Note: Array must be sorted by port order

input:
{}
# tcp:
# service:
# name: your-tcp-service-name
# type: LoadBalancer
# loadBalancerIP:
# ports:
# - name: gelf
# port: 12222
# udp:
# service:
# name: your-udp-service-name
# type: ClusterIP
# ports:
# - name: syslog
# port: 12222

tls:
## If true, Graylog server will run with TLS enabled
##
enabled: false
## TLS certificate key file
##
keyFile: /etc/graylog/server/server.key
## TLS certificate file
##
certFile: /etc/graylog/server/server.cert

externalUri: “”

ingress:
## If true, Graylog server Ingress will be created
##
enabled: false
## Graylog server Ingress annotations
##
annotations: {}

## Graylog server Ingress labels
labels: {}
# labels:
#  traffic-type: public

## Specify the Ingress class name
ingressClassName: ""

## Graylog server Ingress hostnames with optional path
## Must be provided if Ingress is enabled
## Note: Graylog does not support two URL. You can specify only single URL
##
hosts: []
#  - graylog.yourdomain.com

## Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
extraPaths: []
# - path: /*
#   backend:
#     serviceName: ssl-redirect
#     servicePort: use-annotation

## Graylog server Ingress TLS configuration
## Secrets must be manually created in the namespace
##
tls: []
#   - secretName: graylog-server-tls
#     hosts:
#       - graylog.yourdomain.com

## For Kubernetes 1.18 or later. Use Prefix or Exact path types.
pathType: Prefix

## An URL path
path: /

istio:
## If true, Graylog server Istio Ingress Gateway will be created
##
enabled: false

## Change if you use custom ingressgateway controller
gatewaySelector:
  istio: ingressgateway

## Graylog server Istio Ingress Gateway hostnames with optional path
## Must be provided if Ingress is enabled
## Note: Graylog does not support two URL. You can specify only single URL
##
#host: []

## Please note that tls secret should reside in istio-system namespace
## leave blanc for disabling TLS
tlsSecretName: ""

Configure resource requests and limits

resources:
limits:
cpu: “1”
requests:
cpu: “300m”
memory: “1000Mi”

Set Graylog Java heapsize. If this value empty, chart will allocate heapsize using -XX:+UseCGroupMemoryLimitForHeap

heapSize: “16g”

RollingUpdate update strategy

updateStrategy: RollingUpdate

Control the maximum number of Pods that can be unavailable during an update

maxUnavailable: 10%

Graylog server pod termination grace period

terminationGracePeriodSeconds: 120

metrics:
## If true, prometheus annotations will be attached
##
enabled: false
serviceMonitor:
enabled: false
additionalLabels: {}
scrapeTimeout: 10s
interval: 10s

geoip:
## If true, Maxmind GeoLite2 will be installed to ${GRAYLOG_HOME}/geoip location
##
enabled: false
mmdbUri: “”

Graylog root user name

rootUsername: “admin”

Graylog root password

Defaults to a random 16-character alphanumeric string if not set

rootPassword: “”

Graylog root email

rootEmail: “”

Graylog root timezone

rootTimezone: “UTC”

Grayog existing root secret

existingRootSecret: “”

opensearch:
## Major version of the Elasticsearch version used.

# version: "7"

## List of Elasticsearch hosts Graylog should connect to.
## Need to be specified as a comma-separated list of valid URIs for the http ports of your elasticsearch nodes.
## If one or more of your elasticsearch hosts require authentication, include the credentials in each node URI that
## requires authentication.
##

hosts: https://admin:admin@opensearch-cluster-master-headless.graylog.svc.cluster.local:9200
# Allow elasticsearch hosts to be fetched from a k8s secret
# {{ graylog.fullname }}-es will be used as uriSecretName if left empty
uriSecretName: ""
uriSecretKey: ""
uriSSL: false

mongodb:
## MongoDB connection string
#
# uri: mongodb://user:pass@host1:27017,host2:27017,host3:27017/graylog?replicaSet=rs01
uri: “”
# Allow mongodb uri to be fetched from a k8s secret
# {{ graylog.fullname }}-headless will be used as uriSecretName if left empty
uriSecretName: “”
uriSecretKey: “”

## Increase this value according to the maximum connections your MongoDB server can handle from a single client
## if you encounter MongoDB connection problems.
##
maxConnections: 1000

transportEmail:
## If true, enable Email transport.
##
enabled: false
hostname: “”
port: 2587
useAuth: true
useTls: true
useSsl: false
authUsername: “”
authPassword: “”
subjectPrefix: “[graylog]”
fromEmail: “”

Additional graylog config which is defined on graylog.conf.

Graylog config is written in Java properites format. Make sure you write it correctly.

config: |

elasticsearch_connect_timeout = 10s

elasticsearch_socket_timeout = 60s

elasticsearch_idle_timeout = -1s

journal:
## Sometime Graylog journal continually grow up or corrupt and cause Graylog unable to start.
## You need to clean up all journal files in order to run the Graylog.
## Change graylog.journal.deleteBeforeStart to true to delete all journal files before start
## Note: All uncommitted logs will be permanently DELETED when this value is true
##
deleteBeforeStart: false

## Maximum size of the graylog journal.
##
maxSize: 5gb

init:
## Init Container image
##
#image:
# repository: “alpine”
# pullPolicy: “IfNotPresent”
#command:
#- chown
#- -R
#- graylog:graylog
#- /usr/share/graylog/os-cacerts.jks

##
# kubectlVersion: "v1.20"

# Additional environment variables to be added to Graylog initContainer
env: {}

# Configure resource requests and limits for the Graylog StatefulSet initContainer
resources: {}

Additional server files will be deployed to /etc/graylog/server

For example, you can put server certificates or authorized clients certificates here

serverFiles:

 server.key: |
   -----BEGIN PRIVATE KEY-----
   xxxx
   -----END PRIVATE KEY-----
 server.cert: |
   -----BEGIN CERTIFICATE-----
  xxx
   -----END CERTIFICATE-----

Configure whether Graylog pods should log in JSON (one event per line)

logInJson: false

Specify a Bash script to run as Kubernetes Job (running on Alpine with curl and bash packages already installed).

Useful for calling the API to pre-configure some aspect of Graylog, as in the example.

provisioner:
enabled: false
annotations: {}
useGraylogServiceAccount: false
# script: |
# json=‘{
# “username_header”: “X-Auth-Request-User”,
# “fullname_header”: “X-Auth-Request-User”,
# “email_header”: “X-Auth-Request-Email”,
# “default_group”: “Admin”,
# “auto_create_user”: true,
# “require_trusted_proxies”: true,
# “trusted_proxies”: “0.0.0.0/0”,
# “default_email_domain”: “mydomain.com”,
# “sync_roles”: false,
# “roles_header”: “Roles”
# }’
# curl -v -u “admin:$GRAYLOG_PASSWORD_SECRET” -X PUT --header ‘Content-Type: application/json’ --header ‘X-Requested-By: localhost’ --data-binary “${json}”

## Additional environment variables to be added to Graylog provisioner job
##
env: {}

## Additional environment variables in raw yaml format
## - name: POD_IP
##   valueFrom:
##     fieldRef:
##       fieldPath: status.podIP
## - name: SERVICE_8000_NAME
##   value: servicename
envRaw: {}

secret:
## Secret annotations
##
annotations: {}

options:
## Using search result highlighting will result in slightly higher resource consumption of searches.
allowHighlighting: false

## Do you want to allow searches with leading wildcards?
## This can be extremely resource hungry and should only be enabled with care.
allowLeadingWildcardSearches: false

## Size of internal ring buffers.
## Must be a power of 2. (512, 1024, 2048, …)
ringSize: ""
inputBufferRingSize: ""

## The threshold of the garbage collection runs. If GC runs take longer than this
## threshold, a system notification will be generated to warn the administrator
## about possible problems with the system. Default is 1 second.
gc_warning_threshold: 1s

## Buffer processor settings.
## A good rule of thumb is to never go over the number of cores available
outputbufferProcessors: 3
inputbufferProcessors: 2
processbufferProcessors: 5

opensearch:
enabled: false
extraEnvs:
- name: plugins.security.ssl.http.enabled
value: “false”
- name: plugins.security.disabled
value: “true”

- name: OPENSEARCH_JAVA_OPTS

value: “-Xms512m -Xmx512m”

- “OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m”

- “discovery.seed_hosts=opensearch2,opensearch3”

- “cluster.initial_master_nodes=opensearch1,opensearch2,opensearch3”

- “bootstrap.memory_lock=true”

- “action.auto_create_index=false”

resources:

requests:

cpu: “100m”

memory: “512M”

limits:

cpu: “1000m”

memory: “512M”

volumeClaimTemplate:

resources:

requests:

storage: 30Gi # default 30Gi

Specify Mongodb settings. Ignore this section if you install MongoDB manually.

mongodb:
architecture: “replicaset”
useStatefulSet: true
replicaCount: 1
auth:
enabled: false

Here’s another post involving missing trustanchor which might be helpful:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.