PKIX path building failed unable to find valid certifcate pathto requested target in a Graylog running on POD

Description of your problem

i am trying to integrate NUCOOL oss system with Graylog using the webhok URL “https://172.18.72.210:4443/probe/webhook/NetOps” provided by nucool . i am using graylog HTTP alert notification feature to set the webhook URL configuration to send an alert to nucool whenever new alerts get triggered.

while integration i get PKIX path building failed: sun.security.provider.certpath.sunCertPathBuilderException:unable to find valid certificate path to requested target error.

my graylog is running on POD.and webhook URL provided by nucool oss team is using self signed certificate .

may i know how i can fix this error…is there any specific config needs to be done on graylog pod to overcome this issue please suggest.

Description of steps you’ve taken to attempt to solve the issue

i have imported self singed cert to keystore in master node but that didnt solve the issue, i a not sure how it can be done on graylog POD if it is required to do so.

Environmental information

using openshift environment to deploy the Graylog POD

Operating system information

Debian
Containers (OPENSHIFT)

Package versions

Graylog :3.3
MongoDB :mongo:3.6
logstash:7.0.0-alpha2
Elasticsearch /elasticsearch:6.8.7

```
 GRAYLOG YAML 
apiVersion: v1
kind: ConfigMap
metadata:
  name: graylog-conf
data:
  nginx.conf: |
    user nginx;
    worker_processes  3;
    error_log  /var/log/nginx/error.log;
    events {
      worker_connections  10240;
    }
    http {
      log_format  main
              'remote_addr:$remote_addr\t'
              'time_local:$time_local\t'
              'method:$request_method\t'
              'uri:$request_uri\t'
              'host:$host\t'
              'status:$status\t'
              'bytes_sent:$body_bytes_sent\t'
              'referer:$http_referer\t'
              'useragent:$http_user_agent\t'
              'forwardedfor:$http_x_forwarded_for\t'
              'request_time:$request_time';
      access_log        /var/log/nginx/access.log main;
      server {
          listen       80;
          server_name  _;
          location / {
              root   html;
              index  index.html index.htm;
          }
      }
      include /etc/nginx/virtualhost/virtualhost.conf;
    }
  virtualhost.conf: |
    #upstream app {
      #server localhost:8080;
      #keepalive 1024;
    #}
    server
      {
          listen 443 ssl ;
          #listen [::]:80 default_server ipv6only=on;
          ssl_certificate    /etc/ssl/nginx.crt; 
          ssl_certificate_key /etc/ssl/nginx.key;
          ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
          ssl_ciphers HIGH:!aNULL:!MD5;
          server_name {{ .Values.nodes.worker1 }};
          location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Server $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Graylog-Server-URL https://netops-graylog-netopsai.apps.ocp19.nfvdev.tlabs.ca/;
            proxy_pass       http://127.0.0.1:9000/;
          }
          location ^~/input {
            rewrite ^/input/(.*)$ /$1 break;
            proxy_set_header Host $http_host;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Server $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Graylog-Server-URL https://netops-graylog-netopsai.apps.ocp19.nfvdev.tlabs.ca/;
            proxy_pass       http://127.0.0.1:12201/;
          }
      }
  graylog.conf: |
    transportEmail:
    GRAYLOG_TRANSPORT_EMAIL_ENABLED: true
    GRAYLOG_TRANSPORT_EMAIL_HOSTNAME: "205.206.214.22"
    GRAYLOG_TRANSPORT_EMAIL_PORT: 25
    GRAYLOG_TRANSPORT_EMAIL_USE_AUTH: "false"
    GRAYLOG_TRANSPORT_EMAIL_USE_TLS: "false"
    GRAYLOG_TRANSPORT_EMAIL_USE_SSL: "false"
    GRAYLOG_TRANSPORT_EMAIL_AUTH_USERNAME: "false"
    GRAYLOG_TRANSPORT_EMAIL_AUTH_PASSWORD: "false"
    GRAYLOG_TRANSPORT_EMAIL_SUBJECT_PREFIX: "false"
    GRAYLOG_TRANSPORT_EMAIL_FROM_EMAIL: "noreply@telus.com"
  #  # Interpolate
  # sed 's/"/\\\"/g;s/.*/echo "&"/e' ${GRAYLOG_HOME}/config/graylog.conf > ${GRAYLOG_HOME}/graylog.conf.subst
   #echo "Graylog Home ${GRAYLOG_HOME}"
   #echo "JVM Options ${GRAYLOG_SERVER_JAVA_OPTS}"
   #"${JAVA_HOME}/bin/java" \
    # ${GRAYLOG_SERVER_JAVA_OPTS} \
    #-jar \
    #-Dlog4j.configurationFile=${GRAYLOG_HOME}/config/log4j2.xml \
    #-Djava.library.path=${GRAYLOG_HOME}/lib/sigar/ \
    #-Dgraylog2.installation_source=docker \
    #${GRAYLOG_HOME}/graylog.jar \
    #server \
  
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: graylog
spec:
  replicas: {{ .Values.replicaCount }}
  serviceName: graylog
  selector:
    matchLabels:
      app: graylog
  template:
    metadata:
      labels:
        app: graylog
    spec:
      imagePullSecrets:
        - name: {{ .Values.image.imagePullSecrets }}
      serviceAccount: netopsai
      serviceAccountName: netopsai
      containers:
      - name: graylog-nginx
        image: nginx
        ports:
        - containerPort: 9000
        volumeMounts:
        - mountPath: /etc/nginx
          readOnly: true
          name: graylog-conf
        - mountPath: /etc/ssl
          readOnly: true
          name: graylog-certs
      - name: graylog
        image: {{ .Values.graylog.image }}
        env:
        - name: GRAYLOG_PASSWORD_SECRET
          value: {{ .Values.graylog.pswdsecret }}
        - name: GRAYLOG_ROOT_PASSWORD_SHA2
          value: {{ .Values.graylog.pswdsha2 }}
        - name: GRAYLOG_HTTP_BIND_ADDRESS
          value: {{ .Values.graylog.bindaddress }}
        - name: GRAYLOG_ELASTICSEARCH_HOSTS
          value: http://elastic-search:9200
        - name: GRAYLOG_MONGODB_URI
          value: {{ .Values.graylog.mongodburi }}
        - name: GRAYLOG_TRANSPORT_EMAIL_ENABLED
          value: "true"
        - name: GRAYLOG_TRANSPORT_EMAIL_HOSTNAME
          value: "205.206.214.22"
        - name: GRAYLOG_TRANSPORT_EMAIL_PORT
          value: "25"
        - name: GRAYLOG_TRANSPORT_EMAIL_USE_AUTH
          value: "false"
        - name: GRAYLOG_TRANSPORT_EMAIL_USE_TLS
          value: "false"
        - name: GRAYLOG_TRANSPORT_EMAIL_USE_SSL
          value: "false"
        - name: GRAYLOG_TRANSPORT_EMAIL_AUTH_USERNAME
          value: ""
        - name: GRAYLOG_TRANSPORT_EMAIL_AUTH_PASSWORD
          value: ""
        - name: GRAYLOG_TRANSPORT_EMAIL_SUBJECT_PREFIX
          value: "graylog"
        - name: GRAYLOG_TRANSPORT_EMAIL_FROM_EMAIL
          value: "noreply@telus.com"
        ports:
        - containerPort: {{ .Values.graylog.containerport1 }}
        - containerPort: {{ .Values.graylog.containerport2 }}
      volumes:
      - name: graylog-conf
        configMap:
          name: graylog-conf
          items:
            - key: nginx.conf
              path: nginx.conf
            - key: virtualhost.conf
              path: virtualhost/virtualhost.conf
      - name: graylog-certs
#        hostPath: 
#          path: /etc/nginx/ssl-certs/
#          type: Directory
        secret:
          secretName: graylog-certs
#      nodeSelector:
#        name: worker1

---
apiVersion: v1
kind: Service
metadata:
  name: graylog
spec:
  type: {{ .Values.graylog.type }}
  ports:
  - name: "{{ .Values.graylog.port }}"
    port: {{  .Values.graylog.port }}
    targetPort: {{ .Values.graylog.port }}
    nodePort: {{ .Values.graylog.nodeport }}
  - name: "{{ .Values.graylog_http.port }}"
    port: {{ .Values.graylog_http.port }}
    targetPort: {{ .Values.graylog_http.port }}
    nodePort: {{ .Values.graylog_http.nodeport }}

  selector:
    app: graylog

Hello,
I might be able to help

Normally this occur when Graylog does not have access to the keystore or the certificates/correct certificates at not in the keystore.

Also check you Graylog’s config file in this section.

http_bind_address = 8.8.8.8:9000
http_publish_uri = https://graylog.domain.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = secret

As you can see Graylog has access to certs and the Keystore shown below. NOTE: don’t mind the 777 is was a mockup,

EDIT: I forgot in order for the JVM to pick up the new trust store, it has to be started with the JVM parameter -Djavax.net.ssl.trustStore=/path/to/cacerts.jks

Hi gsmith,
do i have change the graylog server config file exist under /etc/graylog/server/server.conf file even when i use helm charts to do the deployment of graylog…please note i am not using rpm based deployment for graylog.

login as: root
root@172.21.109.74’s password:
Last login: Fri Sep 17 04:07:49 2021 from 172.18.204.218
[root@gitlab-repo ~]$ cd /etc/graylog/server/
[root@gitlab-repo server]$ ls
log4j2.xml node-id server.conf
[techm@gitlab-repo server]$

also i found there is no graylog folder inside /etc/ssl/certs folder

[techm@gitlab-repo server]$ cd /etc/ssl/certs/
[techm@gitlab-repo certs]$ ls
ca-bundle.crt cacerts clientcert make-dummy-cert renew-dummy-cert
ca-bundle.trust.crt cacerts.jks client.pem Makefile selfsigned.crt
[techm@gitlab-repo certs]$ ll
total 24
lrwxrwxrwx. 1 root root 49 Jul 16 2020 ca-bundle.crt → /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root 55 Jul 16 2020 ca-bundle.trust.crt → /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
lrwxrwxrwx 1 root root 41 Jun 3 2020 cacerts → …/…/…/…/…/…/…/etc/pki/java/cacerts
lrwxrwxrwx 1 root root 41 Jun 3 2020 cacerts.jks → …/…/…/…/…/…/…/etc/pki/java/cacerts
-rw-rw-r-- 1 techm techm 2302 Sep 5 03:27 clientcert
-rw-rw-r-- 1 techm techm 3089 Sep 5 03:18 client.pem
-rwxr-xr-x. 1 root root 610 Aug 9 2019 make-dummy-cert
-rw-r–r–. 1 root root 2516 Aug 9 2019 Makefile
-rwxr-xr-x. 1 root root 829 Aug 9 2019 renew-dummy-cert
-rw-r–r-- 1 root root 3057 Sep 9 05:31 selfsigned.crt
[root@gitlab-repo certs]$

Hello,

Oh, I see. Sorry, Long day for me.
Unfortunately, I don’t have the same set up as you but your error still stands.

Which means you need to set your Keystore so Graylog can pick it up. Also make sure Graylog can reach your certs. If its owned by someone else it wont be able to reach them. “unable to find valid certificate path” in order for the JVM to pick up the new trust store, it has to be started with the JVM parameter -Djavax.net.ssl.trustStore=/path/to/cacerts.jks. If your using default Java Keystore ( i.e. cacerts) it shouldn’t be a problem.

  1. Check your Keystore
  2. Check your permission on the certs
  3. Check Java

This may help
https://docs.graylog.org/en/3.0/pages/configuration/https.html#adding-a-self-signed-certificate-to-the-jvm-trust-store

EDIT: Here are Default file locations if needed
https://docs.graylog.org/en/4.1/pages/configuration/file_location.html#default-file-location

unfortunately i tried everything and now running out of ideas , how to fix this issue when graylog is running in a pod

Hello,

Showing us what you did would be helpful. This way we are not telling you something you have done already. It will also show us any incorrect configuration/commands you have done and maybe we can correct it.
The error tells use Graylog cant find the certificates (unable to find valid certificate path) but the question is how to configure environment to so solve this. Unfortunately, I unfamiliar with Docker/Pods for commands to solve this.

By chance Is your GRAYLOG_HTTP_TLS_* key/cert on that path you have configured available INSIDE the container?

What I’m not seeing is this part of your Graylog configuration.

volumes:
- ./certs/graylog-key.pem:/etc/docker/certs/graylog-key.pem
- ./certs/graylog-cert.pem:/etc/docker/certs/graylog-cert.pem
environment:
- GRAYLOG_HTTP_TLS_CERT_FILE=/etc/docker/certs/graylog-cert.pem
- GRAYLOG_HTTP_TLS_KEY_FILE=/etc/docker/certs/graylog-key.pem

Maybe something in these posts might help.

I’m sorry I cant be more help to give you a direct answer on what to do. I know this error really well on other type of installation and it a very easy fix. Since your using a Pod I’m unsure how you need to proceed. Maybe someone here has a better understand with your environment. Also I am curious on how you fix this so keep us update if you do.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.