Graylog to nucool integration issue (Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I am trying to integrate graylog with nucool system using webhook URL provided by nucool system and getting pkix path building failed error. I am using graylog alert http notification feature to implement this.
(Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)

2. Describe your environment:

  • OS Information:

cat os-release

PRETTY_NAME=“Debian GNU/Linux 10 (buster)”
NAME=“Debian GNU/Linux”
VERSION_ID=“10”
VERSION=“10 (buster)”
VERSION_CODENAME=buster
ID=debian
HOME_URL=“https://www.debian.org/”
SUPPORT_URL=“Debian -- Support”
BUG_REPORT_URL=“https://bugs.debian.org/”

  • Package Version:

  • Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?
since nuccol oss system is using self signed certificate, so i have imported nuccol certificate to graylog cacerts using keytool utility but still error is not fixed.

webhook url https://172.18.72.210:4443/probe/webhook/NetOps/

image
image1366×768 53.9 KB

4. How can the community help?
Looking for a solution on how to fix this issue . api response and payload details are given below for reference.

https://graylog-netopsai.apps.ocp19.nfvdev.tlabs.ca/api/events/notifications/test

{“id”:“60c84f70f38d927e6effa3a7”,“title”:“cicd notification”,“description”:“cicd notification”,“config”:{“type”:“http-notification-v1”,“url”:"https://172.18.72.210:4443/probe/webhook/NetOps/"}}

image
image1366×768 74 KB

image
image1366×768 62.6 KB

image
image1366×768 57.8 KB

response header: {“Access-Control-Allow-Credentials”:“true”,“Access-Control-Allow-Headers”:“Authorization, Content-Type, X-Graylog-No-Session-Extension, X-Requested-With, X-Requested-By”,“Access-Control-Allow-Methods”:“GET, POST, PUT, DELETE, OPTIONS”,“Access-Control-Allow-Origin”:“https://graylog-netopsai.apps.ocp19.nfvdev.tlabs.ca”,“Access-Control-Max-Age”:“600”,“Connection”:“keep-alive”,“Content-Length”:“178”,“Content-Type”:“application/json”,“Date”:“Wed, 13 Apr 2022 04:56:33 GMT”,“Server”:“nginx/1.21.6”,“X-Graylog-Node-Id”:“d9b29696-31fe-47e4-8eab-55577f654583”,“X-Runtime-Microseconds”:“89328”}

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hello,

I’m also looking into something similar to this. Testing OIDC on Enterprise version and I did exactly what you did I added the Keycloak Certs to my Graylog keystore. I still ended up with the same error.

image
image1898×515 56 KB

I did notice this statement.

Performs a background connection check with the address and credentials defined in the step “Server Configuration”.

Not finding the documentation on HowTo yet.

I might have to post here if I’m unable to fix this.

3

Thanks @gsmith for your prompt response on this query. it would be great if you could provide a solution or workaround for this issue if u fix it.
if you post this query to Github, I request to share that detail so that i can follow up… thanks alot for your support .

1 Like
4

I’m working on it :smiley: Just want to let you know our not alone.

1 Like
5

Thank you …hope we can get some solution soon finger crossed:-)

6

Self-signed certs are a pain. One of the things to check is that the entire chain is trusted, not just the root certificate:

7

Hi Patrick,
Thanks for the details . I have deployed graylog on container so the procedure you have shared doesnt seems to work in my environment . i have tried to check the pki directory inside graylog but couldnt find it … if you could share the configuration w.r.t container deployment would help alot thanks.

8 
# keytool -importcert -keystore /usr/local/openjdk-8/lib/security/cacerts.jks -storepass changeit -alias graylog-self-signed -file /etc/ssl/certs/onsrbh-serv-netcoole-DU09.tlabs.ca.pem
Owner: CN=onsrbh-serv-netcoole-DU09.tlabs.ca, OU=World Class Assurance Team, O=TELUS, L=Toronto, ST=Ontario, C=CA
Issuer: CN=TCSO-issuing-CA, DC=corp, DC=ads
Serial number: 2d00013c40af2bedbaa63d9342000000013c40
Valid from: Thu Apr 07 14:55:59 UTC 2022 until: Fri Apr 05 09:00:00 UTC 2024
Certificate fingerprints:
         SHA1: 96:A8:48:B6:E5:9E:31:E1:04:30:63:F6:A6:9D:46:E5:A4:8F:AF:66
         SHA256: D1:48:0B:B9:C3:29:A3:7B:39:7B:A8:A1:28:B9:BF:71:83:83:FA:0F:1D:6F:53:3D:62:C1:10:45:8F:C5:6E:53
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 1.3.6.1.4.1.311.21.10 Criticality=false
0000: 30 18 30 0A 06 08 2B 06   01 05 05 07 03 02 30 0A  0.0...+.......0.
0010: 06 08 2B 06 01 05 05 07   03 01                    ..+.......


#2: ObjectId: 1.3.6.1.4.1.311.21.7 Criticality=false
0000: 30 2D 06 25 2B 06 01 04   01 82 37 15 08 81 B4 BB  0-.%+.....7.....
0010: 4F 81 D6 DF 7D 87 CD 87   35 81 8A FC 52 85 C2 FA  O.......5...R...
0020: 2B 22 9E FA 3C 86 F2 8D   13 02 01 64 02 01 19     +"..<......d...


#3: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: caIssuers
   accessLocation: URIName: ldap:///CN=TCSO-issuing-CA,CN=AIA,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=corp,DC=ads?cACertificate?base?objectClass=certificationAuthority
,
   accessMethod: caIssuers
   accessLocation: URIName: http://btwp013980/cdp/btwp013979.corp.ads_TCSO-issuing-CA.crt
,
   accessMethod: caIssuers
   accessLocation: URIName: http://btwp013983/cdp/btwp013979.corp.ads_TCSO-issuing-CA.crt
,
   accessMethod: caIssuers
   accessLocation: URIName: http://wp81174/cdp/btwp013979.corp.ads_TCSO-issuing-CA.crt
,
   accessMethod: caIssuers
   accessLocation: URIName: http://wp81175/cdp/btwp013979.corp.ads_TCSO-issuing-CA.crt
,
   accessMethod: caIssuers
   accessLocation: URIName: http://tcsocdp.tsl.telus.com/CertEnroll/btwp013979.corp.ads_TCSO-issuing-CA.crt
,
   accessMethod: ocsp
   accessLocation: URIName: http://tcsocdp.tsl.telus.com/ocsp
]
]

#4: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: AB 49 AF B3 5D A1 42 D3   4A E4 7D 7D B4 93 D9 7B  .I..].B.J.......
0010: C3 2B ED EF                                        .+..
]
]

#5: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [URIName: ldap:///CN=TCSO-issuing-CA,CN=btwp013979,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=corp,DC=ads?certificateRevocationList?base?objectClass=cRLDistributionPoint, URIName: http://tcsocdp.tsl.telus.com/CertEnroll/TCSO-issuing-CA.crl, URIName: http://btwp013980/cdp/TCSO-issuing-CA.crl, URIName: http://btwp013983/cdp/TCSO-issuing-CA.crl, URIName: http://wp81174/cdp/TCSO-issuing-CA.crl, URIName: http://wp81175/cdp/TCSO-issuing-CA.crl]
]]

#6: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  clientAuth
  serverAuth
]

#7: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
  DigitalSignature
  Key_Encipherment
]

#8: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: onsrbh-serv-netcoole-DU09.tlabs.ca
  DNSName: onsrbh-serv-netcoole-DU09
  IPAddress: 172.18.102.20
  IPAddress: 172.18.72.210
]

#9: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: CD B5 8A 68 08 78 DC 8B   AF 13 F3 05 CE 60 18 83  ...h.x.......`..
0010: 68 68 33 C8                                        hh3.
]
]

Trust this certificate? [no]:  yes
Certificate was added to keystore
# keytool -keystore /usr/local/openjdk-8/lib/security/cacerts.jks -storepass changeit -list | grep graylog-self-signed -A1
graylog-self-signed, Apr 14, 2022, trustedCertEntry,
**Certificate fingerprint (SHA-256): D1:48:0B:B9:C3:29:A3:7B:39:7B:A8:A1:28:B9:BF:71:83:83:FA:0F:1D:6F:53:3D:62:C1:10:45:8F:C5:6E:53**
#

when i import the self singed certiticate i can see both SHA1 and sha256 certificate fingerprint … while listing i can see certifiacte fingerprint match with sha256 and as per graylog document it should match with SHA1 …is it expected behavior or i have to change the certifiacte format

9

Hello @Nitin
With your issue did you declare the keystore as shown below?

In order for the JVM to pick up the new trust store, it has to be started with the JVM parameter -Djavax.net.ssl.trustStore=/path/to/cacerts.jks. If you’ve been using another password to encrypt the JVM trust store than the default changeit, you additionally have to set the JVM parameter -Djavax.net.ssl.trustStorePassword=secret.

Example:

GRAYLOG_SERVER_JAVA_OPTS "-Djavax.net.ssl.trustStore=/usr/local/openjdk-8/lib/security/cacerts.jks"
10

Hello,

So I figured my issue out.

Like I said this is different situation then yours but same error.
The error I’m seeing was that I had to get the certificates from Keycloak Realm and scp them over to my Graylog JAVA keystore in which I’m using the default JAVA keystore called cacerts.

I have two certificates.
1.localhost.pem
2.localhost-key.pem

Used the first certificate localhost.pem then executed the following

keytool -importcert -keystore cacerts -storepass changeit -alias keycloak_server -file localhost.pem

Restarted Graylog’s service and add the correct URL, in my case the correct OIDC base URL
The results were successes.

image
image1651×754 59.9 KB

Conclusion.

The correct Certificate in the right keystore Graylog is using. Using the Default JAVA keystore I didn’t have to declare it, but if not you need to do that.

I did test different certificates out till I found one that was correct that would work in my environment.

EDIT: I forgot to add this is Graylog Docker

11

thanks gsmith , i did exactly same but somehow its not working for me. is it possible for you if you can share the graylog yaml file which you could have used for the graylog container deployment will help thanks.

12

@Nitin

Sure,
Just so you know my Graylog Docker container is not using HTTPS. So basically Graylog using HTTP which is connecting to Keycloak using HTTPS. Transferred the certs from Keycloak and insert it into cacerts.

I had this error before on a YUM install. I inserted the wrong / incorrect certificate and graylog did not have access to the keystore.

docker-compose.yaml 
version: '3'
services:
   # MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    image: mongo:4
    network_mode: bridge
   # DB in share for persistence
    volumes:
      - mongo_data:/data/db
   # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.6/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
    network_mode: bridge
    #data folder in share for persistence
    volumes:
      - es_data:/usr/share/elasticsearch/data
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
   # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:4.2-jre11
    network_mode: bridge
    dns:
      - 8.8.8.8
      - 8.8.4.4
   # journal and config directories in local NFS share for persistence
    volumes:
       #- graylog_journal:/usr/share/graylog/data/journal
       - graylog_bin:/usr/share/graylog/bin
       - graylog_data:/usr/share/graylog/data/config
       - graylog_log:/usr/share/graylog/data/log
       - graylog_plugin:/usr/share/graylog/data/plugin
       - graylog_content:/usr/share/graylog/data/contentpacks
      # Mount local configuration directory into Docker container

       #- ./graylog/data/journal:/usr/share/graylog/data/journal
       #- ./graylog/config:/usr/share/graylog/data/config
    environment:
      # Container time Zone
      - TZ=America/Chicago
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=pJod1LTZuyb2YW9eHiKLTifjy7gMtnwZf6Q79HW2nonDhN
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=ef92b778ba166c06659911881f383d4473e94f
      - GRAYLOG_HTTP_BIND_ADDRESS=0.0.0.0:9000
      - GRAYLOG_HTTP_EXTERNAL_URI=http://10.10.10.10:9000/
      - GRAYLOG_ROOT_TIMEZONE=America/Chicago
      - GRAYLOG_ROOT_EMAIL=greg.smith@domain.com
      - GRAYLOG_HTTP_PUBLISH_URI=http://10.210.10.10:9000/
      - GRAYLOG_TRANSPORT_EMAIL_PROTOCOL=smtp
      - GRAYLOG_HTTP_ENABLE_CORS=true
      - GRAYLOG_TRANSPORT_EMAIL_WEB_INTERFACE_URL=http://10.10.10.10:9000/
      - GRAYLOG_TRANSPORT_EMAIL_HOSTNAME=10.10.10.10
      - GRAYLOG_TRANSPORT_EMAIL_ENABLED=true
      - GRAYLOG_TRANSPORT_EMAIL_PORT=25
      - GRAYLOG_TRANSPORT_EMAIL_USE_AUTH=false
      - GRAYLOG_TRANSPORT_EMAIL_USE_TLS=false
      - GRAYLOG_TRANSPORT_EMAIL_USE_SSL=false
      - GRAYLOG_TRANSPORT_FROM_EMAIL=root@localhost
      - GRAYLOG_TRANSPORT_SUBJECT_PREFIX=[graylog]
      - GRAYLOG_REPORT_DISABLE_SANDBOX=true
      # - GRAYLOG_REPORT_RENDER_URI=http://10.10.10.10:9000
      # - GRAYLOG_REPORT_USER=graylog-report
      # - GRAYLOG_REPORT_RENDER_ENGINE_PORT=9515
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 8514:8514
      # Elasticsearch
      - 9200:9200
      - 9300:9300
      # Syslog UDP
      - 8514:8514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
      # Reports
      - 9515:9515
      - 9515:9515/udp
      # beats
      - 5044:5044
      # email
      - 25:25
      - 25:25/udp
      # web
      - 80:80
      - 443:443
      - 21:21
#Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_journal:
    driver: local
  graylog_bin:
    driver: local
  graylog_data:
    driver: local
  graylog_log:
    driver: local
  graylog_plugin:
    driver: local
  graylog_content:
    driver: local

Past couple months I’ve been learning Docker stuff but haven’t started working with encryption yet.
Sorry for my YAML file being a mess this is just my lab stuff.

Please keep in mind, I inserted keys from Keycloak in my JAVA default keystore, this is where Graylog looks for, if not defined.

Hope that helps

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.