Graylog 4.2.7 + docker compose + https = semi fail

nitrogene · March 24, 2022, 11:02am

1. Describe your incident:

I am trying to migrate a working non-https graylog solution to an https one. I am able to connect to the web interface, but as soon as I do, the docker logs are populated in loop with the following lines:

some-graylog        | 2022-03-24 10:29:56,447 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://some-graylog:9000/api/system/metrics/multiple on node <795270ca-9ef2-43de-a800-063115f3a018>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
some-graylog        | 2022-03-24 10:29:58,553 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://some-graylog:9000/api/system/metrics/multiple on node <795270ca-9ef2-43de-a800-063115f3a018>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

None of the inputs are running. If I try to navigate to system/nodes/, I stumble on an error page with following message:

FetchError: There was an error fetching a resource: Internal Server Error. Additional information: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Check your Graylog logs for more information.

2. Describe your environment:

OS Information: Windows 10 + WSL2/ubuntu. I am using wsl2 shells for all the stuff. And a windows 10 browser.
Package Version: graylog/graylog:4.2.7-1
Service logs, configurations, and environment variables:

Here are the logs before connecting to the web interface:

$ docker logs some-graylog
wait-for-it: waiting 15 seconds for elasticsearch:9200
wait-for-it: timeout occurred after waiting 15 seconds for elasticsearch:9200
2022-03-24 10:42:30,285 INFO : org.graylog2.featureflag.ImmutableFeatureFlagsCollector - Following feature flags are used: {}
2022-03-24 10:42:31,336 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: AWS plugins 4.2.7 [org.graylog.aws.AWSPlugin]
2022-03-24 10:42:31,338 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Integrations 4.2.7 [org.graylog.integrations.IntegrationsPlugin]
2022-03-24 10:42:31,340 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 4.2.7 [org.graylog.plugins.collector.CollectorPlugin]
2022-03-24 10:42:31,342 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Threat Intelligence Plugin 4.2.7 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2022-03-24 10:42:31,342 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elasticsearch 6 Support 4.2.7+879e651 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2022-03-24 10:42:31,343 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elasticsearch 7 Support 4.2.7+879e651 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2022-03-24 10:42:31,381 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
2022-03-24 10:42:31,628 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator null
2022-03-24 10:42:37,841 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
2022-03-24 10:42:37,887 INFO : org.graylog2.plugin.system.NodeId - Node ID: 795270ca-9ef2-43de-a800-063115f3a018
2022-03-24 10:42:38,282 INFO : org.graylog.shaded.kafka09.log.LogManager - Loading logs.
2022-03-24 10:42:38,354 INFO : org.graylog.shaded.kafka09.log.LogManager - Logs loading complete.
2022-03-24 10:42:38,361 INFO : org.graylog2.shared.journal.LocalKafkaJournal - Initialized Kafka based journal at data/journal
2022-03-24 10:42:38,432 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[mongo:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-03-24 10:42:38,565 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
2022-03-24 10:42:38,620 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:1}] to mongo:27017
2022-03-24 10:42:38,630 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongo:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 12]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=6334200}
2022-03-24 10:42:38,657 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:2}] to mongo:27017
2022-03-24 10:42:38,991 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2022-03-24 10:42:39,906 INFO : org.graylog2.storage.providers.ElasticsearchVersionProvider - Elasticsearch cluster is running v7.10.2
2022-03-24 10:42:41,288 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-03-24 10:42:41,526 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-24 10:42:41,551 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-03-24 10:42:41,576 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:3}] to mongo:27017
2022-03-24 10:42:41,614 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-24 10:42:41,661 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-24 10:42:41,704 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-24 10:42:41,743 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-03-24 10:42:42,691 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 4.2.7+879e651 starting up
2022-03-24 10:42:42,692 INFO : org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_322 on Linux 5.10.60.1-microsoft-standard-WSL2
2022-03-24 10:42:42,693 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: docker
2022-03-24 10:42:42,694 INFO : org.graylog2.bootstrap.ServerBootstrap - OS: Debian GNU/Linux 11 (bullseye) (debian)
2022-03-24 10:42:42,694 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: amd64
2022-03-24 10:42:42,732 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 29 periodicals ...
2022-03-24 10:42:42,733 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2022-03-24 10:42:42,748 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2022-03-24 10:42:42,774 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2022-03-24 10:42:42,774 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2022-03-24 10:42:42,792 INFO : org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration - Legacy default stream has no connections, no migration needed.
2022-03-24 10:42:42,808 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2022-03-24 10:42:42,811 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2022-03-24 10:42:42,811 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2022-03-24 10:42:42,814 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2022-03-24 10:42:42,815 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2022-03-24 10:42:42,816 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2022-03-24 10:42:42,818 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2022-03-24 10:42:42,831 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2022-03-24 10:42:42,835 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2022-03-24 10:42:42,836 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2022-03-24 10:42:42,845 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:5}] to mongo:27017
2022-03-24 10:42:42,846 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:4}] to mongo:27017
2022-03-24 10:42:42,849 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2022-03-24 10:42:42,859 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2022-03-24 10:42:42,864 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2022-03-24 10:42:42,865 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2022-03-24 10:42:42,878 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2022-03-24 10:42:42,878 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2022-03-24 10:42:42,879 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2022-03-24 10:42:42,887 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:6}] to mongo:27017
2022-03-24 10:42:42,887 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2022-03-24 10:42:42,894 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2022-03-24 10:42:42,907 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2022-03-24 10:42:42,928 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2022-03-24 10:42:42,941 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2022-03-24 10:42:42,959 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2022-03-24 10:42:42,961 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2022-03-24 10:42:43,020 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2022-03-24 10:42:47,024 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [some-graylog:9000]
2022-03-24 10:42:47,026 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
2022-03-24 10:42:47,027 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <some-graylog:9000>
2022-03-24 10:42:47,027 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
2022-03-24 10:42:47,029 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {FailureHandlingService [RUNNING]=3, GracefulShutdownService [RUNNING]=33, LocalKafkaMessageQueueReader [RUNNING]=33, LocalKafkaMessageQueueWriter [RUNNING]=33, UserSessionTerminationService [RUNNING]=35, UrlWhitelistService [RUNNING]=53, PrometheusExporter [RUNNING]=54, LocalKafkaJournal [RUNNING]=54, InputSetupService [RUNNING]=56, BufferSynchronizerService [RUNNING]=62, MongoDBProcessingStatusRecorderService [RUNNING]=76, JobSchedulerService [RUNNING]=96, EtagService [RUNNING]=97, ConfigurationEtagService [RUNNING]=98, OutputSetupService [RUNNING]=101, LookupTableService [RUNNING]=186, PeriodicalsService [RUNNING]=294, StreamCacheService [RUNNING]=338, JerseyService [RUNNING]=4297}
2022-03-24 10:42:47,030 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2022-03-24 10:42:47,049 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
2022-03-24 10:42:47,068 INFO : org.graylog2.inputs.InputStateListener - Input [GELF UDP/62347edb3fbc7b573dc69969] is now STARTING
2022-03-24 10:42:47,198 INFO : org.graylog2.inputs.InputStateListener - Input [GELF UDP/62347edb3fbc7b573dc69969] is now RUNNING

And after connecting to the web interface:

2022-03-24 10:44:30,941 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://some-graylog:9000/api/system/metrics/multiple on node <795270ca-9ef2-43de-a800-063115f3a018>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-03-24 10:44:32,472 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://some-graylog:9000/api/system/metrics/multiple on node <795270ca-9ef2-43de-a800-063115f3a018>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-03-24 10:44:34,436 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://some-graylog:9000/api/system/metrics/multiple on node <795270ca-9ef2-43de-a800-063115f3a018>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-03-24 10:44:36,443 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://some-graylog:9000/api/system/metrics/multiple on node <795270ca-9ef2-43de-a800-063115f3a018>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-03-24 10:44:38,477 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://some-graylog:9000/api/system/metrics/multiple on node <795270ca-9ef2-43de-a800-063115f3a018>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

3. What steps have you already taken to try and solve the problem?

Making a self signed certificate. Here’s my openssl file:

[req]
# Options for the `req` tool (`man req`).
default_bits        = 4096
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions = v3_req

prompt = no

[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
C = US
ST = Some-State
L = Some-City
O = My Company
OU = My Division
CN = some-graylog

[v3_req]
keyUsage =  nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

# IP addresses and DNS names the certificate should include
# Use IP.### for IP addresses and DNS.### for DNS names,
# with "###" being a consecutive number.
[alt_names]
IP.1 = 127.0.0.1
DNS.1 = localhost
DNS.2 = some-graylog

After certificates generation, I added the cert.pem file to Windows, then I modified the docker file to add the cert to the keystore:

FROM graylog/graylog:4.2.7-1

# I don't know ho to do the following at running by altering entrypoint

USER root 

COPY cert.pem /cert.pem

# add certificate to keystore
RUN rm -f /usr/share/graylog/cacerts.jks \
&& cp -a "${JAVA_HOME}/lib/security/cacerts" /usr/share/graylog/cacerts.jks \
&& keytool -noprompt -importcert -keystore /usr/share/graylog/cacerts.jks -storepass changeit -alias graylog-self-signed -file /cert.pem \
&& rm -f /cert.pem \
&& chown graylog:graylog /usr/share/graylog/cacerts.jks

USER graylog

While building the image, I could see that the cert was added to the keystore - and checked it via:

docker exec -it some-graylog bash
keytool -list -v -keystore cacerts.jks | grep graylog-self-signed

Here is the relevant part of the docker compose file:

 graylog:
      build: ./some-graylog
      container_name: some-graylog
      hostname: some-graylog
      volumes:
        - graylog-storage:/usr/share/graylog/data:rw
        - ./some-graylog/plugin:/usr/share/graylog/plugin:ro
        - /etc/certs/graylog:/etc/certs/graylog:ro
      environment:
        - GRAYLOG_SERVER_JAVA_OPTS ="-Xms8G -Xmx8G -Djavax.net.ssl.trustStore=/usr/share/graylog/cacerts.jks"
        # CHANGE ME (must be at least 16 characters)!
        - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
        # User / Password: admin/admin
        - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
        # http://docs.graylog.org/en/3.0/pages/configuration/server.conf.html#web-rest-api
        # https://archivedocs.graylog.org/en/3.0/pages/upgrade/graylog-3.0.html#simplified-http-interface-configuration
        # https://hometechhacker.com/how-to-create-a-graylog-container-in-docker/
        - GRAYLOG_HTTP_BIND_ADDRESS=some-graylog:9000
        - GRAYLOG_HTTP_ENABLE_TLS=true
        - GRAYLOG_HTTP_TLS_CERT_FILE= /etc/certs/graylog/cert.pem
        - GRAYLOG_HTTP_TLS_KEY_FILE= /etc/certs/graylog/pkcs8-encrypted.pem
        # The following env variable is stored in a dedicated env file
        #- GRAYLOG_HTTP_TLS_KEY_PASSWORD=

      entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 -- /docker-entrypoint.sh
      networks:
        - graylog
      restart: "no"
      depends_on:
        - fluentd
        - mongo
        - elasticsearch
      ports:
        # Graylog web interface and REST API
        - 9000:9000
        # Syslog TCP
        - 1514:1514
        # Syslog UDP
        - 1514:1514/udp
        # GELF TCP
        - 12201:12201
        # GELF UDP
        - 12201:12201/udp
      env_file:
        - ./some-graylog/graylog.env

The env file export GRAYLOG_HTTP_TLS_KEY_PASSWORD (used to encrypt the cert).

4. How can the community help?

I think I am almost there. From what I saw on forums, it seems that this error is due to a missing cert in the java keystore, but I verified it was added. Maybe there is an issue with the openssl configuration file ?

Best regards,

Jean-Pierre

gsmith · March 24, 2022, 9:54pm

Hello && Welcome

I would like to direct you to this part of the log shown above.

This issue could be a couple different things.

1.Does and can, Graylog access the certificates needed? Exampe: chown graylog:graylog
2.Does Graylog have access to the keystore?
3.Using the correct certificates?

What I’m seeing is not how the certs were created yet, but Graylog not being able to find them ( i.e. access)

Next set of questions

I think I see another issue perhaps. Can I ask what documentation did you use to set up Graylog Self-singed certificates? If it was from here , then you should have these two certificates being used.

Using HTTPS

From this section as shown below.

The resulting PKCS#8 private key (graylog-key.pem) and the X.509 certificate (graylog-certificate.pem) can now be used to enable encrypted connections with Graylog by enabling TLS for the Graylog REST API and the web interface in the Graylog configuration file:

# Enable HTTPS support for the HTTP interface.
# This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
http_tls_cert_file = /path/to/graylog-certificate.pem

# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
http_tls_key_file = /path/to/graylog-key.pem

# The password to unlock the private key used for securing the HTTP interface. (if key is encrypted)
http_tls_key_password = secret

nitrogene · March 25, 2022, 7:05am

Hello gsmith,

cert.pem and pkcs8-encrypted.pem are shared with host via a volume share:

      volumes:
        - graylog-storage:/usr/share/graylog/data:rw
        - ./some-graylog/plugin:/usr/share/graylog/plugin:ro
        - /etc/certs/graylog:/etc/certs/graylog:ro

And as I was able to log into web app, I assumed that these instructions were OK. The graylog user have indeed access to these files:

docker exec -it some-graylog bash
graylog@some-graylog:/etc/certs/graylog$ cat cert.pem
-----BEGIN CERTIFICATE-----
....
-----END CERTIFICATE-----
graylog@some-graylog:/etc/certs/graylog$ cat pkcs8-encrypted.pem
-----BEGIN ENCRYPTED PRIVATE KEY-----
...
-----END ENCRYPTED PRIVATE KEY-----

I think it does. The cacerts.jks is created in the graylog usr folder, with an extra chown:

USER root 

COPY cert.pem /cert.pem

# add certificate to keystore
RUN rm -f /usr/share/graylog/cacerts.jks \
&& cp -a "${JAVA_HOME}/lib/security/cacerts" /usr/share/graylog/cacerts.jks \
&& keytool -noprompt -importcert -keystore /usr/share/graylog/cacerts.jks -storepass changeit -alias graylog-self-signed -file /cert.pem \
&& rm -f /cert.pem \
&& chown graylog:graylog /usr/share/graylog/cacerts.jks

USER graylog

I was able to check at graylog container runtime that the cert was indeed added to the keystore via:

docker exec -it some-graylog bash
keytool -list -v -keystore cacerts.jks -alias graylog-self-signed

I indeed used Using HTTPS - Configuring Graylog, but I certainly may have misinterpreted these instructions:

CONVERTING A PKCS #12 (PFX) FILE TO PRIVATE KEY AND CERTIFICATE PAIR

PKCS #12 key stores (PFX files) are commonly used on Microsoft Windows. This needs to be done only if you have to convert PKCS #12 Keys to be used with Graylog.

In this example, the PKCS #12 (PFX) file is named keystore.pfx :
$ openssl pkcs12 -in keystore.pfx -nokeys -out graylog-certificate.pem
$ openssl pkcs12 -in keystore.pfx -nocerts -out graylog-pkcs5.pem
$ openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem
The resulting graylog-certificate.pem and graylog-key.pem can be used in the Graylog configuration file.

I have read these above instructions. I don’t think it apply to my case: please take a look to the text with emphasis. I generated cert.pem, pkcs8-encrypted.pem using the instructions before the above text. I integrated cert.pem to the java key store using these instructions:

ADDING A SELF-SIGNED CERTIFICATE TO THE JVM TRUST STORE
…
$ keytool -importcert -keystore /path/to/cacerts.jks -storepass changeit -alias graylog-self-signed -file cert.pem

nitrogene · March 25, 2022, 11:09am

It’s strange.

In the graylog container logs, I found these:

Running with JVM arguments: 
-Dlog4j2.formatMsgNoLookups=true 
-Djdk.tls.acknowledgeCloseNotify=true 
-XX:+UnlockExperimentalVMOptions 
-XX:NewRatio=1 
-XX:MaxMetaspaceSize=256m 
-XX:+ResizeTLAB 
-XX:-OmitStackTraceInFastThrow 
-XX:+UseParNewGC 
-XX:+UseConcMarkSweepGC 
-XX:+CMSConcurrentMTEnabled 
-XX:+CMSClassUnloadingEnabled 
-Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml 
-Djava.library.path=/usr/share/graylog/lib/sigar/ 
-Dgraylog2.installation_source=docker

No sign of:

environment:
        - "GRAYLOG_SERVER_JAVA_OPTS = -Djavax.net.ssl.trustStore=/usr/share/graylog/cacerts.jks"

I’ll continue to investigate

nitrogene · March 25, 2022, 11:20am

Interesting. I hacked the entry point script used in the graylog image:

#!/bin/bash

set -e

# save the settings over the docker(-compose) environment
__GRAYLOG_SERVER_JAVA_OPTS=${GRAYLOG_SERVER_JAVA_OPTS}

echo "GRAYLOG_SERVER_JAVA_OPTS=${GRAYLOG_SERVER_JAVA_OPTS}"
echo "GRAYLOG_PASSWORD_SECRET=${GRAYLOG_PASSWORD_SECRET}"

exit

And here’s the logs content:

some-graylog        | wait-for-it: elasticsearch:9200 is available after 12 seconds
some-graylog        | GRAYLOG_SERVER_JAVA_OPTS=
some-graylog        | GRAYLOG_PASSWORD_SECRET=XXXX
some-graylog exited with code 0

It seems that the GRAYLOG_SERVER_JAVA_OPTS environment variable is not set ?

still investigating

nitrogene · March 25, 2022, 12:22pm

I think I have nailed it. The correct syntax seems to be:

      environment:
        - GRAYLOG_SERVER_JAVA_OPTS="-Xms8G -Xmx8G -Djavax.net.ssl.trustStore=/usr/share/graylog/cacerts.jks"

The following is not correct (please notice the space left of ‘=’:

      environment:
        - GRAYLOG_SERVER_JAVA_OPTS ="-Xms8G -Xmx8G -Djavax.net.ssl.trustStore=/usr/share/graylog/cacerts.jks"

While playing with entry point / environment variables I also found that the following line is misinterpreted by the entry point:

environment:
        - GRAYLOG_SERVER_JAVA_OPTS="-Djavax.net.ssl.trustStore=/usr/share/graylog/cacerts.jks"

some-graylog        | Error: Could not find or load main class "-Djavax.net.ssl.trustStore=.usr.share.graylog.cacerts.jks"

Is it a known issue ? Do you want me to open a new issue ?

Anyway, I can move forward !

gsmith · March 25, 2022, 9:25pm

Hello,

Awesome glad you resolved this issue and way to troubleshoot.

I’m not 100% sure, I do know YAML files are touchy, meaning the indents/spacing/etc… need to be correct.

Not sure if you know about this program but on my Docker server I use yamllint this to check my files. Its not 100 % perfect but gets the job done & simple to use.

root# apt install yamllint

As for the correct syntax you could post what you learned on Github, perhaps someone can enlighten us on this. Not a bad Idea.

If you could mark this post as resolved for future searches that would be great
And thank you for posting your resolve

Thanks

nitrogene · March 28, 2022, 7:20am

Hello,

I will soon post how i setup graylog + fluentd + fluentbit + TLS soon. Thank for your support.

Regards,

Jean-Pierre

system · April 11, 2022, 7:21am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://192.168.0.165:9000/api/system/metrics/multiple on node Graylog Central (peer support)	3	1931	November 19, 2020
Docker-compose - Trusted CA Signed SSL Certificates but "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target" Graylog Central (peer support) docker	4	4036	April 26, 2022
Graylog-server v4.0.7-1 - sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Graylog Central (peer support)	5	4761	June 8, 2021
Need help with setup commercial certificate in Graylog 4.1.x Graylog Central (peer support)	5	1526	October 20, 2021
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Graylog Central (peer support)	5	3357	June 7, 2023

Graylog 4.2.7 + docker compose + https = semi fail

Related topics