Web UI not working only when HTTPS is enabled, not listening on port 9000

Hi everyone,

I am trying to setup HTTPS on Graylog 4.1 - I am new to graylog.

I installed Graylog successfully and it is working: I can reach the HTTP Web UI and it is injesting logs fine. I have also configured the FQDN so I can successfully use graylog.mydomain.com
Now I want to enable enable HTTPS. As we have an internal CA, I generated the key, the .csr and signed it, so I now have the final .pem certificate.
When I change the server.conf file with the following parameters and reboot, even if I let Graylog alone for 1 hour (so that it has more than sufficient time to load the services), it looks like it is not listening at all in port 9000. If instead I disabled the HTTPS settings, when I reboot and I wait a couple of minutes it loads the Web UI again in HTTP without any issue.

netstat -l with only HTTP:

$ netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 graylog.mydomain:37109 0.0.0.0:*               LISTEN
tcp        0      0 localhost:domain        0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN
tcp        0      0 graylog.mydomain.com:27017 0.0.0.0:*               LISTEN
tcp6       0      0 graylog.mydomain:9200 [::]:*                  LISTEN
tcp6       0      0 localhost6.localdo:9200 [::]:*                  LISTEN
tcp6       0      0 graylog.mydomain:9300 [::]:*                  LISTEN
tcp6       0      0 localhost6.localdo:9300 [::]:*                  LISTEN
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN
tcp6       0      0 [::]:9000               [::]:*                  LISTEN
udp        0      0 localhost:domain        0.0.0.0:*
udp        0      0 graylog.mydomain.s:bootpc 0.0.0.0:*
raw6       0      0 [::]:ipv6-icmp          [::]:*                  7

Netstat when HTTPS is enabled

$ netstat -tpln
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:39023         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      -
tcp6       0      0 ::1:9200                :::*                    LISTEN      -
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      -
tcp6       0      0 ::1:9300                :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

server.conf

http_bind_address = 0.0.0.0:9000
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog.mydomain.com.pem
http_tls_key_file = /etc/ssl/certs/graylog.mydomain.com.key
http_publish_uri = https://192.168.8.156:9000/

I also tried to query and start the service manually, but this is what I get. So it seems that with those settings the services are simply not starting

$ systemctl status graylog-server.service
● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2021-07-02 12:34:30 UTC; 1s ago
     Docs: http://docs.graylog.org/
  Process: 2456 ExecStart=/usr/share/graylog-server/bin/graylog-server (code=exited, status=1/FAILURE)
 Main PID: 2456 (code=exited, status=1/FAILURE)

Thanks in advance!

Hi there,

What do you see in the logs?

@0d4y5

Just chimming in, I noticed you may be missing a configuration in your server file.
Here is mine

http_bind_address = 8.8.8.8:9000
http_publish_uri = https://graylog.domain.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = secret

Thanks @gsmith and @aaronsachs for your suggestions,

I set the enable cors, enabled the uri on the hostname, and enabled the tls key password.
By looking at the logs, I could see the following:

2021-07-03T20:14:11.911Z ERROR [CmdLineTool] Invalid configuration
com.github.joschi.jadconfig.ValidationException: Unreadable or missing HTTP private key: /etc/ssl/certs/graylog.mydomain.key
        at org.graylog2.configuration.HttpConfiguration.validateTlsConfig(HttpConfiguration.java:252) ~[graylog.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:100) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:380) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:207) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]

I modified the permissions of the key file so that it can be read, and now that error is fixed.
Unfortunately I get another error, but honestly I can’t figure out what is wrong here:

2021-07-05T08:15:58.836Z INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.1.0 [org.graylog.aws.AWSPlugin]
2021-07-05T08:15:58.841Z INFO  [CmdLineTool] Loaded plugin: Enterprise Integrations 4.1.0 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2021-07-05T08:15:58.843Z INFO  [CmdLineTool] Loaded plugin: Integrations 4.1.0 [org.graylog.integrations.IntegrationsPlugin]
2021-07-05T08:15:58.845Z INFO  [CmdLineTool] Loaded plugin: Collector 4.1.0 [org.graylog.plugins.collector.CollectorPlugin]
2021-07-05T08:15:58.847Z INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise 4.1.0 [org.graylog.plugins.enterprise.EnterprisePlugin]
2021-07-05T08:15:58.849Z INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES6 Support) 4.1.0 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es6.EnterpriseES6Plugin]
2021-07-05T08:15:58.850Z INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES7 Support) 4.1.0 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es7.EnterpriseES7Plugin]
2021-07-05T08:15:58.851Z INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.1.0 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-07-05T08:15:58.851Z INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.1.0+4eb2147 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-07-05T08:15:58.852Z INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.1.0+4eb2147 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-07-05T08:15:58.903Z INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-07-05T08:15:59.104Z INFO  [Version] HV000001: Hibernate Validator null
2021-07-05T08:16:01.464Z INFO  [InputBufferImpl] Message journal is enabled.
2021-07-05T08:16:01.481Z INFO  [NodeId] Node ID: 6bea7bfe-fe87-4144-8300-58f731aa0e4c
2021-07-05T08:16:01.603Z INFO  [LogManager] Loading logs.
2021-07-05T08:16:01.643Z INFO  [LogManager] Logs loading complete.
2021-07-05T08:16:01.645Z INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-07-05T08:16:01.666Z INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-07-05T08:16:01.700Z INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-07-05T08:16:01.724Z INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:74}] to localhost:27017
2021-07-05T08:16:01.731Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 14]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4407295}
2021-07-05T08:16:01.743Z INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:75}] to localhost:27017
2021-07-05T08:16:01.891Z INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-07-05T08:16:02.325Z INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.10.2
2021-07-05T08:16:03.034Z INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-07-05T08:16:03.094Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-07-05T08:16:03.112Z INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-07-05T08:16:03.136Z INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:76}] to localhost:27017
2021-07-05T08:16:03.155Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-07-05T08:16:03.178Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-07-05T08:16:03.198Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-07-05T08:16:03.217Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-07-05T08:16:03.959Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-07-05T08:16:03.962Z INFO  [ServerBootstrap] Graylog server 4.1.0+4eb2147 starting up
2021-07-05T08:16:03.962Z INFO  [ServerBootstrap] JRE: Ubuntu 11.0.11 on Linux 5.4.0-77-generic
2021-07-05T08:16:03.962Z INFO  [ServerBootstrap] Deployment: deb
2021-07-05T08:16:03.963Z INFO  [ServerBootstrap] OS: Ubuntu 18.04.5 LTS (bionic)
2021-07-05T08:16:03.963Z INFO  [ServerBootstrap] Arch: amd64
2021-07-05T08:16:04.015Z INFO  [PeriodicalsService] Starting 36 periodicals ...
2021-07-05T08:16:04.017Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-07-05T08:16:04.041Z INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:77}] to localhost:27017
2021-07-05T08:16:04.046Z INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-07-05T08:16:04.061Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-07-05T08:16:04.062Z INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-07-05T08:16:04.063Z INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-07-05T08:16:04.078Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-07-05T08:16:04.082Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-07-05T08:16:04.082Z INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-07-05T08:16:04.082Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-07-05T08:16:04.084Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-07-05T08:16:04.089Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-07-05T08:16:04.094Z INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-07-05T08:16:04.097Z INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-07-05T08:16:04.106Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-07-05T08:16:04.110Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-07-05T08:16:04.117Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-07-05T08:16:04.117Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-07-05T08:16:04.118Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-07-05T08:16:04.129Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-07-05T08:16:04.144Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-07-05T08:16:04.144Z INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-07-05T08:16:04.172Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2021-07-05T08:16:04.175Z INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-07-05T08:16:04.176Z INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-07-05T08:16:04.210Z INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-07-05T08:16:04.214Z INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-07-05T08:16:04.223Z INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-07-05T08:16:04.229Z INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-07-05T08:16:04.234Z INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-07-05T08:16:04.240Z INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-07-05T08:16:04.241Z INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-07-05T08:16:04.253Z INFO  [Periodicals] Starting [org.graylog.plugins.license.LicenseManagerPeriodical] periodical in [0s], polling every [300s].
2021-07-05T08:16:04.266Z INFO  [Periodicals] Starting [org.graylog.plugins.license.LicenseReportPeriodical] periodical in [300s], polling every [3600s].
2021-07-05T08:16:04.267Z INFO  [Periodicals] Starting [org.graylog.plugins.license.StagedLicenseInstallerPeriodical] periodical, running forever.
2021-07-05T08:16:04.274Z INFO  [Periodicals] Starting [org.graylog.plugins.auditlog.mongodb.MongoAuditLogPeriodical] periodical in [0s], polling every [3600s].
2021-07-05T08:16:04.275Z INFO  [Periodicals] Starting [org.graylog.plugins.report.scheduler.ReportPeriodical] periodical in [120s], polling every [60s].
2021-07-05T08:16:04.275Z INFO  [Periodicals] Starting [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical] periodical in [60s], polling every [180s].
2021-07-05T08:16:04.275Z ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2021-07-05T08:16:04.361Z INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:78}] to localhost:27017
2021-07-05T08:16:04.920Z INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Failed [LB:DEAD]
2021-07-05T08:16:04.921Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2021-07-05T08:16:04.921Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.
2021-07-05T08:16:04.923Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2021-07-05T08:16:04.923Z INFO  [GracefulShutdownService] Running graceful shutdown for <2> shutdown hooks
2021-07-05T08:16:04.923Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2021-07-05T08:16:04.930Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2021-07-05T08:16:04.930Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.930Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2021-07-05T08:16:04.931Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.921Z ERROR [InputSetupService] Not starting any inputs because lifecycle is: Failed [LB:DEAD]
2021-07-05T08:16:04.931Z INFO  [GracefulShutdownService] Initiate shutdown for <BrowserDriverService>
2021-07-05T08:16:04.932Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2021-07-05T08:16:04.932Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.932Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].
2021-07-05T08:16:04.936Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.TrafficCounterCalculator] complete, took <0ms>.
2021-07-05T08:16:04.936Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical].
2021-07-05T08:16:04.936Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.936Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp].
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] complete, took <0ms>.
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ESVersionCheckPeriodical].
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ESVersionCheckPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread].
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] complete, took <0ms>.
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads].
2021-07-05T08:16:04.938Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] complete, took <0ms>.
2021-07-05T08:16:04.939Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob].
2021-07-05T08:16:04.939Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob] complete, took <0ms>.
2021-07-05T08:16:04.940Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp].
2021-07-05T08:16:04.932Z INFO  [GracefulShutdownService] Finished shutdown for <BrowserDriverService>, took 0 ms
2021-07-05T08:16:04.940Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp] complete, took <0ms>.
2021-07-05T08:16:04.940Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2021-07-05T08:16:04.940Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2021-07-05T08:16:04.941Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.license.LicenseManagerPeriodical].
2021-07-05T08:16:04.941Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.license.LicenseManagerPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.942Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.license.LicenseReportPeriodical].
2021-07-05T08:16:04.943Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.license.LicenseReportPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.943Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.report.scheduler.ReportPeriodical].
2021-07-05T08:16:04.943Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.report.scheduler.ReportPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.943Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical].
2021-07-05T08:16:04.943Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.949Z INFO  [GracefulShutdownService] Initiate shutdown for <JobWorkerPool>
2021-07-05T08:16:04.950Z INFO  [GracefulShutdownService] Finished shutdown for <JobWorkerPool>, took 0 ms
2021-07-05T08:16:04.950Z INFO  [LocalKafkaMessageQueueReader] Stopping.
2021-07-05T08:16:04.955Z INFO  [Buffers] Waiting until [INPUT, PROCESS, OUTPUT] buffers are empty.
2021-07-05T08:16:04.957Z INFO  [Buffers] All buffers are empty. Continuing.
2021-07-05T08:16:04.959Z INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2021-07-05T08:16:04.962Z INFO  [LogManager] Shutting down.
2021-07-05T08:16:04.964Z INFO  [LookupDataAdapterRefreshService] Stopping 0 jobs
2021-07-05T08:16:04.988Z INFO  [LogManager] Shutdown complete.
2021-07-05T08:16:04.993Z INFO  [ServiceManagerListener] Services are now stopped.
2021-07-05T08:16:04.993Z ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:773) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:585) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:316) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:249) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]
        Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
        Caused by: java.security.GeneralSecurityException: org.bouncycastle.pkcs.PKCSIOException: malformed data: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:88) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:351) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:171) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:152) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:829) ~[?:?]
        Caused by: org.bouncycastle.pkcs.PKCSIOException: malformed data: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.parseBytes(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:66) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.doBuildKeyStore(PemKeyStore.java:99) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:85) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:351) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:171) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:152) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:829) ~[?:?]
        Caused by: java.lang.IllegalArgumentException: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
                at org.bouncycastle.asn1.ASN1Sequence.getInstance(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.asn1.x509.AlgorithmIdentifier.getInstance(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.asn1.pkcs.EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.asn1.pkcs.EncryptedPrivateKeyInfo.getInstance(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.parseBytes(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:66) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.doBuildKeyStore(PemKeyStore.java:99) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:85) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:351) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:171) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:152) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:829) ~[?:?]
2021-07-05T08:16:04.997Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-07-05T08:16:04.998Z INFO  [Server] SIGNAL received. Shutting down.
2021-07-05T08:16:05.005Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-07-05T08:16:05.005Z INFO  [GracefulShutdown] Graceful shutdown initiated.
2021-07-05T08:16:05.006Z INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2021-07-05T08:16:09.009Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-07-05T08:16:09.010Z INFO  [GracefulShutdown] Goodbye.

Thanks in advance for your help.

I commented this out and it works.

Hello,
Glad you were able to fix it.

The http_tls_key_password is to unlock the private key used for securing the HTTP interface. (if key is encrypted)

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.