Web UI not working only when HTTPS is enabled, not listening on port 9000

Graylog Central (peer support)
1

Hi everyone,

I am trying to setup HTTPS on Graylog 4.1 - I am new to graylog.

I installed Graylog successfully and it is working: I can reach the HTTP Web UI and it is injesting logs fine. I have also configured the FQDN so I can successfully use graylog.mydomain.com
Now I want to enable enable HTTPS. As we have an internal CA, I generated the key, the .csr and signed it, so I now have the final .pem certificate.
When I change the server.conf file with the following parameters and reboot, even if I let Graylog alone for 1 hour (so that it has more than sufficient time to load the services), it looks like it is not listening at all in port 9000. If instead I disabled the HTTPS settings, when I reboot and I wait a couple of minutes it loads the Web UI again in HTTP without any issue.

netstat -l with only HTTP:

$ netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 graylog.mydomain:37109 0.0.0.0:*               LISTEN
tcp        0      0 localhost:domain        0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN
tcp        0      0 graylog.mydomain.com:27017 0.0.0.0:*               LISTEN
tcp6       0      0 graylog.mydomain:9200 [::]:*                  LISTEN
tcp6       0      0 localhost6.localdo:9200 [::]:*                  LISTEN
tcp6       0      0 graylog.mydomain:9300 [::]:*                  LISTEN
tcp6       0      0 localhost6.localdo:9300 [::]:*                  LISTEN
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN
tcp6       0      0 [::]:9000               [::]:*                  LISTEN
udp        0      0 localhost:domain        0.0.0.0:*
udp        0      0 graylog.mydomain.s:bootpc 0.0.0.0:*
raw6       0      0 [::]:ipv6-icmp          [::]:*                  7

Netstat when HTTPS is enabled

$ netstat -tpln
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:39023         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      -
tcp6       0      0 ::1:9200                :::*                    LISTEN      -
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      -
tcp6       0      0 ::1:9300                :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

server.conf

http_bind_address = 0.0.0.0:9000
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog.mydomain.com.pem
http_tls_key_file = /etc/ssl/certs/graylog.mydomain.com.key
http_publish_uri = https://192.168.8.156:9000/

I also tried to query and start the service manually, but this is what I get. So it seems that with those settings the services are simply not starting

$ systemctl status graylog-server.service
● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2021-07-02 12:34:30 UTC; 1s ago
     Docs: http://docs.graylog.org/
  Process: 2456 ExecStart=/usr/share/graylog-server/bin/graylog-server (code=exited, status=1/FAILURE)
 Main PID: 2456 (code=exited, status=1/FAILURE)

Thanks in advance!

2

Hi there,

What do you see in the logs?

3

@0d4y5

Just chimming in, I noticed you may be missing a configuration in your server file.
Here is mine

http_bind_address = 8.8.8.8:9000
http_publish_uri = https://graylog.domain.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = secret
4

Thanks @gsmith and @aaronsachs for your suggestions,

I set the enable cors, enabled the uri on the hostname, and enabled the tls key password.
By looking at the logs, I could see the following:

2021-07-03T20:14:11.911Z ERROR [CmdLineTool] Invalid configuration
com.github.joschi.jadconfig.ValidationException: Unreadable or missing HTTP private key: /etc/ssl/certs/graylog.mydomain.key
        at org.graylog2.configuration.HttpConfiguration.validateTlsConfig(HttpConfiguration.java:252) ~[graylog.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:100) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:380) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:207) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]

I modified the permissions of the key file so that it can be read, and now that error is fixed.
Unfortunately I get another error, but honestly I can’t figure out what is wrong here:

2021-07-05T08:15:58.836Z INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.1.0 [org.graylog.aws.AWSPlugin]
2021-07-05T08:15:58.841Z INFO  [CmdLineTool] Loaded plugin: Enterprise Integrations 4.1.0 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2021-07-05T08:15:58.843Z INFO  [CmdLineTool] Loaded plugin: Integrations 4.1.0 [org.graylog.integrations.IntegrationsPlugin]
2021-07-05T08:15:58.845Z INFO  [CmdLineTool] Loaded plugin: Collector 4.1.0 [org.graylog.plugins.collector.CollectorPlugin]
2021-07-05T08:15:58.847Z INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise 4.1.0 [org.graylog.plugins.enterprise.EnterprisePlugin]
2021-07-05T08:15:58.849Z INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES6 Support) 4.1.0 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es6.EnterpriseES6Plugin]
2021-07-05T08:15:58.850Z INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES7 Support) 4.1.0 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es7.EnterpriseES7Plugin]
2021-07-05T08:15:58.851Z INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.1.0 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-07-05T08:15:58.851Z INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.1.0+4eb2147 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-07-05T08:15:58.852Z INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.1.0+4eb2147 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-07-05T08:15:58.903Z INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-07-05T08:15:59.104Z INFO  [Version] HV000001: Hibernate Validator null
2021-07-05T08:16:01.464Z INFO  [InputBufferImpl] Message journal is enabled.
2021-07-05T08:16:01.481Z INFO  [NodeId] Node ID: 6bea7bfe-fe87-4144-8300-58f731aa0e4c
2021-07-05T08:16:01.603Z INFO  [LogManager] Loading logs.
2021-07-05T08:16:01.643Z INFO  [LogManager] Logs loading complete.
2021-07-05T08:16:01.645Z INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-07-05T08:16:01.666Z INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-07-05T08:16:01.700Z INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-07-05T08:16:01.724Z INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:74}] to localhost:27017
2021-07-05T08:16:01.731Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 14]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4407295}
2021-07-05T08:16:01.743Z INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:75}] to localhost:27017
2021-07-05T08:16:01.891Z INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-07-05T08:16:02.325Z INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.10.2
2021-07-05T08:16:03.034Z INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-07-05T08:16:03.094Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-07-05T08:16:03.112Z INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-07-05T08:16:03.136Z INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:76}] to localhost:27017
2021-07-05T08:16:03.155Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-07-05T08:16:03.178Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-07-05T08:16:03.198Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-07-05T08:16:03.217Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-07-05T08:16:03.959Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-07-05T08:16:03.962Z INFO  [ServerBootstrap] Graylog server 4.1.0+4eb2147 starting up
2021-07-05T08:16:03.962Z INFO  [ServerBootstrap] JRE: Ubuntu 11.0.11 on Linux 5.4.0-77-generic
2021-07-05T08:16:03.962Z INFO  [ServerBootstrap] Deployment: deb
2021-07-05T08:16:03.963Z INFO  [ServerBootstrap] OS: Ubuntu 18.04.5 LTS (bionic)
2021-07-05T08:16:03.963Z INFO  [ServerBootstrap] Arch: amd64
2021-07-05T08:16:04.015Z INFO  [PeriodicalsService] Starting 36 periodicals ...
2021-07-05T08:16:04.017Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-07-05T08:16:04.041Z INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:77}] to localhost:27017
2021-07-05T08:16:04.046Z INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-07-05T08:16:04.061Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-07-05T08:16:04.062Z INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-07-05T08:16:04.063Z INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-07-05T08:16:04.078Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-07-05T08:16:04.082Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-07-05T08:16:04.082Z INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-07-05T08:16:04.082Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-07-05T08:16:04.084Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-07-05T08:16:04.089Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-07-05T08:16:04.094Z INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-07-05T08:16:04.097Z INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-07-05T08:16:04.106Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-07-05T08:16:04.110Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-07-05T08:16:04.117Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-07-05T08:16:04.117Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-07-05T08:16:04.118Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-07-05T08:16:04.129Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-07-05T08:16:04.144Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-07-05T08:16:04.144Z INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-07-05T08:16:04.172Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2021-07-05T08:16:04.175Z INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-07-05T08:16:04.176Z INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-07-05T08:16:04.210Z INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-07-05T08:16:04.214Z INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-07-05T08:16:04.223Z INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-07-05T08:16:04.229Z INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-07-05T08:16:04.234Z INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-07-05T08:16:04.240Z INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-07-05T08:16:04.241Z INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-07-05T08:16:04.253Z INFO  [Periodicals] Starting [org.graylog.plugins.license.LicenseManagerPeriodical] periodical in [0s], polling every [300s].
2021-07-05T08:16:04.266Z INFO  [Periodicals] Starting [org.graylog.plugins.license.LicenseReportPeriodical] periodical in [300s], polling every [3600s].
2021-07-05T08:16:04.267Z INFO  [Periodicals] Starting [org.graylog.plugins.license.StagedLicenseInstallerPeriodical] periodical, running forever.
2021-07-05T08:16:04.274Z INFO  [Periodicals] Starting [org.graylog.plugins.auditlog.mongodb.MongoAuditLogPeriodical] periodical in [0s], polling every [3600s].
2021-07-05T08:16:04.275Z INFO  [Periodicals] Starting [org.graylog.plugins.report.scheduler.ReportPeriodical] periodical in [120s], polling every [60s].
2021-07-05T08:16:04.275Z INFO  [Periodicals] Starting [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical] periodical in [60s], polling every [180s].
2021-07-05T08:16:04.275Z ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2021-07-05T08:16:04.361Z INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:78}] to localhost:27017
2021-07-05T08:16:04.920Z INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Failed [LB:DEAD]
2021-07-05T08:16:04.921Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2021-07-05T08:16:04.921Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
2021-07-05T08:16:04.922Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.
2021-07-05T08:16:04.923Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2021-07-05T08:16:04.923Z INFO  [GracefulShutdownService] Running graceful shutdown for <2> shutdown hooks
2021-07-05T08:16:04.923Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2021-07-05T08:16:04.930Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2021-07-05T08:16:04.930Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.930Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2021-07-05T08:16:04.931Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.921Z ERROR [InputSetupService] Not starting any inputs because lifecycle is: Failed [LB:DEAD]
2021-07-05T08:16:04.931Z INFO  [GracefulShutdownService] Initiate shutdown for <BrowserDriverService>
2021-07-05T08:16:04.932Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2021-07-05T08:16:04.932Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.932Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].
2021-07-05T08:16:04.936Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.TrafficCounterCalculator] complete, took <0ms>.
2021-07-05T08:16:04.936Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical].
2021-07-05T08:16:04.936Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.936Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp].
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] complete, took <0ms>.
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ESVersionCheckPeriodical].
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ESVersionCheckPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread].
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] complete, took <0ms>.
2021-07-05T08:16:04.937Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads].
2021-07-05T08:16:04.938Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] complete, took <0ms>.
2021-07-05T08:16:04.939Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob].
2021-07-05T08:16:04.939Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob] complete, took <0ms>.
2021-07-05T08:16:04.940Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp].
2021-07-05T08:16:04.932Z INFO  [GracefulShutdownService] Finished shutdown for <BrowserDriverService>, took 0 ms
2021-07-05T08:16:04.940Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp] complete, took <0ms>.
2021-07-05T08:16:04.940Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2021-07-05T08:16:04.940Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2021-07-05T08:16:04.941Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.license.LicenseManagerPeriodical].
2021-07-05T08:16:04.941Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.license.LicenseManagerPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.942Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.license.LicenseReportPeriodical].
2021-07-05T08:16:04.943Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.license.LicenseReportPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.943Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.report.scheduler.ReportPeriodical].
2021-07-05T08:16:04.943Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.report.scheduler.ReportPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.943Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical].
2021-07-05T08:16:04.943Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical] complete, took <0ms>.
2021-07-05T08:16:04.949Z INFO  [GracefulShutdownService] Initiate shutdown for <JobWorkerPool>
2021-07-05T08:16:04.950Z INFO  [GracefulShutdownService] Finished shutdown for <JobWorkerPool>, took 0 ms
2021-07-05T08:16:04.950Z INFO  [LocalKafkaMessageQueueReader] Stopping.
2021-07-05T08:16:04.955Z INFO  [Buffers] Waiting until [INPUT, PROCESS, OUTPUT] buffers are empty.
2021-07-05T08:16:04.957Z INFO  [Buffers] All buffers are empty. Continuing.
2021-07-05T08:16:04.959Z INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2021-07-05T08:16:04.962Z INFO  [LogManager] Shutting down.
2021-07-05T08:16:04.964Z INFO  [LookupDataAdapterRefreshService] Stopping 0 jobs
2021-07-05T08:16:04.988Z INFO  [LogManager] Shutdown complete.
2021-07-05T08:16:04.993Z INFO  [ServiceManagerListener] Services are now stopped.
2021-07-05T08:16:04.993Z ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:773) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:585) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:316) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:249) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]
        Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
        Caused by: java.security.GeneralSecurityException: org.bouncycastle.pkcs.PKCSIOException: malformed data: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:88) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:351) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:171) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:152) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:829) ~[?:?]
        Caused by: org.bouncycastle.pkcs.PKCSIOException: malformed data: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.parseBytes(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:66) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.doBuildKeyStore(PemKeyStore.java:99) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:85) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:351) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:171) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:152) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:829) ~[?:?]
        Caused by: java.lang.IllegalArgumentException: unknown object in getInstance: org.bouncycastle.asn1.ASN1Integer
                at org.bouncycastle.asn1.ASN1Sequence.getInstance(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.asn1.x509.AlgorithmIdentifier.getInstance(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.asn1.pkcs.EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.asn1.pkcs.EncryptedPrivateKeyInfo.getInstance(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.parseBytes(Unknown Source) ~[graylog.jar:?]
                at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.<init>(Unknown Source) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:66) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.doBuildKeyStore(PemKeyStore.java:99) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:85) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:351) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:171) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:152) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:829) ~[?:?]
2021-07-05T08:16:04.997Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-07-05T08:16:04.998Z INFO  [Server] SIGNAL received. Shutting down.
2021-07-05T08:16:05.005Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-07-05T08:16:05.005Z INFO  [GracefulShutdown] Graceful shutdown initiated.
2021-07-05T08:16:05.006Z INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2021-07-05T08:16:09.009Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-07-05T08:16:09.010Z INFO  [GracefulShutdown] Goodbye.

Thanks in advance for your help.

5

I commented this out and it works.

1 Like
6

Hello,
Glad you were able to fix it.

The http_tls_key_password is to unlock the private key used for securing the HTTP interface. (if key is encrypted)

1 Like
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.