Cant get HTTPS working on graylog

mkl · May 23, 2017, 11:05am

We have a problem with our setup getting it to work with https.
As soon as we configure this in the server.conf en enable tls the server isnt starting and the webpage isnt showed anymore.
When we uncomment the sections mentioning the tls / certificates it is working again.

SETUP:
Graylog 2.2.3 server with MongoDB running on Ubuntu 16.04 LTS.
Elasticsearch: 2.4.5, Ubuntu 16.04 LTS.

Network:
Graylog = 10.201.1.16
Elasticsearch = 10.201.1.21

CONFIG:
Graylog:

is_master = true
rest_listen_uri =  https://  10.201.1.16:9000/api/
rest_transport_uri =  https://  10.201.1.16:9000/api/
rest_enable_tls = true
rest_tls_cert_file = /etc/graylog/server/certificate/graylog-certificate.pem
rest_tls_key_file = /etc/graylog/server/certificate/graylog-key.pem
rest_tls_key_password = secret
web_listen_uri =  https://  10.201.1.16:9000/
web_enable_tls = true
web_tls_cert_file = /etc/graylog/server/certificate/graylog-certificate.pem
web_tls_key_file = /etc/graylog/server/certificate/graylog-key.pem
web_tls_key_password = secret
elasticsearch_cluster_name = graylog
elasticsearch_discovery_zen_ping_unicast_hosts = 10.201.1.21:9300
elasticsearch_network_host = 10.201.1.16

Elasticsearch:

cluster.name: graylog
network.host: 10.201.1.21
index.merge.scheduler.max_thread_count: 1
script.inline: false
script.indexed: false
script.file: false
transport.tcp.port: 9300

Logfile:

2017-05-23T12:57:00.289+02:00 INFO  [CmdLineTool] Loaded plugin: Single Sign-On (SSO) Authentication Provider 1.0.0 [org.graylog.plugins.auth.sso.SsoAuthPlugin]
2017-05-23T12:57:00.292+02:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.2.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-05-23T12:57:00.292+02:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.2.3 [org.graylog.plugins.collector.CollectorPlugin]
2017-05-23T12:57:00.293+02:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.2.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-05-23T12:57:00.294+02:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.2.3 [org.graylog.plugins.map.MapWidgetPlugin]
2017-05-23T12:57:00.304+02:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.2.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-05-23T12:57:00.305+02:00 INFO  [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.2.3 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-05-23T12:57:00.589+02:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms4g -Xmx4g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2017-05-23T12:57:00.806+02:00 INFO  [Version] HV000001: Hibernate Validator null
2017-05-23T12:57:03.136+02:00 INFO  [InputBufferImpl] Message journal is enabled.
2017-05-23T12:57:03.160+02:00 INFO  [NodeId] Node ID: dd72c5a1-f06e-415e-87aa-e4fab0f234d4
2017-05-23T12:57:03.383+02:00 INFO  [LogManager] Loading logs.
2017-05-23T12:57:03.513+02:00 INFO  [LogManager] Logs loading complete.
2017-05-23T12:57:03.513+02:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2017-05-23T12:57:03.534+02:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2017-05-23T12:57:03.559+02:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2017-05-23T12:57:03.619+02:00 INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-05-23T12:57:03.674+02:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:2811}] to localhost:27017
2017-05-23T12:57:03.676+02:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 13]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=537699}
2017-05-23T12:57:03.687+02:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:2812}] to localhost:27017
2017-05-23T12:57:04.005+02:00 INFO  [node] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] version[2.4.4], pid[30155], build[fcbb46d/2017-01-03T11:33:16Z]
2017-05-23T12:57:04.006+02:00 INFO  [node] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] initializing ...
2017-05-23T12:57:04.012+02:00 INFO  [plugins] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] modules [], plugins [graylog-monitor], sites []
2017-05-23T12:57:05.570+02:00 INFO  [node] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] initialized
2017-05-23T12:57:05.642+02:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-05-23T12:57:07.240+02:00 INFO  [RulesEngineProvider] No static rules file loaded.
2017-05-23T12:57:07.475+02:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-23T12:57:07.482+02:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-05-23T12:57:07.545+02:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-23T12:57:07.603+02:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-23T12:57:07.658+02:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-23T12:57:07.706+02:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-23T12:57:08.327+02:00 INFO  [ServerBootstrap] Graylog server 2.2.3+7adc951 starting up
2017-05-23T12:57:08.328+02:00 INFO  [ServerBootstrap] JRE: Oracle Corporation 1.8.0_131 on Linux 4.4.0-62-generic
2017-05-23T12:57:08.328+02:00 INFO  [ServerBootstrap] Deployment: deb
2017-05-23T12:57:08.328+02:00 INFO  [ServerBootstrap] OS: Ubuntu 16.04.2 LTS (xenial)
2017-05-23T12:57:08.329+02:00 INFO  [ServerBootstrap] Arch: amd64
2017-05-23T12:57:08.334+02:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-05-23T12:57:08.359+02:00 INFO  [PeriodicalsService] Starting 26 periodicals ...
2017-05-23T12:57:08.360+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2017-05-23T12:57:08.365+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2017-05-23T12:57:08.366+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2017-05-23T12:57:08.365+02:00 INFO  [node] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] starting ...
2017-05-23T12:57:08.368+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2017-05-23T12:57:08.370+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2017-05-23T12:57:08.371+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2017-05-23T12:57:08.372+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2017-05-23T12:57:08.374+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2017-05-23T12:57:08.375+02:00 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2017-05-23T12:57:08.376+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2017-05-23T12:57:08.376+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-05-23T12:57:08.377+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2017-05-23T12:57:08.377+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2017-05-23T12:57:08.378+02:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2017-05-23T12:57:08.381+02:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2017-05-23T12:57:08.388+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2017-05-23T12:57:08.391+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2017-05-23T12:57:08.393+02:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:2815}] to localhost:27017
2017-05-23T12:57:08.393+02:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:2813}] to localhost:27017
2017-05-23T12:57:08.394+02:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:2814}] to localhost:27017
2017-05-23T12:57:08.395+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-05-23T12:57:08.402+02:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:2817}] to localhost:27017
2017-05-23T12:57:08.403+02:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:2818}] to localhost:27017
2017-05-23T12:57:08.403+02:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:2816}] to localhost:27017
2017-05-23T12:57:08.500+02:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:2819}] to localhost:27017
2017-05-23T12:57:08.515+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2017-05-23T12:57:08.515+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-05-23T12:57:08.518+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-05-23T12:57:08.547+02:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-05-23T12:57:08.559+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical. Not configured to run on this node.
2017-05-23T12:57:08.560+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-05-23T12:57:08.567+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-05-23T12:57:08.568+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-05-23T12:57:08.572+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2017-05-23T12:57:08.576+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-05-23T12:57:08.614+02:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2017-05-23T12:57:08.703+02:00 INFO  [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable.
2017-05-23T12:57:08.734+02:00 INFO  [transport] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] publish_address {10.201.1.16:9350}, bound_addresses {10.201.1.16:9350}
2017-05-23T12:57:08.739+02:00 INFO  [discovery] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] graylog/s2PfwyN8TneFiaSt8qDPWQ
2017-05-23T12:57:08.908+02:00 ERROR [ServiceManager] Service JerseyService [FAILED] has failed in the STARTING state.
java.io.IOException: ObjectIdentifier() -- data isn't an object ID (tag = 48)
        at sun.security.util.ObjectIdentifier.<init>(ObjectIdentifier.java:253) ~[?:1.8.0_131]
        at sun.security.util.DerInputStream.getOID(DerInputStream.java:315) ~[?:1.8.0_131]
        at com.sun.crypto.provider.PBES2Parameters.engineInit(PBES2Parameters.java:267) ~[sunjce_provider.jar:1.8.0_112]
        at java.security.AlgorithmParameters.init(AlgorithmParameters.java:293) ~[?:1.8.0_131]
        at sun.security.x509.AlgorithmId.decodeParams(AlgorithmId.java:132) ~[?:1.8.0_131]
        at sun.security.x509.AlgorithmId.<init>(AlgorithmId.java:114) ~[?:1.8.0_131]
        at sun.security.x509.AlgorithmId.parse(AlgorithmId.java:372) ~[?:1.8.0_131]
        at javax.crypto.EncryptedPrivateKeyInfo.<init>(EncryptedPrivateKeyInfo.java:95) ~[?:1.8.0_121]
        at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:69) ~[graylog.jar:?]
        at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:385) ~[graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:206) ~[graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:139) ~[graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2017-05-23T12:57:08.933+02:00 ERROR [InputSetupService] Not starting any inputs because lifecycle is: Uninitialized [LB:DEAD]
2017-05-23T12:57:08.942+02:00 WARN  [BufferSynchronizerService] Elasticsearch is unavailable. Not waiting to clear buffers and caches, as we have no healthy cluster.
2017-05-23T12:57:08.944+02:00 INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2017-05-23T12:57:08.944+02:00 INFO  [node] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] stopping ...
2017-05-23T12:57:08.956+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.AlertScannerThread].
2017-05-23T12:57:08.957+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.AlertScannerThread] complete, took <0ms>.
2017-05-23T12:57:08.958+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2017-05-23T12:57:08.958+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2017-05-23T12:57:08.958+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2017-05-23T12:57:08.959+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2017-05-23T12:57:08.959+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2017-05-23T12:57:08.962+02:00 INFO  [LogManager] Shutting down.
2017-05-23T12:57:08.963+02:00 INFO  [JournalReader] Stopping.
2017-05-23T12:57:08.962+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2017-05-23T12:57:08.963+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2017-05-23T12:57:08.965+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2017-05-23T12:57:08.965+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2017-05-23T12:57:08.965+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2017-05-23T12:57:08.965+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
2017-05-23T12:57:08.965+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.
2017-05-23T12:57:08.965+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2017-05-23T12:57:08.967+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2017-05-23T12:57:08.967+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2017-05-23T12:57:08.967+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2017-05-23T12:57:08.968+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2017-05-23T12:57:08.970+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2017-05-23T12:57:08.970+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2017-05-23T12:57:08.971+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2017-05-23T12:57:08.971+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical].
2017-05-23T12:57:08.971+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] complete, took <0ms>.
2017-05-23T12:57:08.972+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical].
2017-05-23T12:57:08.972+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] complete, took <0ms>.
2017-05-23T12:57:08.972+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2017-05-23T12:57:08.972+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2017-05-23T12:57:08.987+02:00 INFO  [node] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] stopped
2017-05-23T12:57:08.987+02:00 INFO  [node] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] closing ...
2017-05-23T12:57:08.991+02:00 INFO  [node] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] closed
2017-05-23T12:57:09.000+02:00 INFO  [LogManager] Shutdown complete.
2017-05-23T12:57:11.753+02:00 WARN  [discovery] [graylog-dd72c5a1-f06e-415e-87aa-e4fab0f234d4] waited for 3s and no initial state was set by the discovery
2017-05-23T12:57:11.754+02:00 ERROR [ServiceManager] Service IndexerSetupService [FAILED] has failed in the STOPPING state.
java.lang.IllegalStateException: Can't move to started state when closed
        at org.elasticsearch.common.component.Lifecycle.canMoveToStarted(Lifecycle.java:114) ~[graylog.jar:?]
        at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:62) ~[graylog.jar:?]
        at org.elasticsearch.node.Node.start(Node.java:291) ~[graylog.jar:?]
        at org.graylog2.initializers.IndexerSetupService.startUp(IndexerSetupService.java:114) ~[graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2017-05-23T12:57:11.754+02:00 INFO  [ServiceManagerListener] Services are now stopped.
2017-05-23T12:57:11.755+02:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-05-23T12:57:11.754+02:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[IndexerSetupService [STARTING]], FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:742) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:555) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:304) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:147) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
2017-05-23T12:57:11.758+02:00 INFO  [Server] SIGNAL received. Shutting down.
2017-05-23T12:57:11.764+02:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2017-05-23T12:57:11.764+02:00 INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <0sec> for possible load balancers to recognize state change.
2017-05-23T12:57:11.764+02:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-05-23T12:57:12.766+02:00 INFO  [GracefulShutdown] Goodbye.

Hope you can help me, thx.

filemakers · May 23, 2017, 12:35pm

I assume you are aware that certificates are required for SSL to encrypt traffic?

mkl · May 23, 2017, 12:43pm

Yes, i made those pem files and uploaded them. We use our own Root-Server so its a internal certificate for our servers.

jan · May 23, 2017, 12:44pm

did you add your own CA to the Java KeyStore?

filemakers · May 23, 2017, 12:57pm

does the webui need to be listening on 443 ?

mkl · May 23, 2017, 12:57pm

No, I dont think so as i am not sure how to do that. Also thought that linking to the certificates in the configuration file was enough…? BTW we dont use a self-signed certificate.

jan · May 23, 2017, 1:31pm

Hej @mkl

you should take a look at the documentation

mkl · May 24, 2017, 6:08am

I tried this and imported the ca-certificate in the java keystore. Which is in a new keystore combined with the JAVA_OPTS when starting graylog it still fails. The other 2 for example the server-certificate and the key were pointed in the config to there location. Also added the server certificate in the keystore no success.

jan · May 24, 2017, 7:02am

did you check if your certificates are readable?


namei -l /etc/graylog/server/certificate/graylog-certificate.pem
namei -l /etc/graylog/server/certificate/graylog-key.pem

Additional I assume that you did not have the whitespaces in your real configuration that you have above …

mkl · May 24, 2017, 7:23am

root@plgra001:/etc/graylog/server/certificate# namei -l /etc/graylog/server/certificate/graylog-certificate.pem 
    f: /etc/graylog/server/certificate/graylog-certificate.pem
    drwxr-xr-x root root    /
    drwxr-xr-x root root    etc
    drwxr-xr-x root root    graylog
    drwxr-xr-x root root    server
    drwxrwxrwx root root    certificate
    -rw-r--r-- root graylog graylog-certificate.pem
root@plgra001:/etc/graylog/server/certificate# namei -l /etc/graylog/server/certificate/graylog-key.pem 
    f: /etc/graylog/server/certificate/graylog-key.pem
    drwxr-xr-x root root    /
    drwxr-xr-x root root    etc
    drwxr-xr-x root root    graylog
    drwxr-xr-x root root    server
    drwxrwxrwx root root    certificate
    -rw-r--r-- root graylog graylog-key.pem
root@plgra001:/etc/graylog/server/certificate#

Thats correct Jan.

As soon as I remove the tls, the graylog is functional.

jochen · May 24, 2017, 7:52am

This error looks similar to the following (closed) issue:
https://github.com/Graylog2/graylog2-server/issues/2752

Make sure that the private key and certificate are in the correct format.

See http://docs.graylog.org/en/2.2/pages/configuration/https.html for details about how to convert these files from various other formats.

mkl · May 24, 2017, 8:32am

I converted them to the right files as mentioned in the documentation:

$ openssl pkcs12 -in keystore.pfx -nokeys -out graylog-certificate.pem
$ openssl pkcs12 -in keystore.pfx -nocerts -out graylog-pkcs5.pem
$ openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem

So im not sure what im doing wrong

system · June 7, 2017, 8:32am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog does not start after enabling HTTPS support Graylog Central (peer support)	6	2086	March 23, 2020
Graylog https Setup Graylog Central (peer support)	20	11323	February 21, 2018
Input failed to start after https Graylog Central (peer support)	13	1360	March 26, 2020
Graylog 3.0.2 https Graylog Central (peer support) sidecar	11	4352	September 2, 2019
Graylog TLS not working Graylog Central (peer support)	16	1290	September 1, 2021

Cant get HTTPS working on graylog

Related topics