HTTPS what port is it listening on?

$ grep "^[^#;]" /etc/graylog/server/server.conf
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = SECRET
root_password_sha2 = SECRET2
root_timezone = EST
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 0.0.0.0:9000
http_publish_uri = https://192.168.3.2:9000/
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = PASS
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_disable_version_check = true
elasticsearch_max_number_of_indices = 20
versionchecks = false
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

Hmmmmm…but it works when you turn off TLS? Out of curiosity, is Elasticsearch running on the same node? Also, what’s your config look like for Elasticsearch?

Is there anything showing before the lines in /var/log/graylog/server.log? I’d be keen to see the lines surrounding the error message, too.

The normal HTTP server works just fine, the server is taking in logs and I use it daily.
This is a small server, so yes everything is one this one box. What do you want specifically in regards to elasticsearch? I don’t remember doing any extensive setup when I installed graylog.

Here is the total server.log from startup (with tls enabled) to running (without web interface):

2021-03-25T15:52:28.858-04:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 3.2.6 [org.graylog.aws.AWSPlugin]
2021-03-25T15:52:28.860-04:00 INFO  [CmdLineTool] Loaded plugin: Collector 3.2.6 [org.graylog.plugins.collector.CollectorPlugin]
2021-03-25T15:52:28.861-04:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 3.2.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-03-25T15:52:29.101-04:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/graylog-key.jks -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-03-25T15:52:29.256-04:00 INFO  [Version] HV000001: Hibernate Validator null
2021-03-25T15:52:31.164-04:00 INFO  [InputBufferImpl] Message journal is enabled.
2021-03-25T15:52:31.174-04:00 INFO  [NodeId] Node ID: 3e272ba8-ed29-40c1-b7db-9a2850d5b65e
2021-03-25T15:52:31.297-04:00 INFO  [LogManager] Loading logs.
2021-03-25T15:52:31.344-04:00 INFO  [LogManager] Logs loading complete.
2021-03-25T15:52:31.346-04:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-03-25T15:52:31.361-04:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-03-25T15:52:31.399-04:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-03-25T15:52:31.419-04:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:150}] to localhost:27017
2021-03-25T15:52:31.424-04:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 4]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2336762}
2021-03-25T15:52:31.433-04:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:151}] to localhost:27017
2021-03-25T15:52:31.567-04:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-03-25T15:52:31.772-04:00 INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2021-03-25T15:52:31.772-04:00 INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2021-03-25T15:52:31.827-04:00 INFO  [JestClientFactory] Using custom ObjectMapper instance
2021-03-25T15:52:31.827-04:00 INFO  [JestClientFactory] Node Discovery disabled...
2021-03-25T15:52:31.827-04:00 INFO  [JestClientFactory] Idle connection reaping disabled...
2021-03-25T15:52:32.215-04:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-03-25T15:52:32.418-04:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-03-25T15:52:32.422-04:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:152}] to localhost:27017
2021-03-25T15:52:32.927-04:00 INFO  [ServerBootstrap] Graylog server 3.2.6+e255fcc starting up
2021-03-25T15:52:32.929-04:00 INFO  [ServerBootstrap] JRE: Debian 11.0.9.1 on Linux 4.19.0-14-amd64
2021-03-25T15:52:32.929-04:00 INFO  [ServerBootstrap] Deployment: deb
2021-03-25T15:52:32.929-04:00 INFO  [ServerBootstrap] OS: Debian GNU/Linux 10 (buster) (debian)
2021-03-25T15:52:32.929-04:00 INFO  [ServerBootstrap] Arch: amd64
2021-03-25T15:52:32.950-04:00 INFO  [PeriodicalsService] Starting 30 periodicals ...
2021-03-25T15:52:32.953-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-03-25T15:52:32.967-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-03-25T15:52:32.968-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-03-25T15:52:32.968-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-03-25T15:52:32.968-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-03-25T15:52:32.969-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-03-25T15:52:32.969-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-03-25T15:52:32.973-04:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-03-25T15:52:32.975-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-03-25T15:52:32.976-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-03-25T15:52:32.978-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-03-25T15:52:32.981-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-03-25T15:52:32.983-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.VersionCheckThread] periodical. Not configured to run on this node.
2021-03-25T15:52:32.983-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-03-25T15:52:32.983-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-03-25T15:52:32.984-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-03-25T15:52:32.994-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-03-25T15:52:32.995-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-03-25T15:52:32.995-04:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:153}] to localhost:27017
2021-03-25T15:52:32.996-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-03-25T15:52:33.006-04:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:154}] to localhost:27017
2021-03-25T15:52:33.012-04:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:155}] to localhost:27017
2021-03-25T15:52:33.013-04:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:156}] to localhost:27017
2021-03-25T15:52:33.054-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-03-25T15:52:33.054-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-03-25T15:52:33.065-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@59fb6a4d] STARTING
2021-03-25T15:52:33.075-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@59fb6a4d] RUNNING
2021-03-25T15:52:33.076-04:00 INFO  [LookupDataAdapterRefreshService] Adding job for <geoip/6032ac73d48b0d27ac998574/@59fb6a4d> [interval=60000ms]
2021-03-25T15:52:33.076-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2021-03-25T15:52:33.114-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2021-03-25T15:52:33.119-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-03-25T15:52:33.120-04:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-03-25T15:52:33.121-04:00 INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-03-25T15:52:33.126-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-03-25T15:52:33.128-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-03-25T15:52:33.130-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-03-25T15:52:33.130-04:00 INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-03-25T15:52:33.131-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-03-25T15:52:33.149-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@9feb2aa] STARTING
2021-03-25T15:52:33.163-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@9feb2aa] RUNNING
2021-03-25T15:52:33.178-04:00 INFO  [LookupTableService] Starting lookup table geoip/6032acafd48b0d27ac9985ba [@df88404] using cache geoip/6032ac98d48b0d27ac99859e [@9feb2aa], data adapter geoip/6032ac73d48b0d27ac998574 [@59fb6a4d]
2021-03-25T15:52:33.445-04:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Failed [LB:DEAD]
2021-03-25T15:52:33.445-04:00 ERROR [InputSetupService] Not starting any inputs because lifecycle is: Failed [LB:DEAD]
2021-03-25T15:52:33.457-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2021-03-25T15:52:33.457-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2021-03-25T15:52:33.458-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2021-03-25T15:52:33.458-04:00 INFO  [GracefulShutdownService] Running graceful shutdown for <1> shutdown hooks
2021-03-25T15:52:33.458-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2021-03-25T15:52:33.458-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2021-03-25T15:52:33.458-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2021-03-25T15:52:33.459-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2021-03-25T15:52:33.459-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2021-03-25T15:52:33.459-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2021-03-25T15:52:33.459-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2021-03-25T15:52:33.459-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2021-03-25T15:52:33.459-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2021-03-25T15:52:33.459-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2021-03-25T15:52:33.459-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2021-03-25T15:52:33.459-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2021-03-25T15:52:33.459-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2021-03-25T15:52:33.460-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2021-03-25T15:52:33.460-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2021-03-25T15:52:33.460-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].
2021-03-25T15:52:33.460-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.TrafficCounterCalculator] complete, took <0ms>.
2021-03-25T15:52:33.460-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical].
2021-03-25T15:52:33.460-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] complete, took <0ms>.
2021-03-25T15:52:33.460-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp].
2021-03-25T15:52:33.460-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] complete, took <0ms>.
2021-03-25T15:52:33.460-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread].
2021-03-25T15:52:33.460-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] complete, took <0ms>.
2021-03-25T15:52:33.461-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads].
2021-03-25T15:52:33.461-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] complete, took <0ms>.
2021-03-25T15:52:33.461-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob].
2021-03-25T15:52:33.461-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob] complete, took <0ms>.
2021-03-25T15:52:33.461-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp].
2021-03-25T15:52:33.461-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp] complete, took <0ms>.
2021-03-25T15:52:33.461-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2021-03-25T15:52:33.461-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2021-03-25T15:52:33.468-04:00 INFO  [LogManager] Shutting down.
2021-03-25T15:52:33.469-04:00 INFO  [GracefulShutdownService] Initiate shutdown for <JobWorkerPool>
2021-03-25T15:52:33.470-04:00 INFO  [GracefulShutdownService] Finished shutdown for <JobWorkerPool>, took 0 ms
2021-03-25T15:52:33.471-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@9feb2aa] STOPPING, was RUNNING
2021-03-25T15:52:33.473-04:00 INFO  [LookupDataAdapterRefreshService] Removing job for <geoip/6032ac73d48b0d27ac998574/@59fb6a4d>
2021-03-25T15:52:33.488-04:00 INFO  [JournalReader] Stopping.
2021-03-25T15:52:33.490-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@59fb6a4d] STOPPING, was RUNNING
2021-03-25T15:52:33.496-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@59fb6a4d] TERMINATED, was STOPPING
2021-03-25T15:52:33.496-04:00 INFO  [Buffers] Waiting until all buffers are empty.
2021-03-25T15:52:33.497-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@9feb2aa] TERMINATED, was STOPPING
2021-03-25T15:52:33.498-04:00 INFO  [Buffers] All buffers are empty. Continuing.
2021-03-25T15:52:33.498-04:00 INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2021-03-25T15:52:33.501-04:00 INFO  [LookupDataAdapterRefreshService] Stopping 0 jobs
2021-03-25T15:52:33.504-04:00 INFO  [LogManager] Shutdown complete.
2021-03-25T15:53:03.475-04:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], PeriodicalsService [TERMINATED], UrlWhitelistService [TERMINATED], GracefulShutdownService [TERMINATED], LookupTableService [TERMINATED], OutputSetupService [TERMINATED], JournalReader [TERMINATED], EtagService [TERMINATED], StreamCacheService [TERMINATED], ConfigurationEtagService [TERMINATED], BufferSynchronizerService [TERMINATED], KafkaJournal [TERMINATED]], FAILED=[JerseyService [FAILED]]}
2021-03-25T15:53:03.476-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) ~[graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:50) ~[graylog.jar:?]
        Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
        Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
                at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122) ~[?:?]
                at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168) ~[?:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:70) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:834) ~[?:?]
2021-03-25T15:53:03.485-04:00 INFO  [Server] SIGNAL received. Shutting down.
2021-03-25T15:53:03.492-04:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2021-03-25T15:53:03.493-04:00 INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2021-03-25T15:53:07.497-04:00 INFO  [GracefulShutdown] Goodbye.

I also have the log from startup (without TLS) to running (functional HTTP web interface) if that would help.

Hmmmm…I’m kind of flummoxed at this point. In the interest of narrowing things down a bit and trying to rule out the certs as being a problem, have you tried looking at snagging a letsencrypt cert to see if that might do the trick? The only other option I think would be to open an issue on https://github.com/Graylog2/graylog-server to get our devs involved.

As far as I am aware, lets encrypt requires an internet connection?
My graylog server is not internet connected.

Any other way to verify the certificates? What is the worry here, wrong format?

I’m wondering if it might be an extensions thing in the cert, which you could examine using OpenSSL. I’m just thinking that since I’ve gone through the exercise of using LE and have verified that their certs work, that would rule out any format or cert extensions. And yes, while LE does require an internet connection, you can generate the certs on a laptop and just scp/rsync them over. Of course, getting LE set up might be a bigger PITA just to get certs and find out that the certs aren’t the issue.

What about just starting with the cert extensions first? You should be able to do that via:

openssl x509 -in <your cert here> -text

$ sudo openssl x509 -in graylog-certificate.pem -text

    X509v3 extensions:
        X509v3 Key Usage: 
            Key Encipherment, Data Encipherment
        X509v3 Extended Key Usage: 
            TLS Web Server Authentication
        X509v3 Subject Alternative Name: 
            IP Address:192.168.3.2, DNS:graylog.domain.com

I did some googling, the NoSuchAlgorithmException means that something is wrong with accessing the keystore and not the certificates, correct?

Where’d you run into that error?

In the server.log

Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available

Ahhhhh ok. I somehow missed that. Let me check.

Ok, yeah. Definitely getting some useful info here. Can you add -Djavax.net.debug=ssl,keymanager to your GRAYLOG_SERVER_JAVA_OPTS? I’m hopeful that will actually give us some useful info when you restart the process and it dies.

I don’t see any more information. Is the debug SSL info put somewhere else?

> 2021-03-25T17:26:51.485-04:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 3.2.6 [org.graylog.aws.AWSPlugin]
> 2021-03-25T17:26:51.487-04:00 INFO  [CmdLineTool] Loaded plugin: Collector 3.2.6 [org.graylog.plugins.collector.CollectorPlugin]
> 2021-03-25T17:26:51.487-04:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 3.2.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
> 2021-03-25T17:26:51.737-04:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/graylog-key.jks -Djavax.net.debug=ssl,keymanager -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
> 2021-03-25T17:26:51.892-04:00 INFO  [Version] HV000001: Hibernate Validator null
> 2021-03-25T17:26:54.000-04:00 INFO  [InputBufferImpl] Message journal is enabled.
> 2021-03-25T17:26:54.016-04:00 INFO  [NodeId] Node ID: 3e272ba8-ed29-40c1-b7db-9a2850d5b65e
> 2021-03-25T17:26:54.182-04:00 INFO  [LogManager] Loading logs.
> 2021-03-25T17:26:54.235-04:00 INFO  [LogManager] Logs loading complete.
> 2021-03-25T17:26:54.238-04:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
> 2021-03-25T17:26:54.256-04:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
> 2021-03-25T17:26:54.292-04:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
> 2021-03-25T17:26:54.312-04:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:186}] to localhost:27017
> 2021-03-25T17:26:54.316-04:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 4]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2348299}
> 2021-03-25T17:26:54.327-04:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:187}] to localhost:27017
> 2021-03-25T17:26:54.459-04:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
> 2021-03-25T17:26:54.720-04:00 INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
> 2021-03-25T17:26:54.721-04:00 INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
> 2021-03-25T17:26:54.773-04:00 INFO  [JestClientFactory] Using custom ObjectMapper instance
> 2021-03-25T17:26:54.773-04:00 INFO  [JestClientFactory] Node Discovery disabled...
> 2021-03-25T17:26:54.773-04:00 INFO  [JestClientFactory] Idle connection reaping disabled...
> 2021-03-25T17:26:55.194-04:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
> 2021-03-25T17:26:55.411-04:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
> 2021-03-25T17:26:55.449-04:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:188}] to localhost:27017
> 2021-03-25T17:26:56.007-04:00 INFO  [ServerBootstrap] Graylog server 3.2.6+e255fcc starting up
> 2021-03-25T17:26:56.007-04:00 INFO  [ServerBootstrap] JRE: Debian 11.0.9.1 on Linux 4.19.0-14-amd64
> 2021-03-25T17:26:56.007-04:00 INFO  [ServerBootstrap] Deployment: deb
> 2021-03-25T17:26:56.008-04:00 INFO  [ServerBootstrap] OS: Debian GNU/Linux 10 (buster) (debian)
> 2021-03-25T17:26:56.008-04:00 INFO  [ServerBootstrap] Arch: amd64
> 2021-03-25T17:26:56.037-04:00 INFO  [PeriodicalsService] Starting 30 periodicals ...
> 2021-03-25T17:26:56.037-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
> 2021-03-25T17:26:56.047-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
> 2021-03-25T17:26:56.049-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
> 2021-03-25T17:26:56.049-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
> 2021-03-25T17:26:56.049-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
> 2021-03-25T17:26:56.057-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
> 2021-03-25T17:26:56.057-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
> 2021-03-25T17:26:56.061-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
> 2021-03-25T17:26:56.062-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
> 2021-03-25T17:26:56.069-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
> 2021-03-25T17:26:56.071-04:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
> 2021-03-25T17:26:56.072-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
> 2021-03-25T17:26:56.080-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.VersionCheckThread] periodical. Not configured to run on this node.
> 2021-03-25T17:26:56.080-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
> 2021-03-25T17:26:56.081-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
> 2021-03-25T17:26:56.081-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
> 2021-03-25T17:26:56.086-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
> 2021-03-25T17:26:56.086-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
> 2021-03-25T17:26:56.088-04:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:189}] to localhost:27017
> 2021-03-25T17:26:56.091-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
> 2021-03-25T17:26:56.098-04:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:190}] to localhost:27017
> 2021-03-25T17:26:56.098-04:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:192}] to localhost:27017
> 2021-03-25T17:26:56.101-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
> 2021-03-25T17:26:56.101-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
> 2021-03-25T17:26:56.106-04:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:191}] to localhost:27017
> 2021-03-25T17:26:56.107-04:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:193}] to localhost:27017
> 2021-03-25T17:26:56.119-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
> 2021-03-25T17:26:56.141-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
> 2021-03-25T17:26:56.153-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
> 2021-03-25T17:26:56.158-04:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
> 2021-03-25T17:26:56.159-04:00 INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
> 2021-03-25T17:26:56.159-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
> 2021-03-25T17:26:56.160-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
> 2021-03-25T17:26:56.160-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
> 2021-03-25T17:26:56.161-04:00 INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
> 2021-03-25T17:26:56.161-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
> 2021-03-25T17:26:56.249-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@49f42058] STARTING
> 2021-03-25T17:26:56.310-04:00 INFO  [LookupDataAdapterRefreshService] Adding job for <geoip/6032ac73d48b0d27ac998574/@49f42058> [interval=60000ms]
> 2021-03-25T17:26:56.309-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@49f42058] RUNNING
> 2021-03-25T17:26:56.393-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@507e9ddb] STARTING
> 2021-03-25T17:26:56.406-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@507e9ddb] RUNNING
> 2021-03-25T17:26:56.414-04:00 INFO  [LookupTableService] Starting lookup table geoip/6032acafd48b0d27ac9985ba [@20d092b7] using cache geoip/6032ac98d48b0d27ac99859e [@507e9ddb], data adapter geoip/6032ac73d48b0d27ac998574 [@49f42058]
> 2021-03-25T17:26:56.579-04:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Failed [LB:DEAD]
> 2021-03-25T17:26:56.581-04:00 ERROR [InputSetupService] Not starting any inputs because lifecycle is: Failed [LB:DEAD]
> 2021-03-25T17:26:56.589-04:00 INFO  [GracefulShutdownService] Running graceful shutdown for <1> shutdown hooks
> 2021-03-25T17:26:56.590-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
> 2021-03-25T17:26:56.590-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
> 2021-03-25T17:26:56.590-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
> 2021-03-25T17:26:56.591-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
> 2021-03-25T17:26:56.592-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
> 2021-03-25T17:26:56.592-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
> 2021-03-25T17:26:56.592-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
> 2021-03-25T17:26:56.592-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
> 2021-03-25T17:26:56.592-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
> 2021-03-25T17:26:56.592-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
> 2021-03-25T17:26:56.593-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
> 2021-03-25T17:26:56.593-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
> 2021-03-25T17:26:56.593-04:00 INFO  [GracefulShutdownService] Initiate shutdown for <JobWorkerPool>
> 2021-03-25T17:26:56.593-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
> 2021-03-25T17:26:56.593-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
> 2021-03-25T17:26:56.593-04:00 INFO  [GracefulShutdownService] Finished shutdown for <JobWorkerPool>, took 0 ms
> 2021-03-25T17:26:56.593-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
> 2021-03-25T17:26:56.593-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
> 2021-03-25T17:26:56.593-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
> 2021-03-25T17:26:56.593-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
> 2021-03-25T17:26:56.594-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].
> 2021-03-25T17:26:56.594-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.TrafficCounterCalculator] complete, took <0ms>.
> 2021-03-25T17:26:56.594-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical].
> 2021-03-25T17:26:56.594-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] complete, took <0ms>.
> 2021-03-25T17:26:56.594-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp].
> 2021-03-25T17:26:56.594-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] complete, took <0ms>.
> 2021-03-25T17:26:56.594-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread].
> 2021-03-25T17:26:56.594-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] complete, took <0ms>.
> 2021-03-25T17:26:56.594-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads].
> 2021-03-25T17:26:56.595-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] complete, took <0ms>.
> 2021-03-25T17:26:56.595-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob].
> 2021-03-25T17:26:56.595-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob] complete, took <0ms>.
> 2021-03-25T17:26:56.595-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp].
> 2021-03-25T17:26:56.595-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp] complete, took <0ms>.
> 2021-03-25T17:26:56.595-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
> 2021-03-25T17:26:56.595-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
> 2021-03-25T17:26:56.602-04:00 INFO  [JournalReader] Stopping.
> 2021-03-25T17:26:56.606-04:00 INFO  [LogManager] Shutting down.
> 2021-03-25T17:26:56.622-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@507e9ddb] STOPPING, was RUNNING
> 2021-03-25T17:26:56.626-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@49f42058] STOPPING, was RUNNING
> 2021-03-25T17:26:56.640-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@507e9ddb] TERMINATED, was STOPPING
> 2021-03-25T17:26:56.640-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@49f42058] TERMINATED, was STOPPING
> 2021-03-25T17:26:56.644-04:00 INFO  [LookupDataAdapterRefreshService] Stopping 1 jobs
> 2021-03-25T17:26:56.644-04:00 INFO  [Buffers] Waiting until all buffers are empty.
> 2021-03-25T17:26:56.652-04:00 INFO  [Buffers] All buffers are empty. Continuing.
> 2021-03-25T17:26:56.653-04:00 INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
> 2021-03-25T17:26:56.658-04:00 INFO  [LogManager] Shutdown complete.
> 2021-03-25T17:27:26.637-04:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], GracefulShutdownService [TERMINATED], PeriodicalsService [TERMINATED], UrlWhitelistService [TERMINATED], JournalReader [TERMINATED], StreamCacheService [TERMINATED], OutputSetupService [TERMINATED], EtagService [TERMINATED], LookupTableService [TERMINATED], ConfigurationEtagService [TERMINATED], BufferSynchronizerService [TERMINATED], KafkaJournal [TERMINATED]], FAILED=[JerseyService [FAILED]]}
> 2021-03-25T17:27:26.638-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
> java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
>         at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
>         at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
>         at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
>         at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) ~[graylog.jar:?]
>         at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) ~[graylog.jar:?]
>         at org.graylog2.bootstrap.Main.main(Main.java:50) ~[graylog.jar:?]
>         Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
>         Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
>                 at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122) ~[?:?]
>                 at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168) ~[?:?]
>                 at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:70) ~[graylog.jar:?]
>                 at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
>                 at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]
>                 at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]
>                 at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]
>                 at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
>                 at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
>                 at java.lang.Thread.run(Thread.java:834) ~[?:?]
> 2021-03-25T17:27:26.646-04:00 INFO  [Server] SIGNAL received. Shutting down.
> 2021-03-25T17:27:26.654-04:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
> 2021-03-25T17:27:26.655-04:00 INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
> 2021-03-25T17:27:30.658-04:00 INFO  [GracefulShutdown] Goodbye.

Hello,
I have a question concerning the certificate you inserted into your keystore.
This is what caught my attention.

I noticed that you made cert called “graylog-certificate.pem” why did you use graylog.crt instead? The reason I ask this was your server.conf file was configured like this:

Also I wanted to ask did you give Grayog permission to the keystore? I didnt see the command from your post above.

chown graylog:graylog -R /path/to/keystore

Not sure if that the issue.

That may be the issue. I was just following the main graylog documenation and it had graylog-certificate.pem, and I wasnt paying a ton of attention to the openssl commands.
Looking at the contents, the graylog.crt is definitely the right one.

As for the file permissions, I already went through and checked them. For both the certificates/keys and keystore I added graylog user as the owner of all the files.

Update:
Still no go, I added graylog.crt to the server.conf and restarted. Looks like the same error:

2021-03-26T13:18:11.457-04:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 3.2.6 [org.graylog.aws.AWSPlugin]
2021-03-26T13:18:11.458-04:00 INFO  [CmdLineTool] Loaded plugin: Collector 3.2.6 [org.graylog.plugins.collector.CollectorPlugin]
2021-03-26T13:18:11.459-04:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 3.2.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-03-26T13:18:11.720-04:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/graylog-key.jks -Djavax.net.debug=ssl,keymanager -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-03-26T13:18:11.879-04:00 INFO  [Version] HV000001: Hibernate Validator null
2021-03-26T13:18:13.906-04:00 INFO  [InputBufferImpl] Message journal is enabled.
2021-03-26T13:18:13.921-04:00 INFO  [NodeId] Node ID: 3e272ba8-ed29-40c1-b7db-9a2850d5b65e
2021-03-26T13:18:14.086-04:00 INFO  [LogManager] Loading logs.
2021-03-26T13:18:14.150-04:00 INFO  [LogManager] Logs loading complete.
2021-03-26T13:18:14.153-04:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-03-26T13:18:14.170-04:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-03-26T13:18:14.205-04:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-03-26T13:18:14.227-04:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:201}] to localhost:27017
2021-03-26T13:18:14.234-04:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 4]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2364359}
2021-03-26T13:18:14.246-04:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:202}] to localhost:27017
2021-03-26T13:18:14.425-04:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-03-26T13:18:14.643-04:00 INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2021-03-26T13:18:14.644-04:00 INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2021-03-26T13:18:14.691-04:00 INFO  [JestClientFactory] Using custom ObjectMapper instance
2021-03-26T13:18:14.691-04:00 INFO  [JestClientFactory] Node Discovery disabled...
2021-03-26T13:18:14.692-04:00 INFO  [JestClientFactory] Idle connection reaping disabled...
2021-03-26T13:18:15.056-04:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-03-26T13:18:15.275-04:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-03-26T13:18:15.311-04:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:203}] to localhost:27017
2021-03-26T13:18:15.848-04:00 INFO  [ServerBootstrap] Graylog server 3.2.6+e255fcc starting up
2021-03-26T13:18:15.848-04:00 INFO  [ServerBootstrap] JRE: Debian 11.0.9.1 on Linux 4.19.0-14-amd64
2021-03-26T13:18:15.848-04:00 INFO  [ServerBootstrap] Deployment: deb
2021-03-26T13:18:15.848-04:00 INFO  [ServerBootstrap] OS: Debian GNU/Linux 10 (buster) (debian)
2021-03-26T13:18:15.849-04:00 INFO  [ServerBootstrap] Arch: amd64
2021-03-26T13:18:15.865-04:00 INFO  [PeriodicalsService] Starting 30 periodicals ...
2021-03-26T13:18:15.865-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-03-26T13:18:15.878-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-03-26T13:18:15.878-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-03-26T13:18:15.878-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-03-26T13:18:15.879-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-03-26T13:18:15.879-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-03-26T13:18:15.879-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-03-26T13:18:15.880-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-03-26T13:18:15.889-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-03-26T13:18:15.889-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-03-26T13:18:15.890-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-03-26T13:18:15.890-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.VersionCheckThread] periodical. Not configured to run on this node.
2021-03-26T13:18:15.890-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-03-26T13:18:15.891-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-03-26T13:18:15.891-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-03-26T13:18:15.899-04:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:206}] to localhost:27017
2021-03-26T13:18:15.902-04:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:204}] to localhost:27017
2021-03-26T13:18:15.903-04:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-03-26T13:18:15.905-04:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:205}] to localhost:27017
2021-03-26T13:18:15.934-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@6b1be7a9] STARTING
2021-03-26T13:18:15.944-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@6b1be7a9] RUNNING
2021-03-26T13:18:15.948-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-03-26T13:18:15.948-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-03-26T13:18:15.950-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-03-26T13:18:15.956-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-03-26T13:18:15.956-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-03-26T13:18:15.945-04:00 INFO  [LookupDataAdapterRefreshService] Adding job for <geoip/6032ac73d48b0d27ac998574/@6b1be7a9> [interval=60000ms]
2021-03-26T13:18:15.973-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2021-03-26T13:18:15.985-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2021-03-26T13:18:15.991-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-03-26T13:18:16.002-04:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-03-26T13:18:16.002-04:00 INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-03-26T13:18:16.008-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-03-26T13:18:16.012-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-03-26T13:18:16.013-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-03-26T13:18:16.013-04:00 INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-03-26T13:18:16.016-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-03-26T13:18:16.071-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@315c6df] STARTING
2021-03-26T13:18:16.080-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@315c6df] RUNNING
2021-03-26T13:18:16.087-04:00 INFO  [LookupTableService] Starting lookup table geoip/6032acafd48b0d27ac9985ba [@2c3965ea] using cache geoip/6032ac98d48b0d27ac99859e [@315c6df], data adapter geoip/6032ac73d48b0d27ac998574 [@6b1be7a9]
2021-03-26T13:18:16.353-04:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Failed [LB:DEAD]
2021-03-26T13:18:16.354-04:00 ERROR [InputSetupService] Not starting any inputs because lifecycle is: Failed [LB:DEAD]
2021-03-26T13:18:16.377-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2021-03-26T13:18:16.378-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2021-03-26T13:18:16.378-04:00 INFO  [GracefulShutdownService] Running graceful shutdown for <1> shutdown hooks
2021-03-26T13:18:16.378-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2021-03-26T13:18:16.378-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2021-03-26T13:18:16.378-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2021-03-26T13:18:16.378-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2021-03-26T13:18:16.378-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2021-03-26T13:18:16.378-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2021-03-26T13:18:16.378-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2021-03-26T13:18:16.379-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2021-03-26T13:18:16.379-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2021-03-26T13:18:16.379-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2021-03-26T13:18:16.379-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2021-03-26T13:18:16.379-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2021-03-26T13:18:16.379-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2021-03-26T13:18:16.380-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2021-03-26T13:18:16.380-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2021-03-26T13:18:16.380-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2021-03-26T13:18:16.380-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].
2021-03-26T13:18:16.380-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.TrafficCounterCalculator] complete, took <0ms>.
2021-03-26T13:18:16.380-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical].
2021-03-26T13:18:16.380-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] complete, took <0ms>.
2021-03-26T13:18:16.380-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp].
2021-03-26T13:18:16.380-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] complete, took <0ms>.
2021-03-26T13:18:16.381-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread].
2021-03-26T13:18:16.381-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] complete, took <0ms>.
2021-03-26T13:18:16.381-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads].
2021-03-26T13:18:16.381-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] complete, took <0ms>.
2021-03-26T13:18:16.381-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob].
2021-03-26T13:18:16.381-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob] complete, took <0ms>.
2021-03-26T13:18:16.381-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp].
2021-03-26T13:18:16.381-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp] complete, took <0ms>.
2021-03-26T13:18:16.381-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2021-03-26T13:18:16.382-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2021-03-26T13:18:16.389-04:00 INFO  [JournalReader] Stopping.
2021-03-26T13:18:16.390-04:00 INFO  [GracefulShutdownService] Initiate shutdown for <JobWorkerPool>
2021-03-26T13:18:16.390-04:00 INFO  [LogManager] Shutting down.
2021-03-26T13:18:16.390-04:00 INFO  [GracefulShutdownService] Finished shutdown for <JobWorkerPool>, took 0 ms
2021-03-26T13:18:16.416-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@315c6df] STOPPING, was RUNNING
2021-03-26T13:18:16.417-04:00 INFO  [LookupTableService] Cache geoip/6032ac98d48b0d27ac99859e [@315c6df] TERMINATED, was STOPPING
2021-03-26T13:18:16.421-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@6b1be7a9] STOPPING, was RUNNING
2021-03-26T13:18:16.436-04:00 INFO  [LookupTableService] Data Adapter geoip/6032ac73d48b0d27ac998574 [@6b1be7a9] TERMINATED, was STOPPING
2021-03-26T13:18:16.437-04:00 INFO  [LookupDataAdapterRefreshService] Stopping 1 jobs
2021-03-26T13:18:16.437-04:00 INFO  [LogManager] Shutdown complete.
2021-03-26T13:18:16.437-04:00 INFO  [Buffers] Waiting until all buffers are empty.
2021-03-26T13:18:16.438-04:00 INFO  [Buffers] All buffers are empty. Continuing.
2021-03-26T13:18:16.439-04:00 INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2021-03-26T13:18:46.418-04:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], PeriodicalsService [TERMINATED], JournalReader [TERMINATED], GracefulShutdownService [TERMINATED], UrlWhitelistService [TERMINATED], StreamCacheService [TERMINATED], OutputSetupService [TERMINATED], ConfigurationEtagService [TERMINATED], LookupTableService [TERMINATED], EtagService [TERMINATED], BufferSynchronizerService [TERMINATED], KafkaJournal [TERMINATED]], FAILED=[JerseyService [FAILED]]}
2021-03-26T13:18:46.419-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) ~[graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:50) ~[graylog.jar:?]
        Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
        Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
                at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122) ~[?:?]
                at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168) ~[?:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:70) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:834) ~[?:?]
2021-03-26T13:18:46.431-04:00 INFO  [Server] SIGNAL received. Shutting down.
2021-03-26T13:18:46.439-04:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2021-03-26T13:18:46.440-04:00 INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2021-03-26T13:18:50.444-04:00 INFO  [GracefulShutdown] Goodbye.

Heres all the file permissions for anyone wondering:

$ ll /etc/ssl/certs/graylog/
total 68
-rw-r--r-- 1 graylog graylog 2098 Mar 25 14:43 cert.pem
-rw-r----- 1 graylog graylog 2421 Mar 25 15:00 graylog-certificate.pem
-rw-r--r-- 1 graylog graylog 2098 Mar 25 14:45 graylog.crt
-rw-r--r-- 1 graylog graylog 6295 Mar 25 14:45 graylog.csr
-rw-r--r-- 1 graylog graylog 1826 Mar 25 15:04 graylog.jks
-rw-r----- 1 graylog graylog 3276 Mar 25 14:45 graylog.key
lrwxrwxrwx 1 graylog graylog   27 Nov 28 12:42 graylog-key.jks -> /etc/ssl/certs/java/cacerts
-rw-r----- 1 graylog graylog 3434 Mar 25 15:01 graylog-key.pem
-rw-r--r-- 1 graylog graylog 1802 Mar 25 14:50 graylog_keystore.jks
-rw-r----- 1 graylog graylog 3546 Mar 25 14:48 graylog-pkcs5.pem
-rw-r--r-- 1 graylog graylog 1802 Mar 25 14:59 keystore.p12
-rw-r----- 1 graylog graylog 4269 Mar 25 14:46 keystore.pfx
-rw-r--r-- 1 graylog graylog  582 Mar 25 14:43 openssl-graylog.cnf
-rw-r----- 1 graylog graylog 3272 Mar 25 14:43 pkcs5-plain.pem
-rw-r----- 1 graylog graylog 3434 Mar 25 14:44 pkcs8-encrypted.pem
-rw-r----- 1 graylog graylog 3272 Mar 25 14:43 pkcs8-plain.pem
$ ll /etc/ssl/certs/java/cacerts            
-rw-r--r-- 1 graylog graylog 161038 Mar 22 18:15 /etc/ssl/certs/java/cacerts

Shame on you :slight_smile:

Just a thought, did you check graylog-certificate.pem to see if that solved your problems.

Have you tried this?

https://docs.graylog.org/en/4.0/pages/configuration/https.html#adding-a-self-signed-certificate-to-the-jvm-trust-store

I was looking at this part from the link above. Maybe installed the wrong cert?

keytool -importcert -keystore /path/to/cacerts.jks -storepass changeit -alias graylog-self-signed -file cert.pem

I had this happen to myself on multiple occasions, which was due form improper Certificates and/or my Graylog server.conf file was not configured correct.

I found some post with the same problem as yourself, hope this wil help you.Some situation might not be exact as yours but the solutions might be what you need. To be honest this is starting to sound like you certs.

https://community.graylog.org/search?q=https%20

graylog-certificate.pem had information appended to the top of the cert, graylog.crt was just the cert itself in pem format, that may have been an issue, but not the one I am currently facing I suppose.

Perhaps I did put the wrong cert in, the commands I posted above are exactly what I did in terminal, so if its written there, then thats what I did, no more no less.
In fairness, I was just following the quicker tutorial here:

I am a little confused by the imports in that tutorial, It has graylog.jks as where the self signed is imported into, but then points the java server to graylog-key.jks which the tutorial never had you import anything into it??

Hello,
Actually I’m glad you posted what commands you did; it gives us more information to narrow down your problem.
You’re not the only one that has this problem and probably not the last.

As for certs go, it’s a trial an error. I know everyone’s environment is different and it took a while till I was able to create certs and use them in Graylog. It was a long learning curve for myself and takes time to do it right.

I agree it is confusing, In my opinion I would start off by using this link, specially as a guide of what you need to do. Use the other links like this link as reference. In this post there are several links on how to create certificates. Some may or may not work for you.
hope this helps

Well, couldn’t figure this one out.

I ended up using nginx as a reverse proxy for the encryption and just using http for graylog itself. Also had to change the http external publish url to the nginx listen block.
Not my first choice, but now its https traffic to the server :+1:

Glad you were able to solve it and so long as your data is secure.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.