1. Describe your incident:
When I start the Graylog service, it goes into error, then tries to start again without ever stopping.
2. Describe your environment:
- OS Information:
Debian 11 - Package Version:
Graylog 5.1.4
Mongo 6.0.9
OpenSearch 2.5.0 - Service logs, configurations, and environment variables:
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret =
root_password_sha2 =
root_timezone = Europe/Paris
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = server.domain.lan:9000
elasticsearch_hosts = http://IP_OPENSEARCH:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
proxied_requests_thread_pool_size = 32
auto_restart_inputs = true
stream_aware_field_types=false
INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {default properties file=[cloud_inputs=on, scripting_api_preview=on, search_filter=on, preflight_web=off]}
INFO [CmdLineTool] Loaded plugin: Aggregation Count Alert Condition 2.2.0 [com.airbus-cyber-security.graylog.AggregationCountPlugin]
INFO [CmdLineTool] Loaded plugin: Alert Wizard 3.3.0 [com.airbus-cyber-security.graylog.AlertWizardPlugin]
INFO [CmdLineTool] Loaded plugin: Correlation Count Alert Condition 2.2.0 [com.airbus-cyber-security.graylog.CorrelationCountPlugin]
INFO [CmdLineTool] Loaded plugin: Logging Alert Notification 2.2.0 [com.airbus-cyber-security.graylog.LoggingAlertPlugin]
INFO [CmdLineTool] Loaded plugin: AWS plugins 5.1.4 [org.graylog.aws.AWSPlugin]
INFO [CmdLineTool] Loaded plugin: Integrations 5.1.4 [org.graylog.integrations.IntegrationsPlugin]
INFO [CmdLineTool] Loaded plugin: Collector 5.1.4 [org.graylog.plugins.collector.CollectorPlugin]
INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 5.1.4+6fa2de3 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 5.1.4+6fa2de3 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
INFO [CmdLineTool] Loaded plugin: OpenSearch 2 Support 5.1.4+6fa2de3 [org.graylog.storage.opensearch2.OpenSearch2Plugin]
INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.8.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.10.0-24-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.8+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@672b72ba]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.8.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.10.0-24-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.8+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@672b72ba]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=18865913}
INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
INFO [MongoDBPreflightCheck] Connected to MongoDB version 6.0.9
INFO [FilePersistedNodeIdProvider] Node ID: 4318a727-7cc4-4614-977e-80ead2452622
INFO [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <OpenSearch:2.5.0>
INFO [Version] HV000001: Hibernate Validator null
INFO [InputBufferImpl] Message journal is enabled.
INFO [FilePersistedNodeIdProvider] Node ID: 4318a727-7cc4-4614-977e-80ead2452622
INFO [LogManager] Loading logs.
INFO [LogManager] Logs loading complete.
INFO [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.8.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.10.0-24-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.8+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@672b72ba]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.8.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.10.0-24-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.8+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@672b72ba]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1299454}
INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
INFO [ElasticsearchVersionProvider] Elasticsearch cluster is running OpenSearch:2.5.0
INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
INFO [DbEntitiesScanner] 16 entities have been scanned and added to DB Entity Catalog, it took 848.6 ms
INFO [ServerBootstrap] Graylog server 5.1.4+6fa2de3 starting up
INFO [ServerBootstrap] JRE: Eclipse Adoptium 17.0.8 on Linux 5.10.0-24-amd64
INFO [ServerBootstrap] Deployment: deb
INFO [ServerBootstrap] OS: Debian GNU/Linux 11 (bullseye) (debian)
INFO [ServerBootstrap] Arch: amd64
INFO [ServerBootstrap] Running 54 migrations...
INFO [PeriodicalsService] Starting 26 periodicals ...
INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
INFO [Periodicals] Starting [org.graylog2.periodical.IndexBlockCheck] periodical in [0s], polling every [30s].
INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [1s].
INFO [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
INFO [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
INFO [Periodicals] Starting [org.graylog2.periodical.UserSessionTerminationPeriodical] periodical, running forever.
INFO [Periodicals] Starting [org.graylog2.telemetry.cluster.TelemetryClusterInfoPeriodical] periodical in [0s], polling every [540s].
INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
INFO [FailureSubmissionQueue] [FailureHandlerService#triggerShutdown] Total number of submitted batches: 0 (0 failures), total number of consumed batches: 0 (0 failures)
INFO [FailureHandlingService] The service's thread has been interrupted. The queue currently contains 0 failure batches.
INFO [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
INFO [GracefulShutdownService] Running graceful shutdown for <1> shutdown hooks
INFO [LogManager] Shutting down.
INFO [GracefulShutdownService] Initiate shutdown for <JobWorkerPool>
INFO [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
INFO [JobSchedulerService] Shutdown of job heartbeat executor
INFO [GracefulShutdownService] Finished shutdown for <JobWorkerPool>, took 1 ms
INFO [LookupDataAdapterRefreshService] Stopping 0 jobs
INFO [LogManager] Shutdown complete.
INFO [Periodicals] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
INFO [Periodicals] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
INFO [Periodicals] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
INFO [Periodicals] Shutting down periodical [org.graylog2.periodical.IndexBlockCheck].
INFO [Periodicals] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
INFO [Periodicals] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
INFO [Periodicals] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
INFO [Periodicals] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
INFO [Periodicals] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
INFO [Periodicals] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Failed [LB:DEAD]
INFO [Periodicals] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
INFO [Periodicals] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].
INFO [Periodicals] Shutting down periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical].
ERROR [InputSetupService] Not starting any inputs because lifecycle is: Failed [LB:DEAD]
INFO [Periodicals] Shutting down periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp].
INFO [Periodicals] Shutting down periodical [org.graylog2.periodical.ESVersionCheckPeriodical].
INFO [Periodicals] Shutting down periodical [org.graylog2.telemetry.cluster.TelemetryClusterInfoPeriodical].
INFO [Periodicals] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread].
INFO [Periodicals] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads].
INFO [Periodicals] Shutting down periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob].
INFO [Periodicals] Shutting down periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp].
INFO [Periodicals] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
INFO [LocalKafkaMessageQueueReader] Stopping.
INFO [Buffers] Waiting until [INPUT, PROCESS, OUTPUT] buffers are empty.
INFO [Buffers] All buffers are empty. Continuing.
INFO [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
INFO [FailureHandlingService] Shutting down the service. Processed 0 remaining failure batches.
INFO [FailureSubmissionQueue] [FailureHandlerService#shutDown] Total number of submitted batches: 0 (0 failures), total number of consumed batches: 0 (0 failures)
INFO [ServiceManagerListener] Services are now stopped.
ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[PeriodicalsService [STARTING]], FAILED=[JerseyService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:769) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:581) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:295) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:321) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:323) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:259) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]
Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
Caused by: java.lang.NoClassDefFoundError: org/graylog2/alerts/AlertService
at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:?]
at java.lang.Class.privateGetDeclaredConstructors(Unknown Source) ~[?:?]
at java.lang.Class.getConstructors(Unknown Source) ~[?:?]
at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.<init>(MethodHandler.java:241) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.MethodHandler.create(MethodHandler.java:131) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethod$Builder.createInvocable(ResourceMethod.java:536) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethod$Builder.build(ResourceMethod.java:522) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.Resource$Builder.processMethodBuilders(Resource.java:647) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.Resource$Builder.buildResourceData(Resource.java:583) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.Resource$Builder.build(Resource.java:639) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.lambda$prefixResources$1(JerseyService.java:228) ~[graylog.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.graylog2.shared.initializers.JerseyService.prefixResources(JerseyService.java:230) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.lambda$prefixPluginResources$0(JerseyService.java:210) ~[graylog.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.graylog2.shared.initializers.JerseyService.prefixPluginResources(JerseyService.java:212) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:173) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:157) ~[graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:121) ~[graylog.jar:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.lang.ClassNotFoundException: org.graylog2.alerts.AlertService
at java.net.URLClassLoader.findClass(Unknown Source) ~[?:?]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
at java.net.FactoryURLClassLoader.loadClass(Unknown Source) ~[?:?]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:?]
at java.lang.Class.privateGetDeclaredConstructors(Unknown Source) ~[?:?]
at java.lang.Class.getConstructors(Unknown Source) ~[?:?]
at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.<init>(MethodHandler.java:241) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.MethodHandler.create(MethodHandler.java:131) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethod$Builder.createInvocable(ResourceMethod.java:536) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethod$Builder.build(ResourceMethod.java:522) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.Resource$Builder.processMethodBuilders(Resource.java:647) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.Resource$Builder.buildResourceData(Resource.java:583) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.Resource$Builder.build(Resource.java:639) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.lambda$prefixResources$1(JerseyService.java:228) ~[graylog.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.graylog2.shared.initializers.JerseyService.prefixResources(JerseyService.java:230) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.lambda$prefixPluginResources$0(JerseyService.java:210) ~[graylog.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.graylog2.shared.initializers.JerseyService.prefixPluginResources(JerseyService.java:212) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:173) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:157) ~[graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:121) ~[graylog.jar:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
INFO [Server] SIGNAL received. Shutting down.
INFO [GracefulShutdown] Graceful shutdown initiated.
INFO [GracefulShutdown] Node status: [Override lb:DEAD [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
INFO [GracefulShutdown] Goodbye.
3. What steps have you already taken to try and solve the problem?
Upgrade Opensearch 1.3.2 to 2.5.0 => log OK
Up^grade Mongod 4.2 to 4.4 to 5.0.22 to 6.0.9 =>log OK
Upgrade Graylog latest v4 to v5.1.4
When I start the Graylog service, it goes into error, then tries to start again without ever stopping.
4. How can the community help?
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]