When I try to start the service there is no information going to the log file and the status is stuck in “activating”
So normally I would just check the log file for errors but since nothing is being logged im not sure what to do, I compared the config to my master node (which is working fine) and everything looks the same other then ips/uri/is_master
I deleted the log file and tried to start and no log files is being created… I restarted my server a few times but I think the fact the service is stuck in “activating” is leaving me without any logs to troubleshoot with but these were the last lines I have saved form the log file
LOG
2019-09-05T19:10:21.285Z INFO [GracefulShutdown] Graceful shutdown initiated.
2019-09-05T19:10:21.287Z INFO [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2019-09-05T19:10:25.355Z WARN [BufferSynchronizerService] Elasticsearch is unavailable. Not waiting to clear buffers and caches, as we have no healthy cluster.
2019-09-05T19:10:25.366Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2019-09-05T19:10:25.366Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2019-09-05T19:10:25.367Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2019-09-05T19:10:25.367Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2019-09-05T19:10:25.367Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2019-09-05T19:10:25.367Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2019-09-05T19:10:25.367Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].
2019-09-05T19:10:25.367Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.TrafficCounterCalculator] complete, took <0ms>.
2019-09-05T19:10:25.367Z INFO [GracefulShutdown] Goodbye.
2019-09-05T19:10:25.369Z INFO [JournalReader] Stopping.
2019-09-05T19:10:25.369Z INFO [JerseyService] Shutting down HTTP listener at <X.X.X.X:9000>
2019-09-05T19:10:25.385Z INFO [LogManager] Shutting down.
2019-09-05T19:10:25.404Z INFO [LookupDataAdapterRefreshService] Stopping 0 jobs
2019-09-05T19:10:25.430Z INFO [LogManager] Shutdown complete.
2019-09-05T19:10:25.445Z INFO [NetworkListener] Stopped listener bound to [X.X.X.X:9000]
SERVICE
service graylog-server status
● graylog-server.service - Graylog server
Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; v
Active: activating (auto-restart) (Result: exit-code) since Fri 2019-09-06
Docs: http://docs.graylog.org/
Process: 22714 ExecStart=/usr/share/graylog-server/bin/graylog-server (code
Main PID: 22714 (code=exited, status=1/FAILURE)
LOG config
<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.graylog2.log4j" shutdownHook="disable">
<Appenders>
<RollingFile name="rolling-file" fileName="/var/log/graylog-server/server.log" filePattern="/var/log/graylog-server/server.log.%i.gz">
<PatternLayout pattern="%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5p [%c{1}] %m%n"/>
<Policies>
<SizeBasedTriggeringPolicy size="50MB"/>
</Policies>
<DefaultRolloverStrategy max="10" fileIndex="min"/>
</RollingFile>
<!-- Internal Graylog log appender. Please do not disable. This makes internal log messages available via REST calls. -->
<Memory name="graylog-internal-logs" bufferSize="500"/>
</Appenders>
<Loggers>
<!-- Application Loggers -->
<Logger name="org.graylog2" level="info"/>
<Logger name="com.github.joschi.jadconfig" level="warn"/>
<!-- This emits a harmless warning for ActiveDirectory every time which we can't work around :( -->
<Logger name="org.apache.directory.api.ldap.model.message.BindRequestImpl" level="error"/>
<!-- Prevent DEBUG message about Lucene Expressions not found. -->
<Logger name="org.elasticsearch.script" level="warn"/>
<!-- Disable messages from the version check -->
<Logger name="org.graylog2.periodical.VersionCheckThread" level="off"/>
<!-- Silence chatty natty -->
<Logger name="com.joestelmach.natty.Parser" level="warn"/>
<!-- Silence Kafka log chatter -->
<Logger name="kafka.log.Log" level="warn"/>
<Logger name="kafka.log.OffsetIndex" level="warn"/>
<!-- Silence useless session validation messages -->
<Logger name="org.apache.shiro.session.mgt.AbstractValidatingSessionManager" level="warn"/>
<Root level="warn">
<AppenderRef ref="rolling-file"/>
<AppenderRef ref="graylog-internal-logs"/>
</Root>
</Loggers>
</Configuration>
config
is_master = false
node_id_file = /etc/graylog/server/node-id
password_secret = REDACTED
root_password_sha2 = REDACTED
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = REDACTED
http_publish_uri = REDACTED
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog-cert.pem
http_tls_key_file = /etc/ssl/certs/graylog-key.pem
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = REDACTED
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32