Graylog service wont start or log to server.log

When I try to start the service there is no information going to the log file and the status is stuck in “activating”

So normally I would just check the log file for errors but since nothing is being logged im not sure what to do, I compared the config to my master node (which is working fine) and everything looks the same other then ips/uri/is_master

I deleted the log file and tried to start and no log files is being created… I restarted my server a few times but I think the fact the service is stuck in “activating” is leaving me without any logs to troubleshoot with but these were the last lines I have saved form the log file

LOG

2019-09-05T19:10:21.285Z INFO  [GracefulShutdown] Graceful shutdown initiated.
2019-09-05T19:10:21.287Z INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2019-09-05T19:10:25.355Z WARN  [BufferSynchronizerService] Elasticsearch is unavailable. Not waiting to clear buffers and caches, as we have no healthy cluster.
2019-09-05T19:10:25.366Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2019-09-05T19:10:25.366Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2019-09-05T19:10:25.367Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2019-09-05T19:10:25.367Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2019-09-05T19:10:25.367Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2019-09-05T19:10:25.367Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2019-09-05T19:10:25.367Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].
2019-09-05T19:10:25.367Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.TrafficCounterCalculator] complete, took <0ms>.
2019-09-05T19:10:25.367Z INFO  [GracefulShutdown] Goodbye.
2019-09-05T19:10:25.369Z INFO  [JournalReader] Stopping.
2019-09-05T19:10:25.369Z INFO  [JerseyService] Shutting down HTTP listener at <X.X.X.X:9000>
2019-09-05T19:10:25.385Z INFO  [LogManager] Shutting down.
2019-09-05T19:10:25.404Z INFO  [LookupDataAdapterRefreshService] Stopping 0 jobs
2019-09-05T19:10:25.430Z INFO  [LogManager] Shutdown complete.
2019-09-05T19:10:25.445Z INFO  [NetworkListener] Stopped listener bound to [X.X.X.X:9000]

SERVICE

service graylog-server status
● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; v
   Active: activating (auto-restart) (Result: exit-code) since Fri 2019-09-06
     Docs: http://docs.graylog.org/
  Process: 22714 ExecStart=/usr/share/graylog-server/bin/graylog-server (code
 Main PID: 22714 (code=exited, status=1/FAILURE)

LOG config

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.graylog2.log4j" shutdownHook="disable">
    <Appenders>
        <RollingFile name="rolling-file" fileName="/var/log/graylog-server/server.log" filePattern="/var/log/graylog-server/server.log.%i.gz">
            <PatternLayout pattern="%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5p [%c{1}] %m%n"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="50MB"/>
            </Policies>
            <DefaultRolloverStrategy max="10" fileIndex="min"/>
        </RollingFile>

        <!-- Internal Graylog log appender. Please do not disable. This makes internal log messages available via REST calls. -->
        <Memory name="graylog-internal-logs" bufferSize="500"/>
    </Appenders>
    <Loggers>
        <!-- Application Loggers -->
        <Logger name="org.graylog2" level="info"/>
        <Logger name="com.github.joschi.jadconfig" level="warn"/>
        <!-- This emits a harmless warning for ActiveDirectory every time which we can't work around :( -->
        <Logger name="org.apache.directory.api.ldap.model.message.BindRequestImpl" level="error"/>
        <!-- Prevent DEBUG message about Lucene Expressions not found. -->
        <Logger name="org.elasticsearch.script" level="warn"/>
        <!-- Disable messages from the version check -->
        <Logger name="org.graylog2.periodical.VersionCheckThread" level="off"/>
        <!-- Silence chatty natty -->
        <Logger name="com.joestelmach.natty.Parser" level="warn"/>
        <!-- Silence Kafka log chatter -->
        <Logger name="kafka.log.Log" level="warn"/>
        <Logger name="kafka.log.OffsetIndex" level="warn"/>
        <!-- Silence useless session validation messages -->
        <Logger name="org.apache.shiro.session.mgt.AbstractValidatingSessionManager" level="warn"/>
        <Root level="warn">
            <AppenderRef ref="rolling-file"/>
            <AppenderRef ref="graylog-internal-logs"/>
        </Root>
    </Loggers>
</Configuration>

config

is_master = false
node_id_file = /etc/graylog/server/node-id
password_secret = REDACTED
root_password_sha2 = REDACTED
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = REDACTED
http_publish_uri = REDACTED
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog-cert.pem
http_tls_key_file = /etc/ssl/certs/graylog-key.pem
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = REDACTED
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

Resolved my issue, I had a typo in the JVM settings

Sep 06 19:21:12 graylog systemd[1]: graylog-server.service: Scheduled restart job, restart counter is at 3.
Sep 06 19:21:12 graylog systemd[1]: Stopped Graylog server.
Sep 06 19:21:12 graylog systemd[1]: Started Graylog server.
Sep 06 19:21:12 graylog graylog-server[26158]: Unrecognized VM option ‘OmitStackTraceInFastThrowi’
Sep 06 19:21:12 graylog graylog-server[26158]: Did you mean ‘(+/-)OmitStackTraceInFastThrow’?
Sep 06 19:21:12 graylog graylog-server[26158]: Error: Could not create the Java Virtual Machine.
Sep 06 19:21:12 graylog graylog-server[26158]: Error: A fatal exception has occurred. Program will exit.
Sep 06 19:21:12 graylog systemd[1]: graylog-server.service: Main process exited, code=exited, status=1/FAILURE
Sep 06 19:21:12 graylog systemd[1]: graylog-server.service: Failed with result ‘exit-code’.
Sep 06 19:21:14 graylog systemd[1]: Stopped Graylog server.
root@graylog:/home/dshasho#

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.