Graylog service failed (KafkaJournal error)


(Carlos) #1

Hello

The graylog-server.service cant run…

Message: systemctl status graylog-server.service:

● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Qua 2018-03-14 11:05:27 -03; 84ms ago
     Docs: http://docs.graylog.org/
  Process: 17823 ExecStart=/usr/share/graylog-server/bin/graylog-server (code=exited, status=254)
 Main PID: 17823 (code=exited, status=254)

Mar 14 11:05:27 graylog systemd[1]: graylog-server.service: main process exited, code=exited, status=254/n/a
Mar 14 11:05:27 graylog systemd[1]: Unit graylog-server.service entered failed state.
Mar 14 11:05:27 graylog systemd[1]: graylog-server.service failed.

LOGs: /var/log/graylog-server/server.log

2018-03-14T11:11:04.883-03:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 2.4.3 [org.graylog.aws.plugin.AWSPlugin]
2018-03-14T11:11:04.891-03:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.4.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-03-14T11:11:04.893-03:00 INFO  [CmdLineTool] Loaded plugin: CEF Input 2.4.3 [org.graylog.plugins.cef.CEFInputPlugin]
2018-03-14T11:11:04.891-03:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.4.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-03-14T11:11:04.893-03:00 INFO  [CmdLineTool] Loaded plugin: CEF Input 2.4.3 [org.graylog.plugins.cef.CEFInputPlugin]
2018-03-14T11:11:04.894-03:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.4.3 [org.graylog.plugins.collector.CollectorPlugin]
2018-03-14T11:11:04.896-03:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.4.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationP$
2018-03-14T11:11:04.897-03:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.4.3 [org.graylog.plugins.map.MapWidgetPlugin]
2018-03-14T11:11:04.899-03:00 INFO  [CmdLineTool] Loaded plugin: NetFlow Plugin 2.4.3 [org.graylog.plugins.netflow.NetFlowPlugin]
2018-03-14T11:11:04.914-03:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.4.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-03-14T11:11:04.915-03:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.4.3 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2018-03-14T11:11:05.502-03:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMT$
2018-03-14T11:11:05.885-03:00 INFO  [Version] HV000001: Hibernate Validator 5.1.3.Final
2018-03-14T11:11:10.384-03:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-03-14T11:11:10.424-03:00 INFO  [NodeId] Node ID: d1a5987d-f5f9-46a2-b3c3-2b90b6f5e43e
2018-03-14T11:11:10.668-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:10.675-03:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-03-14T11:11:10.680-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:10.715-03:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=$
2018-03-14T11:11:10.796-03:00 INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type$
2018-03-14T11:11:10.883-03:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:615}] to localhost:27017
2018-03-14T11:11:10.891-03:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALO$
2018-03-14T11:11:10.912-03:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:616}] to localhost:27017
2018-03-14T11:11:11.456-03:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-03-14T11:11:11.457-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:11.638-03:00 INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://localhost:9200]
2018-03-14T11:11:11.639-03:00 INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2018-03-14T11:11:11.773-03:00 INFO  [JestClientFactory] Using custom ObjectMapper instance
2018-03-14T11:11:11.774-03:00 INFO  [JestClientFactory] Node Discovery disabled...
2018-03-14T11:11:11.774-03:00 INFO  [JestClientFactory] Idle connection reaping disabled...
2018-03-14T11:11:11.788-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.001-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.002-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.033-03:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:12.035-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.038-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.288-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.312-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.327-03:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:12.328-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.349-03:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:12.351-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.352-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.389-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.395-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.415-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.424-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.457-03:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:12.458-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.533-03:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:12.534-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.536-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.548-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.552-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.558-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.564-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.570-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.591-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.601-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.618-03:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:12.620-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.638-03:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:12.639-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.650-03:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:12.651-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.652-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.674-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.683-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.697-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.707-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.725-03:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-03-14T11:11:12.727-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.730-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.738-03:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:12.739-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.747-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.758-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.767-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.785-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.794-03:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-03-14T11:11:12.795-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.806-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.827-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.844-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.851-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.858-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.870-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.880-03:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-03-14T11:11:12.882-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.895-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.911-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:12.988-03:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-03-14T11:11:12.989-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:13.042-03:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-03-14T11:11:13.043-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.559-03:00 INFO  [RulesEngineProvider] No static rules file loaded.
2018-03-14T11:11:15.603-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.610-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.615-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.620-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.624-03:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:15.625-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.626-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.632-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.634-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.641-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.646-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.659-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.666-03:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:15.667-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.669-03:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:15.669-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.670-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.679-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.684-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.691-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.698-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.705-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.715-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.728-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.742-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.746-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.748-03:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:15.748-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.749-03:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-03-14T11:11:15.750-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.751-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.765-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.768-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.772-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.779-03:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-03-14T11:11:15.780-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.783-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.787-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.792-03:00 ERROR [KafkaJournal] Cannot create journal directory at /data/journal, please check the permissions
2018-03-14T11:11:15.798-03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Tried proxying org.graylog2.outputs.OutputRegistry to support a circula$
2018-03-14T11:11:15.798-03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Tried proxying org.graylog2.outputs.OutputRegistry to support a circula$
2018-03-14T11:11:15.798-03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Tried proxying org.graylog2.outputs.OutputRegistry to support a circula$
2018-03-14T11:11:15.798-03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Tried proxying org.graylog2.outputs.OutputRegistry to support a circula$
2018-03-14T11:11:15.799-03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Tried proxying org.graylog2.outputs.OutputRegistry to support a circula$
2018-03-14T11:11:15.799-03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Tried proxying org.graylog2.outputs.OutputRegistry to support a circula$
2018-03-14T11:11:15.799-03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Tried proxying org.graylog2.outputs.OutputRegistry to support a circula$
2018-03-14T11:11:15.799-03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Tried proxying org.graylog2.outputs.OutputRegistry to support a circula$
2018-03-14T11:11:15.799-03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Tried proxying org.graylog2.outputs.OutputRegistry to support a circula$
2018-03-14T11:11:15.801-03:00 ERROR [CmdLineTool]

################################################################################

ERROR: Unable to access file /data/journal: Could not create journal directory.

Need help?

* Official documentation: http://docs.graylog.org/
* Community support: https://www.graylog.org/community-support/
* Commercial support: https://www.graylog.com/support/

Terminating. :(

################################################################################

Any ideias?

thanks


(Jochen) #2

Yes, make sure that the system user running Graylog has sufficient permissions to write into the directory you’ve configured in the message_journal_dir setting.


(Carlos) #3

Great! Thank you @jochen


(system) #4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.