Graylog is restarting...

dpcloud · 2018-03-23 10:37:48 UTC

Hi,

we have configured graylog server but after few days it is continuously showing that Graylog is restarting…

please provide resolution for the same and also note the status of running services…

run: elasticsearch: (pid 14963) 2s; run: log: (pid 789) 56868s
run: etcd: (pid 810) 56868s; run: log: (pid 794) 56868s
run: graylog-server: (pid 19135) 11958s; run: log: (pid 787) 56868s
down: mongodb: 1s, normally up, want up; run: log: (pid 786) 56868s
run: nginx: (pid 793) 56868s; run: log: (pid 788) 56868s

jochen · 2018-03-23 10:38:19 UTC

What’s in the logs of your Graylog node?
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html#omnibus-package

dpcloud · 2018-03-26 07:34:14 UTC

Hi,

below is the logs…

2018-03-25_21:36:09.29543       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
2018-03-25_21:36:09.29646       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
2018-03-25_21:36:09.29775       at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
2018-03-25_21:36:09.30009       at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
2018-03-25_21:36:09.30154       ... 11 more
2018-03-25_21:36:09.30255 WARN  [NodeChecker] Removing host http://192.168.0.72:9200
2018-03-25_21:36:09.30524 INFO  [NodeChecker] Discovered server pool is now:
2018-03-25_21:36:09.81118 INFO  [cluster] No server chosen by WritableServerSelector from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out
2018-03-25_21:36:39.13958 ERROR [NodeChecker] Connect exception executing NodesInfo!
2018-03-25_21:36:39.14047 io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://192.168.0.72:9200
2018-03-25_21:36:39.14201       at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?]
2018-03-25_21:36:39.14291       at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:64) ~[graylog.jar:?]
2018-03-25_21:36:39.14459       at io.searchbox.client.config.discovery.NodeChecker.runOneIteration(NodeChecker.java:65) [graylog.jar:?]
2018-03-25_21:36:39.14691       at com.google.common.util.concurrent.AbstractScheduledService$ServiceDelegate$Task.run(AbstractScheduledService.java:188) [graylog.jar:?]
2018-03-25_21:36:39.14760       at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
2018-03-25_21:36:39.14920       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_151]
2018-03-25_21:36:39.15258       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_151]
2018-03-25_21:36:39.15354       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_151]
2018-03-25_21:36:39.15528       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_151]
2018-03-25_21:36:39.15893       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
2018-03-25_21:36:39.16022       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
2018-03-25_21:36:39.16248       at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
2018-03-25_21:36:39.16433 Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.0.72:9200 [/192.168.0.72] failed: Connection refused (Connection refused)
2018-03-25_21:36:39.16628       at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[graylog.jar:?]
2018-03-25_21:36:39.16710       at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) ~[graylog.jar:?]
2018-03-25_21:36:39.16858       at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
2018-03-25_21:36:39.17044       at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
2018-03-25_21:36:39.17259       at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
2018-03-25_21:36:39.17328       at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[graylog.jar:?]
2018-03-25_21:36:39.17640       at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
2018-03-25_21:36:39.18763       at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
2018-03-25_21:36:39.18856       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
2018-03-25_21:36:39.19129       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
2018-03-25_21:36:39.20289       at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
2018-03-25_21:36:39.20442       at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
2018-03-25_21:36:39.20563       ... 11 more
2018-03-25_21:36:39.20785 Caused by: java.net.ConnectException: Connection refused (Connection refused)
2018-03-25_21:36:39.21027       at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_151]
2018-03-25_21:36:39.21288       at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_151]

jan · 2018-03-26 07:49:57 UTC

it looks like Graylog is not able to connect to elasticsearch.

Did you checked if elasticsearch is running? Did you checked the elasticsearch logfile?

with kind regards
Jan

dpcloud · 2018-03-27 12:40:41 UTC

Yes i checked the elasticsearch log file but could not identify the issue…

pls check the below logs…

2018-03-27_17:47:01.74461 2018-03-27_17:47:08.48828 2018-03-27_17:47:08.49312 2018-03-27_17:47:08.50850 2018-03-27_17:47:08.51479 2018-03-27_17:47:08.52360 2018-03-27_17:47:08.53609 2018-03-27_17:47:08.53950 2018-03-27_17:47:08.54465 2018-03-27_17:47:08.54984 2018-03-27_17:47:08.55415 2018-03-27_17:47:08.56402 2018-03-27_17:47:08.56602 2018-03-27_17:47:08.56975 2018-03-27_17:47:08.57448 2018-03-27_17:47:08.57717 2018-03-27_17:47:08.57945 2018-03-27_17:47:08.58361 2018-03-27_17:47:08.58697 2018-03-27_17:47:08.58981 2018-03-27_17:47:08.60252 2018-03-27_17:47:08.60763 2018-03-27_17:47:08.60792 2018-03-27_17:47:08.60906 2018-03-27_17:47:08.61228 2018-03-27_17:47:08.61743 2018-03-27_17:47:08.61908 2018-03-27_17:47:08.62158 2018-03-27_17:47:08.62227 2018-03-27_17:47:08.62457 2018-03-27_17:47:08.62525 2018-03-27_17:47:08.62768 2018-03-27_17:47:08.62914 2018-03-27_17:47:08.63272 2018-03-27_17:47:08.63547 2018-03-27_17:47:08.63863 2018-03-27_17:47:08.64103 2018-03-27_17:47:08.64225 2018-03-27_17:47:08.64510 2018-03-27_17:47:08.64715 2018-03-27_17:47:08.65115 vm.max_map_count = 262144
[INFO ][o.e.n.Node ] [] initializing …
2018-03-27 17:47:08,489 main ERROR Unable to write to stream /var/log/graylog/elasticsearch/graylog.log for appender rolling: org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /var/log/graylog/elasticsearch/graylog.log
2018-03-27 17:47:08,492 main ERROR An exception occurred processing Appender rolling org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to stream /var/log/graylog/elasticsearch/graylog.log
at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:264)
at org.apache.logging.log4j.core.appender.FileManager.writeToDestination(FileManager.java:261)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.writeToDestination(RollingFileManager.java:219)
at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:294)
at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:303)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:179)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161)
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:308)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(ExtendedLoggerWrapper.java:217)
at org.elasticsearch.common.logging.PrefixLogger.logMessage(PrefixLogger.java:102)
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2116)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2100)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1994)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1966)
at org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1303)
at org.elasticsearch.node.Node.(Node.java:254)
at org.elasticsearch.node.Node.(Node.java:245)
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91)

system · 2018-04-10 12:40:51 UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.