MongoDB crash suddenly [SOLVED]


(Davide Pala) #1

It’s a free space issue … i’ve resized the graylog volume and all works fine!

Hi, today my installation have some problem … graylog become unreachable suddenly,
this is the log of garylog:

Blockquote
2018-07-19T18:07:34.148+02:00 INFO [cluster] No server chosen by WritableServerSelector from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out
2018-07-19T18:07:34.176+02:00 WARN [Errors] The following warnings have been detected: WARNING: Unknown HK2 failure detected:
MultiException stack 1 of 1
com.google.inject.ProvisionException: Unable to provision, see the following errors:

  1. Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
    at org.graylog2.users.UserServiceImpl.(UserServiceImpl.java:67)
    while locating org.graylog2.users.UserServiceImpl
    while locating org.graylog2.shared.users.UserService
    1 error
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028)
    at org.jvnet.hk2.guice.bridge.internal.GuiceServiceHk2Bean.create(GuiceServiceHk2Bean.java:114)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:479)

and this is the journalctl otput after try to restart mongo

Blockquote
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at com.google.inject.Guice.createInjector(Guice.java:99)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:379)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:193)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at org.graylog2.bootstrap.Main.main(Main.java:44)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: Caused by: java.io.IOException: No space left on device
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at java.io.FileOutputStream.writeBytes(Native Method)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at java.io.FileOutputStream.write(FileOutputStream.java:326)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:262)
Jul 19 18:16:22 localhost.localdomain graylog-server[970]: … 122 more
Jul 19 18:17:55 localhost.localdomain polkitd[644]: Registered Authentication Agent for unix-process:1509:58402 (system bus name :1.20 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object
Jul 19 18:17:55 localhost.localdomain systemd[1]: Starting MongoDB Database Server…
– Subject: Unit mongod.service has begun start-up
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

– Unit mongod.service has begun starting up.
Jul 19 18:17:55 localhost.localdomain rsyslogd[972]: action ‘action 0’ resumed (module ‘builtin:omfile’) [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jul 19 18:17:55 localhost.localdomain rsyslogd[972]: action ‘action 0’ resumed (module ‘builtin:omfile’) [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jul 19 18:17:55 localhost.localdomain rsyslogd[972]: action ‘action 0’ resumed (module ‘builtin:omfile’) [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jul 19 18:17:55 localhost.localdomain rsyslogd[972]: action ‘action 0’ resumed (module ‘builtin:omfile’) [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jul 19 18:17:55 localhost.localdomain rsyslogd[972]: action ‘action 0’ resumed (module ‘builtin:omfile’) [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jul 19 18:17:55 localhost.localdomain rsyslogd[972]: action ‘action 0’ resumed (module ‘builtin:omfile’) [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jul 19 18:17:55 localhost.localdomain rsyslogd[972]: action ‘action 0’ resumed (module ‘builtin:omfile’) [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jul 19 18:17:55 localhost.localdomain rsyslogd[972]: action ‘action 0’ resumed (module ‘builtin:omfile’) [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jul 19 18:17:55 localhost.localdomain rsyslogd[972]: action ‘action 0’ resumed (module ‘builtin:omfile’) [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jul 19 18:17:55 localhost.localdomain rsyslogd[972]: action ‘action 0’ resumed (module ‘builtin:omfile’) [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Jul 19 18:17:55 localhost.localdomain rsyslogd[972]: action ‘action 0’ suspended, next retry is Thu Jul 19 18:18:25 2018 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
Jul 19 18:17:55 localhost.localdomain mongod[1524]: 2018-07-19T18:17:55.890+0200 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'no
Jul 19 18:17:55 localhost.localdomain mongod[1524]: about to fork child process, waiting until server is ready for connections.
Jul 19 18:17:55 localhost.localdomain mongod[1524]: forked process: 1527
Jul 19 18:17:55 localhost.localdomain mongod[1524]: ERROR: child process failed, exited with error number 1
Jul 19 18:17:55 localhost.localdomain mongod[1524]: To see additional information in this output, start without the “–fork” option.
Jul 19 18:17:55 localhost.localdomain systemd[1]: mongod.service: control process exited, code=exited status=1
Jul 19 18:17:55 localhost.localdomain systemd[1]: Failed to start MongoDB Database Server.
– Subject: Unit mongod.service has failed
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

– Unit mongod.service has failed.

– The result is failed.
Jul 19 18:17:55 localhost.localdomain systemd[1]: Unit mongod.service entered failed state.
Jul 19 18:17:55 localhost.localdomain systemd[1]: mongod.service failed.
Jul 19 18:17:55 localhost.localdomain polkitd[644]: Unregistered Authentication Agent for unix-process:1509:58402 (system bus name :1.20, object path /org/freedesktop/PolicyKit1/Authenticati

if i restart entire machine all work fine. any idea?


(Jochen) #2

Did you change anything before MongoDB failed to start?
Is there enough disk space for MongoDB to run?
Is there enough main memory for MongoDB to run?
Are the data files uncorrupted?

See https://docs.mongodb.com/manual/administration/production-checklist-operations/ for an operations checklist.


(system) #3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.