Web interface breaks after reboot

Graylog Central (peer support)
1

I don’t really know what’t happening. Deployment works fine until doing a reboot or restarting the graylog service. After either one of those actions, web interface breaks. “Connection Refused”

Output from graylog server.log file:

2017-04-27T08:42:32.733-05:00 INFO  [cluster] No server chosen by WritableServerSelector from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out

There is a mongod.lock file created after rebooting in /var/lib/mongodb
Deleted that file and issued “mongod --repair”

Anyone else have this issue and more importantly, fix it?

2

is your mongodb running?

how much space did you have on the device where your mongoDB is stored?

3

Nope, and when I use ‘service mongodb start’ i get the same error as this:

sudo service mongodb status
● mongodb.service - An object/document-oriented database
   Loaded: loaded (/lib/systemd/system/mongodb.service; enabled; vendor preset:
   Active: failed (Result: exit-code) since Thu 2017-04-27 13:12:23 CDT; 3min 48
     Docs: man:mongod(1)
  Process: 777 ExecStart=/usr/bin/mongod --config /etc/mongodb.conf (code=exited
 Main PID: 777 (code=exited, status=100)

Apr 27 13:12:22 <server> systemd[1]: Started An object/document-oriented d
Apr 27 13:12:23 <server> systemd[1]: mongodb.service: Main process exited,
Apr 27 13:12:23 <server> systemd[1]: mongodb.service: Unit entered failed
Apr 27 13:12:23 <server> systemd[1]: mongodb.service: Failed with result '
...skipping...
● mongodb.service - An object/document-oriented database

here’s my space: don’t know why /dev/mapper/system–vg–01-var is @ 98%

/var/lib/mongodb$ df -h
Filesystem                                Size  Used Avail Use% Mounted on
udev                                      2.0G     0  2.0G   0% /dev
tmpfs                                     395M  5.5M  389M   2% /run
/dev/mapper/system--vg--01-root            46G  1.7G   42G   4% /
tmpfs                                     2.0G     0  2.0G   0% /dev/shm
tmpfs                                     5.0M     0  5.0M   0% /run/lock
tmpfs                                     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/system--vg--01-var            4.5G  4.2G  110M  98% /var
/dev/mapper/system--vg--01-home           4.5G  9.4M  4.3G   1% /home
/dev/mapper/system--vg--01-tmp            4.5G  9.7M  4.3G   1% /tmp
/dev/sda2                                 501M  105M  360M  23% /boot
/dev/sda1                                 523M  6.8M  517M   2% /boot/efi
/dev/mapper/system--vg--01-var_log        9.1G   63M  8.5G   1% /var/log
/dev/mapper/system--vg--01-var_log_audit  9.1G   21M  8.6G   1% /var/log/audit
cgmfs                                     100K     0  100K   0% /run/cgmanager/fs
tmpfs                                     395M     0  395M   0% /run/user/1001
4

check you mongoDB log file - it will give you the information what is wrong!

I guess it is the filled /var/ - keep in mind that this is the default file store …

5

/var/lib/mongodb/journal had three files j._7 j._8 and prealloc.* over 1G a piece. this was causing var to show 98% disk usage. I deleted these three files and also edited /etc/mongo.conf by adding the line “smallfiles=true” under journal=true

Issue seems to be fixed, web interface survives reboot as well as mongodb starting upon machine start.