Graylog Server File System Error

Graylog Central (peer support)
1

I tried connecting to the Graylog web interface this morning and it was refusing to load. I rebooted the server and was presented with the following failure on restart:

image
image919×168 13.7 KB

I have tried running a manual fsck, but the server is stuck in the root filesystem (initramfs) and cannot exit out, exiting returns the same error as above.

The server is a VM in vSphere running Ubuntu (64-bit), I’m not sure of the exact version and I cant run lsb_release command from (initrams).
We are running Graylog version 4.something (again I apologise, I cant check while the servers down)

We are looking to restore from a backup of the VM, but I was hoping to get to the bottom of what caused this issue, does anyone have any suggestions? It would be much appreciated, thanks!

2

Hi @Linedo ,
I think something with your (virtual) disk is broken. The OS does not boot because some struggle on the storage. I hope you have a backup?

3

Hi @ihe , thanks for your response! We do have a backup but the backup seems to have the same file system issue and it has overwritten the previous backup.

I have tried running “fsck -y /dev/mapper/ubuntu–vg-ubuntu–lv” but as the drive is mounted it refuses to scan it. I cannot seem to unmount the drive as it returns an “Invalid Argument” error. I cannot exit out of (initramfs) without it getting stuck.

4

Hey

Generally you can always fsck from a Live system (i.e., boot the installer)

5

Hi @gsmith , thanks for your response! When I do this, it fixes a load of free inodes and directories counts and cleans the file system, but it doesn’t have any effect in terms of restoring the server to its former glory.

We have started to rebuild the server from scratch, does anyone know a way to recover individual files? Specifically the lookup tables we had created.

Also does anyone know the default file locations on the server for configured pipeline rules, widgets, reports, alerts, sidecars, etc? If they exist, can you just copy these files to a new Graylog server?

Thanks

6

Hi @Linedo,
Config from Pipelines, Widgets, Reports and Alerts is saved in the MongoDB. According to the docs this is supposed to be here: /data/db . I am not sure how easy it will be to read this file and put it into a new instance.

If you start from scratch I recomment to go for OpenSearch right from the start, instead of Elastic. It will be the future Logdatabase of Graylog.

1 Like
7

Hello,

These are in MongoDb you can execute mongodump and then scp that database over to the new node. Thats where all your configurations made are there.

All you file/s are stored in Elasticsearch/Opensearch, you would have to create a Repo on that node then you can transport it to the new node.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.