Recovering a corrupted mongodb database

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

recovering a corrupted mongodb database
1. Describe your incident:
Hi,
I have a cluster of 2 servers replicating (following are the services in the servers
Garlog
Mongdb
Elasticsearch
All worked fine but since I extended the LVM disk on the first server I was not able to connect to that server lately
I checked the log and found that one of the servers was not able to connect to mongo db at port 27017; the second server can connect

2. Describe your environment:

  • OS Information:
    Two Ubuntu 20.04 servers in a cluster

  • Package Version:

  • Service logs, configurations, and environment variables:
    Graylog server
    Mongodab
    Elasticsearch
    All in a cluster

**3. What steps have you already taken to try and solve the problem? **

I checked the logs and found that one of the servers was not able to connect to mongo db at port 27017; the second server can connect
Tests and logs are showing that the mongdb database is corrupted
I tried repairing the db with the command below, but failed
sudo mongod --repair --dbpath /var/lib/mongodb
The mongod command is showing the following output

"on":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2021-12-15T17:15:32.290-05:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
{"t":{"$date":"2021-12-15T17:15:32.290-05:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{}}}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"E",  "c":"STORAGE",  "id":20557,   "ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"NonExistentPath: Data directory /data/db not found. Create the missing directory or specify another path using (1) the --dbpath command line option, or (2) by adding the 'storage.dbPath' option in the configuration file."}}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"REPL",     "id":4784900, "ctx":"initandlisten","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":10000}}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"COMMAND",  "id":4784901, "ctx":"initandlisten","msg":"Shutting down the MirrorMaestro"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"SHARDING", "id":4784902, "ctx":"initandlisten","msg":"Shutting down the WaitForMajorityService"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"NETWORK",  "id":20562,   "ctx":"initandlisten","msg":"Shutdown: going to close listening sockets"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"NETWORK",  "id":4784905, "ctx":"initandlisten","msg":"Shutting down the global connection pool"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"STORAGE",  "id":4784906, "ctx":"initandlisten","msg":"Shutting down the FlowControlTicketholder"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"-",        "id":20520,   "ctx":"initandlisten","msg":"Stopping further Flow Control ticket acquisitions."}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"NETWORK",  "id":4784918, "ctx":"initandlisten","msg":"Shutting down the ReplicaSetMonitor"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"SHARDING", "id":4784921, "ctx":"initandlisten","msg":"Shutting down the MigrationUtilExecutor"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"CONTROL",  "id":4784925, "ctx":"initandlisten","msg":"Shutting down free monitoring"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"STORAGE",  "id":4784927, "ctx":"initandlisten","msg":"Shutting down the HealthLog"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"STORAGE",  "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"FTDC",     "id":4784926, "ctx":"initandlisten","msg":"Shutting down full-time data capture"}
{"t":{"$date":"2021-12-15T17:15:32.292-05:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"initandlisten","msg":"Now exiting"}"

4. How can the community help?

I need to know how can I recover the database using the replica from the other working server
If yes, what are the steps of doing it.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

Hello,

Sorry I had to edit your post, the mongod output was very hard to read.

Can I ask what database are you trying to repair? If its Graylog database then this command might not be right.

sudo mongod --repair --dbpath /var/lib/mongodb

To help you further we may need more information to help us, help you. For a better understand please have a look at this posts.

When posting configuration files or Logs please use the markup so its easier to read and understand.

Have you check the status on all your services ( i.e. Graylog, Elasticsearch, MongoDb)?
Did you check your Firewalls, selinux/apparmor, etc…?

Data directory /data/db not found.

Did you check you mongo.config file where the db path is located?

vi /etc/mongod.conf

Should look something like this

# Where and how to store data.
storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true

This is also mentioned in your log output.

by adding the 'storage.dbPath' option in the configuration file."

Maybe compare your other MongoDb configuration file and match them up so there both the same. Just an idea.

Hi,
Thank you for your response
A part of the mongodb configuration file /etc/mongod.conf is follow and the db path is right, but I am not sure why the log is showing “Data directory /data/db not found.”

Where and how to store data.

storage:
dbPath: /var/lib/mongodb
journal:
enabled: true

The other server has the same configuration and the same database path
The Linux Firewall is off so there is not firewall issue
The database I was trying to recover is not a Graylog database but the mongodb database; if the command I used is not the right one to repair the corrupted database, can you please the right command?

Hello,

My apologies I thought you were trying to repair Graylog Database.

Not 100% sure if this would work but check out this documentation below

Also make sure the permission are correct, maybe Mongo can find it because it under another name.

chown -R mongod:mongod /var/lib/mongo/

Have you tried restarting Mongo service?

Since you have two MongoDb servers for redundancy. If all else fails you could try to re-install MongoDb. Be Careful, make sure you have backups of your GL database and mongo’s config file. You can execute mongdump and put it in a safe place then reinstall Mongodb.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.