Config-only backup questions

[dbcrvl]

Hi,

a few months ago I opened this thread: Graylog backups

but I can’t reply to it anymore as it is now closed.

@macko003 replied on it and gave me a script to back up Graylog, but I have confirmed I only need to backup the configuration (streams, dashboards, extractors…), not the actual logs.

So, just wanted to confirm that what I am planning to do is right.

Backup these files:
/etc/mongodb-keyfile
/etc/mongod.conf
/etc/elasticsearch/elasticsearch.yml
/etc/graylog/server/server.conf
/etc/graylog/server/node-id

I can’t find any “mongodb-keyfile” file in my system, is this normal?

Then do a mongodump of the “graylog” database (only his DB? not the admin, local or or config ones?):
/usr/bin/mongodump -d graylog -o /backup/graylog.mongodump

when I do this, it does not create a file, but a folder with lots of json and bson files in it. Is this what’s supposed to happen?

It was also mentioned to use “a script to make snapshot based on the official solution
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html”. Is this also required if I do NOT want to have a backup of the data, just the configs?

Thanks,

[macko003]

I think not totally clear the components in your mind.
I suggest start with that, I think it will help to answer your questions.
After you have more questions, please let us know.

http://docs.graylog.org/en/3.0/pages/faq.html

[dbcrvl]

Well, as I am backing up to a different server, I ended up setting up a script to copy (rsync) all the relevant folders for each component (http://docs.graylog.org/en/2.4/pages/configuration/file_location.html) to the external backup server every night:

mongodump -d graylog -o /graylog_backup/graylog.mongodump

rsync -avHzR /etc/graylog/ /usr/share/graylog-server/ /etc/default/ /var/lib/graylog-server/journal/ /etc/elasticsearch/ /etc/mongod.conf /graylog_backup/ /var/lib/mongodb/ root@x.x.x.x::backup_graylog

except for the ElasticSearch data files (/var/lib/elasticsearch/nodes/0), as it is the large dir that contains the actual data, and I do not need to backup this data, I just need graylog configs.

The whole backup is under 1GB, which is very small.

I think this will suffice in the event of the server dying and having to restore its config. What do you think?

Thanks!

[macko003]

It should be ok, but to be sure do a disaster recovery. And make a doc about the steps.
When you have to do it in real life you just open your doc, it can help a lot, and you will calm instead of hasty.

Some thing,

• /var/lib/mongodb/ - I think you do the backup with running DB, so it will contains inconsistent data.
• /var/lib/graylog-server/journal/ - It contains the journal for messages. You wrote you need the config, so you don’t need it.
• /usr/share/graylog-server/ - You will install graylog from apt/yum, so you won’t use eg. the binaries (but you should have the plugins…
• /etc - I suggest do a full etc backup. It’s small, but you will have a lot of usefull information at restore. Eg. IP address, gateway, ntp, dns servers, etc. faster if you restore the important files instead of searching for some IPs, or settings.
• you can compress the things for a compact, smaller file. eg. change the rsync to tar, and rsync the final file. (eg. my mongo dump is 350 MB, tar make it 15 mb under 10s)
• if you compress/tar the whole thing, you can put a date/number in your file name, and you will have a little svn, and you can restore to a good config few backup later

BUT you did a good job.

[dbcrvl]

Thanks for your reply:

@macko003

• /var/lib/mongodb/ - I think you do the backup with running DB, so it will contains inconsistent data.

I know, that’s why I do a mongodump and copy the resultant folder (/graylog_backup/) over. So I can probably remove this folder (/var/lib/mongodb/) from the rsync command, I don’t need it if I have the dump.

@macko003

• /var/lib/graylog-server/journal/ - It contains the journal for messages. You wrote you need the config, so you don’t need it.

I thought so but I wasn’t fully sure. I will remove this folder from the rsync command as well then.

@macko003

• /usr/share/graylog-server/ - You will install graylog from apt/yum, so you won’t use eg. the binaries (but you should have the plugins…

Yes, I do not need the binaries, but I do not want to have to copy multiple subdirs, so I just copy the whole dir as it is only a few MBs.

@macko003

• /etc - I suggest do a full etc backup. It’s small, but you will have a lot of usefull information at restore. Eg. IP address, gateway, ntp, dns servers, etc. faster if you restore the important files instead of searching for some IPs, or settings.

Might do that as the whole /etc/ is only 65MB.

@macko003

• you can compress the things for a compact, smaller file. eg. change the rsync to tar, and rsync the final file. (eg. my mongo dump is 350 MB, tar make it 15 mb under 10s)
• if you compress/tar the whole thing, you can put a date/number in your file name, and you will have a little svn, and you can restore to a good config few backup later

My setup is really small, just a 1MB database, so no need to compress for now. Will think about it if it grows over time.

Thanks!