Graylog wiped/reset itself?

Graylog Central (peer support)
1

Currently running graylog using the AWS ami’s.

1 host configured as server and 2 hosts as ElasticSearch data nodes. This was working find for around a month, but as of today, it appears like it completely wiped itself out. I see the data is still on the Elastic Search nodes, but when logging into the website, it’s basically like the first time I’ve ever logged in to it. All inputs and users have been wiped.

Logged into mongo, at I see 3 DB’s (warning, graylog, local), but I am not seeing any tables or collections in any of these. I’ve looking at the logs in /var/log/graylog/ but everything in there is current and this could have happened during the last 3-4 days.

Is there any way I can restore all the configuration changes I have made? If not, how do I stop this from occurring in the future?

Thank you!

2

Hey @gaveup,

do you have anything in your MongoDB log files that could be related to this?

See mongodb log if any dropDatabase command has been executed or not. Generally its /var/log/mongo/mongo.log. Or see /etc/mongo.conf for log file location.
Source

Greetings,
Philipp

3

@derPhlipsi

Thinking the locations might be completely different for the AMI appliance as I can not find these files. Only logs I could find are located here:

/var/log/graylog/mongodb:
drwx------ 2 graylog root  4096 Sep 10 15:37 ./
-rwxr--r-- 1 root    root   194 Sep  2 14:16 @400000005b8c370106b99884.s*
-rwxr--r-- 1 root    root   476 Sep  3 14:16 @400000005b8d88810b4e5b14.s*
-rwxr--r-- 1 root    root   543 Sep  4 14:16 @400000005b8eda0111d3dd9c.s*
-rwxr--r-- 1 root    root   356 Sep  5 14:16 @400000005b902b811873ba3c.s*
-rwxr--r-- 1 root    root   246 Sep  6 14:16 @400000005b917d0119a50a64.s*
-rwxr--r-- 1 root    root   228 Sep  7 14:16 @400000005b92ce812096db7c.s*
-rwxr--r-- 1 root    root   147 Sep  8 14:16 @400000005b9420012770cb4c.s*
-rwxr--r-- 1 root    root   193 Sep  9 14:16 @400000005b9571812e8654ec.s*
-rwxr--r-- 1 root    root   760 Sep 10 14:16 @400000005b96c30135a47d94.s*
-rw-r--r-- 1 root    root 16130 Sep 10 15:37 @400000005b96d62c058a57b4.u
-rw-r--r-- 1 root    root    30 Aug 23 14:16 config
-rw-r--r-- 1 root    root  7045 Sep 10 15:38 current
-rw------- 1 root    root     0 Aug 23 14:16 lock
-rw-r--r-- 1 root    root     0 Sep 10 14:16 state

I have no way to read the old ones as well. Doing a find, I couldn’t even find that conf on the server. Maybe I am missing something, but this is a definite problem.

4

You are correct, these are the log files for mongodb. (should have pointed you to the docs, sorry ^^ :smiley:)

You cannot read the log files? Or what do you mean by “old ones”? Why should cat /var/log/graylog/mongodb/@400000005b8c370106b99884.s* not work?

Have a look at the Graylog default file locations (the docs link above). The MongoDB config is stored inside graylog.json for the appliances.

Greetings,
Philipp

5

EDIT:
So I was able to read these files using gunzip -c filename and found this:

2018-09-10_11:21:21.65153 2018-09-10T06:21:21.651-0500 I COMMAND  [conn79] dropDatabase admin finished
2018-09-10_11:21:23.43423 2018-09-10T06:21:23.434-0500 I COMMAND  [conn80] dropDatabase graylog starting
2018-09-10_11:21:23.48414 2018-09-10T06:21:23.483-0500 I COMMAND  [conn80] dropDatabase graylog finished

But the odd part is, no one would have had access to this server to do this, so uh…what happened? Assuming there isn’t a way to restore (as I dont think I saved a restore point)

6

Oh ok, good to know that they are gziped. :smiley:

This is odd. I don’t think that Graylog would have a function to drop its entire database. At least it doesn’t make sense to me that that would be implemented. Somebody from the dev team @Graylog_staff could clarify this.

I cannot really find information about seeing who triggered a MongoDB command, so I won’t be able to help you find out who deleted the database…

And no, sadly there is no way to restore the configuration without a database snapshot/backup I’m afraid…

Greetings,
Philipp

7

I guess that somone from the outside has just connected to the MongoDB and then deleted everything on it.

2018-09-10T06:21:23.434-0500 I COMMAND  [conn80] dropDatabase graylog starting

That is exactly the time this happens.

Not sure how this happend, but as MongoDB at the AWS AMI is listening not on localhost this can happen. The AMI is just a test/showcase and not hardened for production use cases.

1 Like
8

Thank you guys!

Looks like I will need to set a mongo login as well. Though I did this, but it appears not.

9

all you can configure on the AMI is documented on this page: http://docs.graylog.org/en/2.4/pages/configuration/graylog_ctl.html

Including:


sudo graylog-ctl set-mongodb-password [-a|-g] -u <username> -p <password>

What will enabel authentication

10

Thank you so much. Only question I have revolves around -a and -g.

    -u, --username USERNAME          MongoDB username
    -p, --password PASSWORD          MongoDB password
    -a, --admin                      User is the DB admin
    -g, --graylog                    User should be used for Graylog server

I’d obviously want this user to be admin, but what exactly does -g do? Assuming I just want to lock down mongo set-mongodb-password -a -u user -p pass should be good enough, or will graylog not be able to access mongo then?

11

You have the option that you create one Admin user (-a) that can be used to work with MongoDB and the other option the create a user for Graylog and write that user/password into the Graylog configuration (-g).

You might want to look in this section too: http://docs.graylog.org/en/2.4/pages/configuration/graylog_ctl.html#securing-an-appliance

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.