Sidecar confgurations lost

Graylog Central (peer support)
1

Hi,
I am doing the installation and testing. I created a test linux host and installed sidecar. I did all the configuration and started receiving logs. Then I switched to Windows host did some configurations to sidecar. Was not directly successful and It was late in the night and I said “ok I will continue tomorrow”

When I came in the morning there was nothing left from sidecars on the interface.
All the inputs were gone also.

I checked logs but could not see anything.
Mongodb seems to be ok rs.status() showed all good.

what happened to my configurations why are they lost?

2

I found only this error : 2020-04-20T00:50:29+02:00 07e0b5f1 / graylogpr1 Notification condition [NO_MASTER] has been fixed.

3

this has happened again as of now I was working on the sidecar configuration and both sidecar configs got deleted suddenly. Where should I look for problem?

4

Here is the logs from graylog-server
2020-04-24T00:32:45.546+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T01:32:45.542+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T02:32:45.542+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T03:32:45.549+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T04:32:45.541+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T05:32:45.545+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T06:32:45.546+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T07:32:45.543+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T08:32:45.553+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T09:32:45.541+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T10:32:45.542+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T11:32:45.548+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
2020-04-24T11:38:02.368+02:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering.
2020-04-24T11:38:02.373+02:00 ERROR [NodePingThread] Uncaught exception in periodical
2020-04-24T11:38:03.359+02:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering.

5

and a bit more from the logs

2020-04-24T10:32:45.542+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'E11000 duplicate key error collection: graylog.inde
x_field_types index: index_name_1 dup key: { : “graylog_0” }

020-04-24T11:32:45.548+02:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message ‘E11000 duplicate key error collection: graylog.inde
x_field_types index: index_name_1 dup key: { : “graylog_0” }’

2020-04-24T11:38:02.368+02:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering.
2020-04-24T11:38:02.373+02:00 ERROR [NodePingThread] Uncaught exception in periodical
com.mongodb.WriteConcernException: Write failed with error code 215 and error message ‘Cannot create collection graylog.nodes - database is in the process of being dropped.’

Please Help.

6

seems like this problem is still not solved ?

7

@bernd
I have the version Graylog 3.2.4+a407287 and ssadly I still get this error :frowning:

8

Did you check the Index mencioned? “Graylog_0” ? Is there an “Graylog_1” without being the active index? you may try to drop the index in mongod and then try to reindex. This action you may lose all the index information. What´s the Mongodb logs says?

9

I could not find the index “Graylog_0” or 1
graylog:PRIMARY> db.collection.
… getFullName()
test.collection
graylog:PRIMARY> db.collection.getFullName()
test.collection
graylog:PRIMARY> db.collection.getIndexes()

graylog:PRIMARY> db.collection.getIndices()

10

@lcosta
but I found this
graylog:PRIMARY> show dbs
READ_ME_TO_RECOVER_YOUR_DATA 0.000GB
admin 0.000GB
config 0.000GB
graylog 0.002GB
local 0.531GB

so graylog exists

11

seems like everything is deleted and dbs are defaulted
I could not see anything in the logs

12

So I found something like this
Seems like someone has connecte somehow to my unsecure server [conn83] and did drop Databese
can it be because of this?

2020-04-24T11:38:01.976+0200 I NETWORK [conn83] received client metadata from 195.54.160.225:33308 conn83: { driver: { name: “PyMongo”, version: “3.10.1” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “4.9.0-12-amd64” }, platform: “CPython 3.5.3.final.0” }
2020-04-24T11:38:02.240+0200 I COMMAND [conn83] dropDatabase graylog - starting
2020-04-24T11:38:02.240+0200 I COMMAND [conn83] dropDatabase graylog - dropping 40 collections
2020-04-24T11:38:02.240+0200 I COMMAND [conn83] dropDatabase graylog - dropping collection: graylog.content_packs
2020-04-24T11:38:02.240+0200 I STORAGE [conn83] dropCollection: graylog.content_packs (25443518-8f79-4e55-bb2a-aee8ad8fb402) - renaming to drop-pending collection: graylog.system.drop.1587721082i3t18.content_packs with drop optime { ts: Timestamp(1587721082, 3), t: 18 }
2020-04-24T11:38:02.240+0200 I STORAGE [conn83] renameCollection: renaming collection 25443518-8f79-4e55-bb2a-aee8ad8fb402 from graylog.content_packs to graylog.system.drop.1587721082i3t18.content_packs
2020-04-24T11:38:02.242+0200 I COMMAND [conn83] dropDatabase graylog - dropping collection: graylog.streams
2020-04-24T11:38:02.242+0200 I STORAGE [conn83] dropCollection: graylog.streams (ebe2de9f-ed25-4f53-9377-407864a35de6) - renaming to drop-pending collection: graylog.system.drop.1587721082i4t18.streams with drop optime { ts: Timestamp(1587721082, 4), t: 18 }
2020-04-24T11:38:02.242+0200 I STORAGE [conn83] renameCollection: renaming collection ebe2de9f-ed25-4f53-9377-407864a35de6 from graylog.streams to graylog.system.drop.1587721082i4t18.streams
2020-04-24T11:38:02.243+0200 I COMMAND [conn83] dropDatabase graylog - dropping collection: graylog.index_field_types
2020-04-24T11:38:02.243+0200 I STORAGE [conn83] dropCollection: graylog.index_field_types (19deee44-b18e-49f8-94b5-bd4e5c33d4a4) - renaming to drop-pending collection: graylog.system.drop.1587721082i5t18.index_field_types with drop optime { ts: Timestamp(1587721082, 5), t: 18 }
2020-04-24T11:38:02.243+0200 I STORAGE [conn83] renameCollection: renaming collection 19deee44-b18e-49f8-94b5-bd4e5c33d4a4 from graylog.index_field_types to graylog.system.drop.1587721082i5t18.index_field_types
2020-04-24T11:38:02.246+0200 I COMMAND [conn83] dropDatabase graylog - dropping collection: graylog.sidecar_collector_actions

13

Someone drop the database you must reindex (recalculate index ranges) if not work, you can create manually

14

your public available mongodb is deleted by a random guy.

15

it takes less than 6 hours to be hacked without security :sweat_smile:

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.