Endless loop of "Fetching archive catalog failed: Multiple entries with same key"

hriley · 2018-02-14 09:00:26 UTC

I’ve been working on fixing my problems with archiving and it looks like something has gotten corrupted. When I go to the Archives Overview, it never loads the list of archives and I get a constant stream of errors like the below (each one with a different string after ‘IndexArchiveContext@’). I have moved all archives related to index graylog_39 from any of the backend directories and have tried to rebuild the catalog, force refresh the page, restart graylog, all to no avail. Any ideas on how to sort this out?

Fetching archive catalog failed: Multiple entries with same key: graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@7efeb37c and graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@579a7237

Fetching archive catalog failed: Multiple entries with same key: graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@705de73f and graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@e5002f9

Fetching archive catalog failed: Multiple entries with same key: graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@59bebd8 and graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@649dc0b7

Fetching archive catalog failed: Multiple entries with same key: graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@59290803 and graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@15d5ec98

Fetching archive catalog failed: Multiple entries with same key: graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@72014350 and graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@59b08cc0

Fetching archive catalog failed: Multiple entries with same key: graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@448a0690 and graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@1a67052a

Fetching archive catalog failed: Multiple entries with same key: graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@4b2b88e8 and graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@28e0570b

Fetching archive catalog failed: Multiple entries with same key: graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@5cda5436 and graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@264f9b6

Fetching archive catalog failed: Multiple entries with same key: graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@2688d6f5 and graylog_39=org.graylog.plugins.archive.rest.ArchivesResource$IndexArchiveContext@7b26d0d2

jochen · 2018-02-14 09:04:10 UTC

What’s in the logs of your Graylog node?
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

Please also include a dump of the archive-related collections from Graylog’s MongoDB database.

hriley · 2018-02-14 17:59:24 UTC

There is nothing in either the Graylog or the Mongod logs related to this. I even turned on DEBUG logging on Graylog, but couldn’t see anything related to “graylog_39”.

If you tell me how to get the “dump of the archive-related collections from Graylog’s MongoDB database”, I should be able to get that to you.

jochen · 2018-02-15 08:48:54 UTC

How do you know?

https://docs.mongodb.com/manual/reference/program/mongoexport/

hriley · 2018-02-15 13:53:08 UTC

I know there’s nothing there because I looked. The only thing that’s in the graylog log is messages about archive jobs for other indices. There’s no reference to either graylog_39 or repeated messages about something related to fetching archive catalogs or multiple entries with same key. As to the mongod logs, I can’t see anything related to archives other then when it’s starting a new job. Also, no references to graylog_39 or ArchiveContext or the id string after the @ or any kind of repeated messages that occur at the same pace as the errors in the Archive page. Now, if there are any log entries that are indeed related to this, they’re cleverly hidden away and I’ll ask you to tell me how to decode them or specifically search for them.

As to the dump from mongod, now that I know how to generally perform exports of data from mongod, could you tell me how to specifically get the “dump of archive-related collections”? I don’t know the mongod database schema and have never used mongod before.

jochen · 2018-02-15 15:19:04 UTC

How do you know none of the log messages is related?
In other words, please post the complete logs of your Graylog nodes.

Every collection in the MongoDB database your Graylog cluster is using whose name is starting with “archive_”.

hriley · 2018-02-20 17:02:47 UTC

This forum software won’t allow me to upload non-images. Here’s 20 minutes worth of the Graylog log:

2018-02-20T10:02:14.493-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Number of indices (52) higher than limit (50). Running retention for 2 indices.
2018-02-20T10:02:17.357-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy] for index <graylog_44>
2018-02-20T10:02:17.404-06:00 INFO  [SystemJobManager] Submitted SystemJob <6c8e6fc2-1657-11e8-8d26-fee5de21aa98> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob]
2018-02-20T10:02:17.405-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy] for index <graylog_41>
2018-02-20T10:02:17.406-06:00 WARN  [ArchiveCreateJob] Archive "graylog_44" for index "graylog_44" already exists, not running archive job!
2018-02-20T10:02:17.406-06:00 INFO  [SystemJobManager] SystemJob <6c8e6fc2-1657-11e8-8d26-fee5de21aa98> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob] finished in 1ms.
2018-02-20T10:02:17.407-06:00 WARN  [ArchiveRetentionStrategy] Not running archive job for index "graylog_41", there is already a job running for index "graylog_44"!
2018-02-20T10:07:05.476-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Number of indices (52) higher than limit (50). Running retention for 2 indices.
2018-02-20T10:07:07.266-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy] for index <graylog_44>
2018-02-20T10:07:07.295-06:00 INFO  [SystemJobManager] Submitted SystemJob <19584af0-1658-11e8-8d26-fee5de21aa98> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob]
2018-02-20T10:07:07.295-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy] for index <graylog_41>
2018-02-20T10:07:07.296-06:00 WARN  [ArchiveCreateJob] Archive "graylog_44" for index "graylog_44" already exists, not running archive job!
2018-02-20T10:07:07.296-06:00 WARN  [ArchiveRetentionStrategy] Not running archive job for index "graylog_41", there is already a job running for index "graylog_44"!
2018-02-20T10:07:07.296-06:00 INFO  [SystemJobManager] SystemJob <19584af0-1658-11e8-8d26-fee5de21aa98> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob] finished in 1ms.
2018-02-20T10:12:05.817-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Number of indices (52) higher than limit (50). Running retention for 2 indices.
2018-02-20T10:12:07.610-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy] for index <graylog_44>
2018-02-20T10:12:07.635-06:00 INFO  [SystemJobManager] Submitted SystemJob <cc5c8a30-1658-11e8-8d26-fee5de21aa98> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob]
2018-02-20T10:12:07.636-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy] for index <graylog_41>
2018-02-20T10:12:07.636-06:00 WARN  [ArchiveCreateJob] Archive "graylog_44" for index "graylog_44" already exists, not running archive job!
2018-02-20T10:12:07.636-06:00 INFO  [SystemJobManager] SystemJob <cc5c8a30-1658-11e8-8d26-fee5de21aa98> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob] finished in 0ms.
2018-02-20T10:12:07.637-06:00 WARN  [ArchiveRetentionStrategy] Not running archive job for index "graylog_41", there is already a job running for index "graylog_44"!
2018-02-20T10:17:05.704-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Number of indices (52) higher than limit (50). Running retention for 2 indices.
2018-02-20T10:17:07.216-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy] for index <graylog_44>
2018-02-20T10:17:07.245-06:00 INFO  [SystemJobManager] Submitted SystemJob <7ef165d0-1659-11e8-8d26-fee5de21aa98> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob]
2018-02-20T10:17:07.246-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy] for index <graylog_41>
2018-02-20T10:17:07.246-06:00 WARN  [ArchiveRetentionStrategy] Not running archive job for index "graylog_41", there is already a job running for index "graylog_44"!
2018-02-20T10:17:07.247-06:00 WARN  [ArchiveCreateJob] Archive "graylog_44" for index "graylog_44" already exists, not running archive job!
2018-02-20T10:17:07.247-06:00 INFO  [SystemJobManager] SystemJob <7ef165d0-1659-11e8-8d26-fee5de21aa98> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob] finished in 1ms.
2018-02-20T10:22:08.744-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Number of indices (52) higher than limit (50). Running retention for 2 indices.
2018-02-20T10:22:11.880-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy] for index <graylog_44>
2018-02-20T10:22:11.902-06:00 INFO  [SystemJobManager] Submitted SystemJob <34885de0-165a-11e8-8d26-fee5de21aa98> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob]
2018-02-20T10:22:11.902-06:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy] for index <graylog_41>
2018-02-20T10:22:11.903-06:00 WARN  [ArchiveRetentionStrategy] Not running archive job for index "graylog_41", there is already a job running for index "graylog_44"!
2018-02-20T10:22:11.903-06:00 WARN  [ArchiveCreateJob] Archive "graylog_44" for index "graylog_44" already exists, not running archive job!
2018-02-20T10:22:11.903-06:00 INFO  [SystemJobManager] SystemJob <34885de0-165a-11e8-8d26-fee5de21aa98> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob] finished in 0ms.

There are no log entries from the Mongod log from that time frame.

As to the mongod dump, please tell me what the name of the database is. Or, better yet, you could actually tell me the commands to issue to get me the information you’re requesting.

If this would be better suited for a support ticket, I can do that since my company is an Enterprise customer. I was just hoping this thread could provide answers to anyone else who might experience this issue.

jochen · 2018-02-20 17:04:51 UTC

It’s the MongoDB database you’ve configured in your Graylog configuration file in the mongodb_uri setting.
You can use the aforementioned mongoexport to export/dump the relevant collections (i. e. every collection starting with “archive_”).

I’d suggest you do this.

system · 2018-03-06 17:05:08 UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.