Graylog trying to find non-existent index

Graylog Central (peer support)
1

I have a strange problem with a non-existent index. I created a new index then deleted it but graylog thinks it is still there.

If i open the search page I get

  • Index not found for query: beta_0. Try recalculating your index ranges.

If I check the API I only see 3 index sets and nothing with beta_*

/api/system/indices/index_sets?skip=0&limit=0&stats=false

{
“total”: 3,
“index_sets”: [
{
“id”: “5b76bc34534e50078495bcc8”,
“title”: “Default index set”,
“description”: “The Graylog default index set - events that don’t match any stream rules end up here”,
“index_prefix”: “graylog”,
“shards”: 1,
“replicas”: 1,
“rotation_strategy_class”: “org.graylog2.indexer.rotation.strategies.TimeBasedRotationStrategy”,
“rotation_strategy”: {
“type”: “org.graylog2.indexer.rotation.strategies.TimeBasedRotationStrategyConfig”,
“rotation_period”: “P1D”
},
“retention_strategy_class”: “org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy”,
“retention_strategy”: {
“type”: “org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig”,
“max_number_of_indices”: 7
},
“creation_date”: “2018-08-17T12:14:44.01Z”,
“index_analyzer”: “standard”,
“index_optimization_max_num_segments”: 1,
“index_optimization_disabled”: false,
“writable”: true,
“default”: true
},
{
“id”: “5b9b6264534e500434c8ae6f”,
“title”: “Output”,
“description”: “Short time index set for sending to x”,
“index_prefix”: “output”,
“shards”: 1,
“replicas”: 0,
“rotation_strategy_class”: “org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy”,
“rotation_strategy”: {
“type”: “org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategyConfig”,
“max_docs_per_index”: 20000000
},
“retention_strategy_class”: “org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy”,
“retention_strategy”: {
“type”: “org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig”,
“max_number_of_indices”: 1
},
“creation_date”: “2018-09-14T07:25:24.356Z”,
“index_analyzer”: “standard”,
“index_optimization_max_num_segments”: 1,
“index_optimization_disabled”: false,
“writable”: true,
“default”: false
},
{
“id”: “5b7d2e24534e500c10346f40”,
“title”: “Media”,
“description”: “x 365 day index set”,
“index_prefix”: “media-”,
“shards”: 2,
“replicas”: 1,
“rotation_strategy_class”: “org.graylog2.indexer.rotation.strategies.TimeBasedRotationStrategy”,
“rotation_strategy”: {
“type”: “org.graylog2.indexer.rotation.strategies.TimeBasedRotationStrategyConfig”,
“rotation_period”: “P1D”
},
“retention_strategy_class”: “org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy”,
“retention_strategy”: {
“type”: “org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig”,
“max_number_of_indices”: 365
},
“creation_date”: “2018-08-22T09:34:28.957Z”,
“index_analyzer”: “standard”,
“index_optimization_max_num_segments”: 1,
“index_optimization_disabled”: false,
“writable”: true,
“default”: false
}
],
“stats”: {}
}

Now If I check the API /api/system/indices/ranges the first entry is:

{
  "index_name": "beta_0",
  "begin": "1970-01-01T00:00:00.000Z",
  "end": "1970-01-01T00:00:00.000Z",
  "calculated_at": "2018-11-09T11:09:34.414Z",
  "took_ms": 0
},

I checked elastic and there is no index with beta_* in it.

I think I need to delete this from mongo somewhere but not sure how and wanted to check with you guys first.

Any help is much appreciated thanks!

2

Just to add some more info as graylog is doing some other weird stuff:

When I go to the indices page I get a few different error messages. Not all at once but different when I refresh the page:

Fetching index sets list failed: Cat response did not contain a JSON Array

Fetching index sets list failed: Couldn't check stats of indices [graylog_*]

That’s strange as the API calls work so I’m struggling to understand why that is.
I still think it’s something to do with the issue above. Everything else works fine as events are still getting written to the three elastic nodes we have and are forwarded to our SIEM.

I triple checked the configs and can’t see anything wrong there but there was an issue with mongo when all of this started and I had to rebuild one of the replica set members. I’m not a mongo expert so I’m wondering how to see where metadata about index sets is stored and how to clear any entry about the beta_* index ?

3

Did you follow the advice: Try recalculating your index ranges ?

4

yeah did that and still same issue

5

I’m pretty sure I need to delete some metadata from mongo to solve this but I really don’t know enough about mongo or how the graylog mongo db is structured to be comfortable doing that without some advice from you guys

really appreciate the help though as this one has me totally stumped

6

it might be possible if you stop Graylog, delete “index ranges” collection. check index_sets collection.
Start Graylog and Recalculate the index ranges.

no guarantee for that - just what might work!

7

Yeah I managed to get rid of the entry for the non-existent shard in the index ranges collection and then recalculated ranges thanks so thats one problem solved :slight_smile:

Now I am getting errors with the API :

image
image.png716×71 9.09 KB

but when I run that API call manually it’s not returning an error?

Request URL

http://192.168.168.51:9000/api/system/indices/index_sets/stats

Response Body

{
  "indices": 94,
  "documents": 6883554532,
  "size": 4580514658168
}

Response Code

200

any idea what might cause that?

Thanks again!

8

never mind was my mistake. added a new elastic node and had port 9000 in the list for it instead of 9200

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.