Range recalculation on index rotation

Graylog Central (peer support)
1

Hello.

I’m using Graylog-3.1.0, but i had the same issue with 3.0.2.

I’ve must have make a mistake somewhere, but can’t find it. I have to manually recalculate index ranges each time an index is rotated (i only have the default index set), otherwise i can’t see new messages in either streams.

Disable index optimization after rotation is not checked in configuration.

Thanks for any pointers.

2

Did you manipulate ES outside of Graylog? Is that cluster using authentication or similar (based on x-pack or searchguard)? Did you see anything in your logs that indicates that something did not work as it should?

3

ES is used with grafana & kibana for metrics in another indices. I may have read once or twice the graylog indices with kibana and configure an index pattern for that.

Otherwise, no auth, and unfortunatly nothing in the logs (and i lost the logs in graylog 3.1, nothing in event streams or in graylog.log).

It does not happen when i rotate the index manually.

4

we would need to ability to reproduce that or identify in any way what happens.

Can you check the logs after/during rotation of the indices?

The complete Cycle should be something like:

2019-08-20T02:00:07.800+02:00 INFO  [AbstractRotationStrategy] Deflector index <Default index set> (index set <graylog_258>) should be rotated, Pointing deflector to new index now!
2019-08-20T02:00:07.805+02:00 INFO  [MongoIndexSet] Cycling from <graylog_258> to <graylog_259>.
2019-08-20T02:00:07.805+02:00 INFO  [MongoIndexSet] Creating target index <graylog_259>.
2019-08-20T02:00:07.825+02:00 INFO  [Indices] Successfully created index template graylog-internal
2019-08-20T02:00:07.933+02:00 INFO  [MongoIndexSet] Waiting for allocation of index <graylog_259>.
2019-08-20T02:00:07.938+02:00 INFO  [MongoIndexSet] Index <graylog_259> has been successfully allocated.
2019-08-20T02:00:07.938+02:00 INFO  [MongoIndexSet] Pointing index alias <graylog_deflector> to new index <graylog_259>.
2019-08-20T02:00:07.960+02:00 INFO  [SystemJobManager] Submitted SystemJob <78abb270-c2dd-11e9-9f34-7a9860de6ce3> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
2019-08-20T02:00:07.960+02:00 INFO  [MongoIndexSet] Successfully pointed index alias <graylog_deflector> to index <graylog_259>.

The System Job is what runs the index range calculation.

5

Logs are quite strange, I have nothing from august 4th to 21th, despite the fact my graylog_7 index has been create 4 days ago:
graylog_7 Contains messages from 4 days ago up to 2 hours ago

Hence, I have nothing for this rotation. Graylog 3.1.0 does not logs anymore in graylog-server.log. In the console log, I have the following, on august 21, when I updated graylog to 3.1, realize I did not have any log in streams and recalculate the ranges:
2019-08-21 16:29:25,891 INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <13a21e20-c420-11e9-ba76-0cc47a306f45> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
2019-08-21 16:29:25,891 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges.
2019-08-21 16:29:25,898 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges for index set Default index set (graylog_*): 6 indices affected.
2019-08-21 16:29:27,158 INFO : org.graylog2.indexer.ranges.MongoIndexRangeService - Calculated range of [graylog_6] in [1256ms].
2019-08-21 16:29:27,187 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Created ranges for index graylog_6: MongoIndexRange{id=null, indexName=graylog_6, begin=2019-08-04T03:52:00.720Z, end=2019-08-19T19:56:04.740Z, calculatedAt=2019-08-21T14:29:25.901Z, calculationDuration=1256, streamIds=[000000000000000000000001, 5d303decd6816b304090b0c6, 5d30408fd6816b304090b3ba]}
2019-08-21 16:29:28,555 INFO : org.graylog2.indexer.ranges.MongoIndexRangeService - Calculated range of [graylog_4] in [1365ms].
2019-08-21 16:29:28,556 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Created ranges for index graylog_4: MongoIndexRange{id=null, indexName=graylog_4, begin=2019-07-11T17:15:11.600Z, end=2019-07-22T17:55:25.844Z, calculatedAt=2019-08-21T14:29:27.189Z, calculationDuration=1365, streamIds=[000000000000000000000001, 5d039e38d6816b304060586f, 5d303decd6816b304090b0c6, 5d30408fd6816b304090b3ba, 5ce40155d6816b65fe037e36]}
2019-08-21 16:29:29,910 INFO : org.graylog2.indexer.ranges.MongoIndexRangeService - Calculated range of [graylog_3] in [1351ms].
2019-08-21 16:29:29,911 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Created ranges for index graylog_3: MongoIndexRange{id=null, indexName=graylog_3, begin=2019-07-02T07:14:20.792Z, end=2019-07-11T17:15:10.605Z, calculatedAt=2019-08-21T14:29:28.558Z, calculationDuration=1351, streamIds=[000000000000000000000001, 5d039e38d6816b304060586f, 5ce40155d6816b65fe037e36]}
2019-08-21 16:29:31,394 INFO : org.graylog2.indexer.ranges.MongoIndexRangeService - Calculated range of [graylog_5] in [1481ms].
2019-08-21 16:29:31,395 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Created ranges for index graylog_5: MongoIndexRange{id=null, indexName=graylog_5, begin=2019-07-22T17:55:26.298Z, end=2019-08-04T03:51:59.725Z, calculatedAt=2019-08-21T14:29:29.913Z, calculationDuration=1481, streamIds=[000000000000000000000001, 5d303decd6816b304090b0c6, 5d30408fd6816b304090b3ba, 5d4190e5d6816b3040a3778a]}
2019-08-21 16:29:32,780 INFO : org.graylog2.indexer.ranges.MongoIndexRangeService - Calculated range of [graylog_2] in [1383ms].
2019-08-21 16:29:32,781 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Created ranges for index graylog_2: MongoIndexRange{id=null, indexName=graylog_2, begin=1973-04-04T02:50:59.000Z, end=2019-07-02T07:14:19.807Z, calculatedAt=2019-08-21T14:29:31.397Z, calculationDuration=1383, streamIds=[000000000000000000000001, 5d039e38d6816b304060586f, 5ce40155d6816b65fe037e36, 5c112171d6816b11dc00af75]}
2019-08-21 16:29:32,783 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Done calculating index ranges for 6 indices. Took 6885ms.
2019-08-21 16:29:32,784 INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <13a21e20-c420-11e9-ba76-0cc47a306f45> [org.graylog2.indexer.ranges.RebuildIndexRangesJob] finished in 6892ms.

6

Could it be related to a restart of MongoDB without restarting graylog?

7

2019-08-21 16:29:32,783 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Done calculating index ranges for 6 indices. Took 6885ms.

  • What is the load of your system?
  • how many shards did you have in your elasticsearch?
  • do you share one system for everything?
8
  1. very very low, 0.4/0.2/0.2 for a 10 cores system
  2. 130 shards on a single machine
  3. yes, mongo+es+graylog
9

what kind of storage did you use? that is really long run for index recalc on such a system.

10

One spinning disk, SATA 4TB on XFS.

It’s a long run, but eventually succeed when i launch it manually

11

it might be I/O related - but I can’t say for sure.

From what you have written, it looks like the duration of the recalculation takes longer than it should.

12

From my point of view, it should fails when launched by hand too, but it does not.

However, I understand there is no configuration for this, I can’t have made a mistake changing a parameter?

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.