Wrong index range | Contains messages from value

avdhoot · January 29, 2018, 12:09pm

What will be reason behind wrong Contains messages value. Please check image below.
This index contain only yesterdays log. Already tried index rebuild option.

jan · January 29, 2018, 3:37pm

did you try to recalculate the index range?

jochen · January 29, 2018, 3:38pm

If this is the currently write-active index, everything is fine.

avdhoot · January 30, 2018, 5:57am

@jan tried recalculate the index range. same result. @jochen It is not active index.
When I search for 5 min of log graylog is searching in all index instead current active index . Please check below screenshot.

Screenshot_20180130_112521

jochen · January 30, 2018, 9:08am

Please post the complete screenshot of the System / Indices / Index Set page or (preferably) the output of the following commands (replace credentials and URL to your Graylog REST API accordingly):

$ curl -u admin:password -H 'Accept: application/json' http://graylog.example.org/api/system/indexer/indices?pretty=true
$ curl -u admin:password -H 'Accept: application/json' http://graylog.example.org/api/system/indices/ranges?pretty=true

Also make sure to check the logs of your Graylog node(s): http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

avdhoot · January 30, 2018, 9:39am

Please check attached gist for output.

gist.github.com

https://gist.github.com/avdhoot/b8363fe8ef67842db55d7e0a9db6c36b

indices.json

{
  "closed" : {
    "indices" : [ "graylog_512" ],
    "total" : 1
  },
  "reopened" : {
    "indices" : [ ],
    "total" : 0
  },
  "all" : {

This file has been truncated. show original

ranges.json

{
  "total" : 32,
  "ranges" : [ {
    "index_name" : "graylog_514",
    "begin" : "1970-01-01T00:00:00.000Z",
    "end" : "1970-01-01T00:00:00.000Z",
    "calculated_at" : "2017-12-31T00:09:30.173Z",
    "took_ms" : 0
  }, {
    "index_name" : "graylog_515",

This file has been truncated. show original

jochen · January 30, 2018, 10:04am

Please add the output of the following command (against your Elasticsearch nodes):

$ curl -H 'Accept: application/json' 'http://elasticsearch.example.org:9200/_mapping?pretty'

And to repeat what I wrote in my last post:

avdhoot · January 30, 2018, 10:19am

I do not see any error or warning in graylog logs. Please check mapping.json in gist.

gist.github.com

https://gist.github.com/avdhoot/b8363fe8ef67842db55d7e0a9db6c36b#file-mapping-json

indices.json

{
  "closed" : {
    "indices" : [ "graylog_512" ],
    "total" : 1
  },
  "reopened" : {
    "indices" : [ ],
    "total" : 0
  },
  "all" : {

This file has been truncated. show original

mapping.json

{
  "graylog_543" : {
    "mappings" : {
      "message" : {
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string"
            },

This file has been truncated. show original

ranges.json

{
  "total" : 32,
  "ranges" : [ {
    "index_name" : "graylog_514",
    "begin" : "1970-01-01T00:00:00.000Z",
    "end" : "1970-01-01T00:00:00.000Z",
    "calculated_at" : "2017-12-31T00:09:30.173Z",
    "took_ms" : 0
  }, {
    "index_name" : "graylog_515",

This file has been truncated. show original

jochen · January 30, 2018, 10:30am

The mapping for the “timestamp” field looks correct.

What happens, if you recalculate index ranges for all indices in your default index set? (System / Indices / Index Set / Maintenance)
What’s in the logs when you do that?

avdhoot · January 30, 2018, 1:53pm

Freshly started index recalculate job. I can see below traceback.

2018-01-30 13:47:43,997 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Could not calculate range of index [graylog_528]. Skipping.
org.graylog2.indexer.ElasticsearchException: Couldn’t build index range of index graylog_528
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:52) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.indexRangeStatsOfIndex(Indices.java:695) ~[graylog.jar:?]
at org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:142) ~[graylog.jar:?]
at org.graylog2.indexer.ranges.RebuildIndexRangesJob.execute(RebuildIndexRangesJob.java:138) [graylog.jar:?]
at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72-internal]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72-internal]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72-internal]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_72-internal]
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[graylog.jar:?]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[graylog.jar:?]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[graylog.jar:?]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[graylog.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[graylog.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:150) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:47) ~[graylog.jar:?]
… 13 more

jochen · January 30, 2018, 2:19pm

Please post the complete logs when triggering the recalculation of index ranges.

Which version of Elasticsearch are you using?

avdhoot · January 30, 2018, 2:33pm

elasticsearch version 2.3.5
Graylog version 2.3.0+81f8228

Here is complete log:-

gist.github.com

https://gist.github.com/avdhoot/f457aaa1773529f74a79bb3a0b3ada67

graylog_recalculation.log


2018-01-30 13:37:43,471 INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <bfd03dd0-05c2-11e8-b071-0242ac120002> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
2018-01-30 13:37:43,471 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges.
2018-01-30 13:37:43,476 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges for index set Default index set (graylog_*): 32 indices affected.
2018-01-30 13:38:15,232 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2018-01-30 13:38:15,234 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2018-01-30 13:38:38,724 WARN : org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor - AWS plugin is not fully configured. No instance lookups will happen.
2018-01-30 13:38:42,859 WARN : org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor - AWS plugin is not fully configured. No instance lookups will happen.
2018-01-30 13:38:42,860 WARN : org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor - AWS plugin is not fully configured. No instance lookups will happen.
2018-01-30 13:38:42,860 WARN : org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor - AWS plugin is not fully configured. No instance lookups will happen.

This file has been truncated. show original

jochen · January 30, 2018, 2:48pm

2018-01-30 14:08:45,676 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Could not calculate range of index [graylog_535]. Skipping.
org.graylog2.indexer.ElasticsearchException: Couldn't build index range of index graylog_535
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:52) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.indexRangeStatsOfIndex(Indices.java:695) ~[graylog.jar:?]
        at org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:142) ~[graylog.jar:?]
        [...]
Caused by: java.net.SocketTimeoutException: Read timed out

Your Elasticsearch cluster doesn’t respond in a timely fashion. You have several options for fixing (or “fixing”) this:

Throw more hardware at your Elasticsearch cluster (more RAM, SSDs)
Reduce the size of your individual indices (rotate more often)
Increase the Elasticsearch client timeout in Graylog:

github.com

Graylog2/graylog2-server/blob/2.3.0/misc/graylog.conf#L180-L188


      
          # Maximum amount of time to wait for successfull connection to Elasticsearch HTTP port.
          #
          # Default: 10 Seconds
          #elasticsearch_connect_timeout = 10s
          
          # Maximum amount of time to wait for reading back a response from an Elasticsearch server.
          #
          # Default: 60 seconds
          #elasticsearch_socket_timeout = 60s

For reference, the code calculating index ranges:

github.com

Graylog2/graylog2-server/blob/2.3.0/graylog2-server/src/main/java/org/graylog2/indexer/indices/Indices.java#L668-L718


      
          public IndexRangeStats indexRangeStatsOfIndex(String index) {
              final FilterAggregationBuilder builder = AggregationBuilders.filter("agg")
                      .filter(QueryBuilders.existsQuery(Message.FIELD_TIMESTAMP))
                      .subAggregation(AggregationBuilders.min("ts_min").field(Message.FIELD_TIMESTAMP))
                      .subAggregation(AggregationBuilders.max("ts_max").field(Message.FIELD_TIMESTAMP))
                      .subAggregation(AggregationBuilders.terms("streams").field(Message.FIELD_STREAMS));
              final String query = searchSource()
                      .aggregation(builder)
                      .size(0)
                      .toString();
          
              final Search request = new Search.Builder(query)
                      .addIndex(index)
                      .setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
                      .ignoreUnavailable(true)
                      .build();
          
              if (LOG.isDebugEnabled()) {
                  String data = "{}";
                  try {

This file has been truncated. show original

avdhoot · January 31, 2018, 6:53am

I have increased timeout in graylog. I have little experience with elasticsearch.

Our ES setup in aws:

We have 5 node cluster(1 master t2.small & 4 data node m4.4xlarge ).
Total size of index is 8TB (4TB without replica).
6TB gp2 attached to every data node.

From you feedback

I will reduce index size. Rotate hourly instead daily.
Upgrade ES to lastest version
after that if required I will add one more data node in ES cluster.

Let me know if I missing something or you need more info.

system · February 14, 2018, 6:53am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Index ranges wrong - manual recalculation yields dates in the future Graylog Central (peer support) pipeline-rules	13	1618	August 20, 2020
Messages not shown after reindex Graylog Tech Challenges	10	1726	July 23, 2021
Index range issue Graylog Central (peer support)	2	2474	June 5, 2019
Upgrade from 2.2x to 2.3 broke searches Graylog Central (peer support)	10	1311	September 6, 2017
Range recalculation on index rotation Graylog Central (peer support)	12	3403	September 9, 2019

Wrong index range | Contains messages from value

Related topics