Wrong index range | Contains messages from value

Hello @graylogDev/All

What will be reason behind wrong Contains messages value. Please check image below.
This index contain only yesterdays log. Already tried index rebuild option.

did you try to recalculate the index range?

If this is the currently write-active index, everything is fine.

@jan tried recalculate the index range. same result. @jochen It is not active index.
When I search for 5 min of log graylog is searching in all index instead current active index . Please check below screenshot.

Screenshot_20180130_112521

Please post the complete screenshot of the System / Indices / Index Set page or (preferably) the output of the following commands (replace credentials and URL to your Graylog REST API accordingly):

$ curl -u admin:password -H 'Accept: application/json' http://graylog.example.org/api/system/indexer/indices?pretty=true
$ curl -u admin:password -H 'Accept: application/json' http://graylog.example.org/api/system/indices/ranges?pretty=true

Also make sure to check the logs of your Graylog node(s): http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

Please check attached gist for output.

Please add the output of the following command (against your Elasticsearch nodes):

$ curl -H 'Accept: application/json' 'http://elasticsearch.example.org:9200/_mapping?pretty'

And to repeat what I wrote in my last post:

I do not see any error or warning in graylog logs. Please check mapping.json in gist.

The mapping for the “timestamp” field looks correct.

What happens, if you recalculate index ranges for all indices in your default index set? (System / Indices / Index Set / Maintenance)
What’s in the logs when you do that?

Freshly started index recalculate job. I can see below traceback.

2018-01-30 13:47:43,997 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Could not calculate range of index [graylog_528]. Skipping.
org.graylog2.indexer.ElasticsearchException: Couldn’t build index range of index graylog_528
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:52) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.indexRangeStatsOfIndex(Indices.java:695) ~[graylog.jar:?]
at org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:142) ~[graylog.jar:?]
at org.graylog2.indexer.ranges.RebuildIndexRangesJob.execute(RebuildIndexRangesJob.java:138) [graylog.jar:?]
at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72-internal]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72-internal]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72-internal]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_72-internal]
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[graylog.jar:?]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[graylog.jar:?]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[graylog.jar:?]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[graylog.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[graylog.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:150) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:47) ~[graylog.jar:?]
… 13 more

Please post the complete logs when triggering the recalculation of index ranges.

Which version of Elasticsearch are you using?

elasticsearch version 2.3.5
Graylog version 2.3.0+81f8228

Here is complete log:-

Your Elasticsearch cluster doesn’t respond in a timely fashion. You have several options for fixing (or “fixing”) this:

  • Throw more hardware at your Elasticsearch cluster (more RAM, SSDs)
  • Reduce the size of your individual indices (rotate more often)
  • Increase the Elasticsearch client timeout in Graylog:

For reference, the code calculating index ranges:

I have increased timeout in graylog. I have little experience with elasticsearch.

Our ES setup in aws:

  • We have 5 node cluster(1 master t2.small & 4 data node m4.4xlarge ).
  • Total size of index is 8TB (4TB without replica).
  • 6TB gp2 attached to every data node.

From you feedback

  • I will reduce index size. Rotate hourly instead daily.
  • Upgrade ES to lastest version
  • after that if required I will add one more data node in ES cluster.

Let me know if I missing something or you need more info.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.