Wrong index range | Contains messages from value


(Avdhoot) #1

Hello @graylogDev/All

What will be reason behind wrong Contains messages value. Please check image below.
This index contain only yesterdays log. Already tried index rebuild option.


(Jan Doberstein) #2

did you try to recalculate the index range?


(Jochen) #3

If this is the currently write-active index, everything is fine.


(Avdhoot) #4

@jan tried recalculate the index range. same result. @jochen It is not active index.
When I search for 5 min of log graylog is searching in all index instead current active index . Please check below screenshot.

Screenshot_20180130_112521


(Jochen) #5

Please post the complete screenshot of the System / Indices / Index Set page or (preferably) the output of the following commands (replace credentials and URL to your Graylog REST API accordingly):

$ curl -u admin:password -H 'Accept: application/json' http://graylog.example.org/api/system/indexer/indices?pretty=true
$ curl -u admin:password -H 'Accept: application/json' http://graylog.example.org/api/system/indices/ranges?pretty=true

Also make sure to check the logs of your Graylog node(s): http://docs.graylog.org/en/2.4/pages/configuration/file_location.html


(Avdhoot) #6

Please check attached gist for output.


(Jochen) #7

Please add the output of the following command (against your Elasticsearch nodes):

$ curl -H 'Accept: application/json' 'http://elasticsearch.example.org:9200/_mapping?pretty'

And to repeat what I wrote in my last post:


(Avdhoot) #8

I do not see any error or warning in graylog logs. Please check mapping.json in gist.


(Jochen) #9

The mapping for the “timestamp” field looks correct.

What happens, if you recalculate index ranges for all indices in your default index set? (System / Indices / Index Set / Maintenance)
What’s in the logs when you do that?


(Avdhoot) #10

Freshly started index recalculate job. I can see below traceback.

2018-01-30 13:47:43,997 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Could not calculate range of index [graylog_528]. Skipping.
org.graylog2.indexer.ElasticsearchException: Couldn’t build index range of index graylog_528
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:52) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.indexRangeStatsOfIndex(Indices.java:695) ~[graylog.jar:?]
at org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:142) ~[graylog.jar:?]
at org.graylog2.indexer.ranges.RebuildIndexRangesJob.execute(RebuildIndexRangesJob.java:138) [graylog.jar:?]
at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72-internal]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72-internal]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72-internal]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_72-internal]
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[graylog.jar:?]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[graylog.jar:?]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[graylog.jar:?]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[graylog.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[graylog.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:150) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:47) ~[graylog.jar:?]
… 13 more


(Jochen) #11

Please post the complete logs when triggering the recalculation of index ranges.

Which version of Elasticsearch are you using?


(Avdhoot) #12

elasticsearch version 2.3.5
Graylog version 2.3.0+81f8228

Here is complete log:-


(Jochen) #13

Your Elasticsearch cluster doesn’t respond in a timely fashion. You have several options for fixing (or “fixing”) this:

  • Throw more hardware at your Elasticsearch cluster (more RAM, SSDs)
  • Reduce the size of your individual indices (rotate more often)
  • Increase the Elasticsearch client timeout in Graylog:

For reference, the code calculating index ranges:


(Avdhoot) #14

I have increased timeout in graylog. I have little experience with elasticsearch.

Our ES setup in aws:

  • We have 5 node cluster(1 master t2.small & 4 data node m4.4xlarge ).
  • Total size of index is 8TB (4TB without replica).
  • 6TB gp2 attached to every data node.

From you feedback

  • I will reduce index size. Rotate hourly instead daily.
  • Upgrade ES to lastest version
  • after that if required I will add one more data node in ES cluster.

Let me know if I missing something or you need more info.


(system) #15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.