Wrong index range | Contains messages from value

avdhoot · 2018-01-29 12:09:48 UTC

What will be reason behind wrong Contains messages value. Please check image below.
This index contain only yesterdays log. Already tried index rebuild option.

jan · 2018-01-29 15:37:28 UTC

did you try to recalculate the index range?

jochen · 2018-01-29 15:38:11 UTC

If this is the currently write-active index, everything is fine.

avdhoot · 2018-01-30 05:57:39 UTC

@jan tried recalculate the index range. same result. @jochen It is not active index.
When I search for 5 min of log graylog is searching in all index instead current active index . Please check below screenshot.

Screenshot_20180130_112521

jochen · 2018-01-30 09:08:50 UTC

Please post the complete screenshot of the System / Indices / Index Set page or (preferably) the output of the following commands (replace credentials and URL to your Graylog REST API accordingly):

$ curl -u admin:password -H 'Accept: application/json' http://graylog.example.org/api/system/indexer/indices?pretty=true
$ curl -u admin:password -H 'Accept: application/json' http://graylog.example.org/api/system/indices/ranges?pretty=true

Also make sure to check the logs of your Graylog node(s): http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

avdhoot · 2018-01-30 09:39:55 UTC

Please check attached gist for output.

gist.github.com

https://gist.github.com/avdhoot/b8363fe8ef67842db55d7e0a9db6c36b

indices.json

{
  "closed" : {
    "indices" : [ "graylog_512" ],
    "total" : 1
  },
  "reopened" : {
    "indices" : [ ],
    "total" : 0
  },
  "all" : {

This file has been truncated. show original

ranges.json

{
  "total" : 32,
  "ranges" : [ {
    "index_name" : "graylog_514",
    "begin" : "1970-01-01T00:00:00.000Z",
    "end" : "1970-01-01T00:00:00.000Z",
    "calculated_at" : "2017-12-31T00:09:30.173Z",
    "took_ms" : 0
  }, {
    "index_name" : "graylog_515",

This file has been truncated. show original

jochen · 2018-01-30 10:04:56 UTC

Please add the output of the following command (against your Elasticsearch nodes):

$ curl -H 'Accept: application/json' 'http://elasticsearch.example.org:9200/_mapping?pretty'

And to repeat what I wrote in my last post:

avdhoot · 2018-01-30 10:19:41 UTC

I do not see any error or warning in graylog logs. Please check mapping.json in gist.

gist.github.com

https://gist.github.com/avdhoot/b8363fe8ef67842db55d7e0a9db6c36b#file-mapping-json

indices.json

{
  "closed" : {
    "indices" : [ "graylog_512" ],
    "total" : 1
  },
  "reopened" : {
    "indices" : [ ],
    "total" : 0
  },
  "all" : {

This file has been truncated. show original

mapping.json

{
  "graylog_543" : {
    "mappings" : {
      "message" : {
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string"
            },

This file has been truncated. show original

ranges.json

{
  "total" : 32,
  "ranges" : [ {
    "index_name" : "graylog_514",
    "begin" : "1970-01-01T00:00:00.000Z",
    "end" : "1970-01-01T00:00:00.000Z",
    "calculated_at" : "2017-12-31T00:09:30.173Z",
    "took_ms" : 0
  }, {
    "index_name" : "graylog_515",

This file has been truncated. show original

jochen · 2018-01-30 10:30:41 UTC

The mapping for the “timestamp” field looks correct.

What happens, if you recalculate index ranges for all indices in your default index set? (System / Indices / Index Set / Maintenance)
What’s in the logs when you do that?

avdhoot · 2018-01-30 13:53:22 UTC

Freshly started index recalculate job. I can see below traceback.

2018-01-30 13:47:43,997 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Could not calculate range of index [graylog_528]. Skipping.
org.graylog2.indexer.ElasticsearchException: Couldn’t build index range of index graylog_528
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:52) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.indexRangeStatsOfIndex(Indices.java:695) ~[graylog.jar:?]
at org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:142) ~[graylog.jar:?]
at org.graylog2.indexer.ranges.RebuildIndexRangesJob.execute(RebuildIndexRangesJob.java:138) [graylog.jar:?]
at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72-internal]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72-internal]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72-internal]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_72-internal]
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[graylog.jar:?]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[graylog.jar:?]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[graylog.jar:?]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[graylog.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[graylog.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:150) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:47) ~[graylog.jar:?]
… 13 more

jochen · 2018-01-30 14:19:11 UTC

Please post the complete logs when triggering the recalculation of index ranges.

Which version of Elasticsearch are you using?

avdhoot · 2018-01-30 14:33:14 UTC

elasticsearch version 2.3.5
Graylog version 2.3.0+81f8228

Here is complete log:-

gist.github.com

https://gist.github.com/avdhoot/f457aaa1773529f74a79bb3a0b3ada67

graylog_recalculation.log


2018-01-30 13:37:43,471 INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <bfd03dd0-05c2-11e8-b071-0242ac120002> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
2018-01-30 13:37:43,471 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges.
2018-01-30 13:37:43,476 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges for index set Default index set (graylog_*): 32 indices affected.
2018-01-30 13:38:15,232 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2018-01-30 13:38:15,234 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2018-01-30 13:38:38,724 WARN : org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor - AWS plugin is not fully configured. No instance lookups will happen.
2018-01-30 13:38:42,859 WARN : org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor - AWS plugin is not fully configured. No instance lookups will happen.
2018-01-30 13:38:42,860 WARN : org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor - AWS plugin is not fully configured. No instance lookups will happen.
2018-01-30 13:38:42,860 WARN : org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor - AWS plugin is not fully configured. No instance lookups will happen.

This file has been truncated. show original

jochen · 2018-01-30 14:48:16 UTC

2018-01-30 14:08:45,676 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Could not calculate range of index [graylog_535]. Skipping.
org.graylog2.indexer.ElasticsearchException: Couldn't build index range of index graylog_535
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:52) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.indexRangeStatsOfIndex(Indices.java:695) ~[graylog.jar:?]
        at org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:142) ~[graylog.jar:?]
        [...]
Caused by: java.net.SocketTimeoutException: Read timed out

Your Elasticsearch cluster doesn’t respond in a timely fashion. You have several options for fixing (or “fixing”) this:

Throw more hardware at your Elasticsearch cluster (more RAM, SSDs)
Reduce the size of your individual indices (rotate more often)
Increase the Elasticsearch client timeout in Graylog:

github.com

Graylog2/graylog2-server/blob/2.3.0/misc/graylog.conf#L180-L188


# Maximum amount of time to wait for successfull connection to Elasticsearch HTTP port.
#
# Default: 10 Seconds
#elasticsearch_connect_timeout = 10s


# Maximum amount of time to wait for reading back a response from an Elasticsearch server.
#
# Default: 60 seconds
#elasticsearch_socket_timeout = 60s

For reference, the code calculating index ranges:

github.com

Graylog2/graylog2-server/blob/2.3.0/graylog2-server/src/main/java/org/graylog2/indexer/indices/Indices.java#L668-L718


public IndexRangeStats indexRangeStatsOfIndex(String index) {
    final FilterAggregationBuilder builder = AggregationBuilders.filter("agg")
            .filter(QueryBuilders.existsQuery(Message.FIELD_TIMESTAMP))
            .subAggregation(AggregationBuilders.min("ts_min").field(Message.FIELD_TIMESTAMP))
            .subAggregation(AggregationBuilders.max("ts_max").field(Message.FIELD_TIMESTAMP))
            .subAggregation(AggregationBuilders.terms("streams").field(Message.FIELD_STREAMS));
    final String query = searchSource()
            .aggregation(builder)
            .size(0)
            .toString();


    final Search request = new Search.Builder(query)
            .addIndex(index)
            .setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
            .ignoreUnavailable(true)
            .build();


    if (LOG.isDebugEnabled()) {
        String data = "{}";
        try {

This file has been truncated. show original

avdhoot · 2018-01-31 06:53:46 UTC

I have increased timeout in graylog. I have little experience with elasticsearch.

Our ES setup in aws:

We have 5 node cluster(1 master t2.small & 4 data node m4.4xlarge ).
Total size of index is 8TB (4TB without replica).
6TB gp2 attached to every data node.

From you feedback

I will reduce index size. Rotate hourly instead daily.
Upgrade ES to lastest version
after that if required I will add one more data node in ES cluster.

Let me know if I missing something or you need more info.

system · 2018-02-14 06:53:56 UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.