Hello @graylogDev/All
What will be reason behind wrong Contains messages value. Please check image below.
This index contain only yesterdays log. Already tried index rebuild option.
Hello @graylogDev/All
What will be reason behind wrong Contains messages value. Please check image below.
This index contain only yesterdays log. Already tried index rebuild option.
@jan tried recalculate the index range. same result. @jochen It is not active index.
When I search for 5 min of log graylog is searching in all index instead current active index . Please check below screenshot.
Please post the complete screenshot of the System / Indices / Index Set page or (preferably) the output of the following commands (replace credentials and URL to your Graylog REST API accordingly):
$ curl -u admin:password -H 'Accept: application/json' http://graylog.example.org/api/system/indexer/indices?pretty=true
$ curl -u admin:password -H 'Accept: application/json' http://graylog.example.org/api/system/indices/ranges?pretty=true
Also make sure to check the logs of your Graylog node(s): http://docs.graylog.org/en/2.4/pages/configuration/file_location.html
Please add the output of the following command (against your Elasticsearch nodes):
$ curl -H 'Accept: application/json' 'http://elasticsearch.example.org:9200/_mapping?pretty'
And to repeat what I wrote in my last post:
I do not see any error or warning in graylog logs. Please check mapping.json in gist.
The mapping for the “timestamp” field looks correct.
What happens, if you recalculate index ranges for all indices in your default index set? (System / Indices / Index Set / Maintenance)
What’s in the logs when you do that?
Freshly started index recalculate job. I can see below traceback.
2018-01-30 13:47:43,997 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Could not calculate range of index [graylog_528]. Skipping.
org.graylog2.indexer.ElasticsearchException: Couldn’t build index range of index graylog_528
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:52) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:63) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.indexRangeStatsOfIndex(Indices.java:695) ~[graylog.jar:?]
at org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:142) ~[graylog.jar:?]
at org.graylog2.indexer.ranges.RebuildIndexRangesJob.execute(RebuildIndexRangesJob.java:138) [graylog.jar:?]
at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72-internal]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72-internal]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72-internal]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_72-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_72-internal]
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[graylog.jar:?]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[graylog.jar:?]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[graylog.jar:?]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[graylog.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[graylog.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:150) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:47) ~[graylog.jar:?]
… 13 more
Please post the complete logs when triggering the recalculation of index ranges.
Which version of Elasticsearch are you using?
elasticsearch version 2.3.5
Graylog version 2.3.0+81f8228
Here is complete log:-
Your Elasticsearch cluster doesn’t respond in a timely fashion. You have several options for fixing (or “fixing”) this:
For reference, the code calculating index ranges:
I have increased timeout in graylog. I have little experience with elasticsearch.
Our ES setup in aws:
From you feedback
Let me know if I missing something or you need more info.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.