Graylog is not searching

Graylog Central (peer support)
1

Dear Sirs,

if i make an search nothing will happens

Screenshot 2021-02-17 102307

In the Server.log i have this errors
021-02-17T10:12:56.601+01:00 WARN [IndexRotationThread] Deflector is pointing to [graylog_729], not the newest one: [graylog_730]. Re-pointing.
2021-02-17T10:13:32.276+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #12).
2021-02-17T10:13:56.657+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #2).
2021-02-17T10:14:03.396+01:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Couldn’t read health status for index graylog_730

    at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:95) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
    at org.graylog2.indexer.indices.Indices.waitForStatus(Indices.java:642) ~[graylog.jar:?]
    at org.graylog2.indexer.indices.Indices.waitForRecovery(Indices.java:633) ~[graylog.jar:?]
    at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:163) ~[graylog.jar:?]
    at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
    at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_161]
    at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
    at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_161]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

2021-02-17T10:14:56.716+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #15).
2021-02-17T10:15:32.893+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #13).
2021-02-17T10:15:56.777+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #3).
2021-02-17T10:16:34.421+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #16).
2021-02-17T10:17:07.205+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #14).
2021-02-17T10:17:36.753+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #4).
2021-02-17T10:18:05.464+01:00 WARN [IndexRotationThread] Deflector is pointing to [graylog_729], not the newest one: [graylog_730]. Re-pointing.
2021-02-17T10:19:05.498+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #17).
2021-02-17T10:19:08.513+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #15).
2021-02-17T10:19:36.665+01:00 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2021-02-17T10:20:08.567+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #5).
2021-02-17T10:20:08.716+01:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Couldn’t read health status for index graylog_730

    at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:95) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
    at org.graylog2.indexer.indices.Indices.waitForStatus(Indices.java:642) ~[graylog.jar:?]
    at org.graylog2.indexer.indices.Indices.waitForRecovery(Indices.java:633) ~[graylog.jar:?]
    at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:163) ~[graylog.jar:?]
    at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
    at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_161]
    at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
    at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_161]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

2021-02-17T10:21:08.625+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #18).
2021-02-17T10:21:12.146+01:00 ERROR [IndexerClusterCheckerThread] Uncaught exception in periodical
org.graylog2.indexer.ElasticsearchException: Unable to read Elasticsearch node information
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.Cluster.catNodes(Cluster.java:121) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.Cluster.getFileDescriptorStats(Cluster.java:126) ~[graylog.jar:?]
at org.graylog2.periodical.IndexerClusterCheckerThread.doRun(IndexerClusterCheckerThread.java:58) ~[graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_161]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_161]
at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_161]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_161]
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[graylog.jar:?]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[graylog.jar:?]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[graylog.jar:?]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[graylog.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[graylog.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
… 12 more
2021-02-17T10:22:08.677+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #16).
2021-02-17T10:22:12.197+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #6).

What does that means ?

Greets
Ralf

2

he @RalfThomas

it looks like your elasticsearch server is not up or reachable by Graylog.

3

Hallo,

ok thank you. Yes i read Elasticsearch . Now i make “sudo service elasticsearch start”

Screenshot 2021-02-17 135140
Screenshot 2021-02-17 135140961×341 13.8 KB

But Problem still exist.
qa-log:/var/log/graylog-server$ tail -n 100 server.log
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:77) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_161]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_161]
at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
… 28 more
2021-02-17T13:50:46.477+01:00 WARN [ProxiedResource] Unable to call http://127.0.0.1:12900/system/metrics/multiple on node <65e329a3-5a60-45b2-8460-9bd37051b1a8>
java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:230) ~[graylog.jar:?]
at okio.AsyncTimeout

What i see often : Unable to call http://127.0.0.1:12900/system/metrics/multiple on node <65e329a3-5a60-45b2-8460-9bd37051b1a8>
Greets
Ralf

4

Screenshot 2021-02-17 135825
Screenshot 2021-02-17 1358251296×338 33.8 KB

By Elastic Search Status i see it is on Index 729 not 730. How can i move to the correct one ?

Greets
Ralf

5

When i check /var/log/elasticsearch$ tail -n 100 graylog.log

i geht many things : without Memory !

org.elasticsearch.index.shard.TranslogRecoveryPerformer.performRecoveryOperation(TranslogRecoveryPerformer.java:194)
at org.elasticsearch.index.shard.TranslogRecoveryPerformer.recoveryFromSnapshot(TranslogRecoveryPerformer.java:107)
at org.elasticsearch.index.shard.IndexShard$1.recoveryFromSnapshot(IndexShard.java:1582)
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:235)
… 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
[2021-02-17 13:59:27,138][WARN ][cluster.action.shard ] [Rush] [graylog_729][0] received shard failed for target shard [[graylog_729][0], node[un–O4QERTKBaLyuwqkUXQ], [P], v[1337], s[INITIALIZING], a[id=J1kNA6QDTcKRqbSug–BMA], unassigned_info[[reason=ALLOCATION_FAILED], at[2021-02-17T12:57:44.710Z], details[engine failure, reason [out of memory (source: [index])], failure OutOfMemoryError[Java heap space]]]], indexUUID [DpGZd52cScm21T1t6AXkMA], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [message#e60cf3e2-67a7-11eb-8e34-0050560202c7]]; nested: OutOfMemoryError[Java heap space]; ]
[graylog_729][[graylog_729][0]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [message#e60cf3e2-67a7-11eb-8e34-0050560202c7]]; nested: OutOfMemoryError[Java heap space];
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: [graylog_729][[graylog_729][0]] EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [message#e60cf3e2-67a7-11eb-8e34-0050560202c7]]; nested: OutOfMemoryError[Java heap space];
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:174)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1513)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1497)
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:970)
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:942)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
… 5 more
Caused by: [graylog_729][[graylog_729][0]] EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [message#e60cf3e2-67a7-11eb-8e34-0050560202c7]]; nested: OutOfMemoryError[Java heap space];
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:237)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:171)
… 11 more
Caused by: [graylog_729][[graylog_729][0]] IndexFailedEngineException[Index failed for [message#e60cf3e2-67a7-11eb-8e34-0050560202c7]]; nested: OutOfMemoryError[Java heap space];
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:459)
at org.elasticsearch.index.shard.TranslogRecoveryPerformer.performRecoveryOperation(TranslogRecoveryPerformer.java:194)
at org.elasticsearch.index.shard.TranslogRecoveryPerformer.recoveryFromSnapshot(TranslogRecoveryPerformer.java:107)
at org.elasticsearch.index.shard.IndexShard$1.recoveryFromSnapshot(IndexShard.java:1582)
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:235)
… 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
[2021-02-17 13:59:27,195][WARN ][cluster.action.shard ] [Rush] [graylog_729][0] received shard failed for target shard [[graylog_729][0], node[un–O4QERTKBaLyuwqkUXQ], [P], v[1337], s[INITIALIZING], a[id=g_VeTSjoRKW_OjCwV2jVVA], unassigned_info[[reason=ALLOCATION_FAILED], at[2021-02-17T12:59:27.125Z], details[engine failure, reason [out of memory (source: [index])], failure OutOfMemoryError[Java heap s

6

I want change the Heap Size. But i not see this Entry

Screenshot 2021-02-17 141808
Screenshot 2021-02-17 1418081291×526 17.2 KB

7

i set now the double size

Unbenannt
Unbenannt1644×210 7.15 KB

8

Screenshot 2021-02-17 144053
Screenshot 2021-02-17 144053979×351 32.6 KB

After change is now Error away wit HEAP Size.
But i have still in Server.log
2021-02-17T14:31:30.068+01:00 WARN [IndexRotationThread] Deflector is pointing to [graylog_729], not the newest one: [graylog_730]. Re-pointing.
2021-02-17T14:31:50.100+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #3).
2021-02-17T14:32:25.029+01:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Couldn’t read health status for index graylog_730

    at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:95) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
    at org.graylog2.indexer.indices.Indices.waitForStatus(Indices.java:642) ~[graylog.jar:?]

How can i fix this ?

9

he @RalfThomas

your problem is Elasticsearch, not Graylog - at the moment.

So you need to raise the HEAP of Elasticsearch and not Graylog.

10

ok where can i change the HEAP of Elasticsearch ?

11

I not have under etc/elasticsearch the file jvm.options I have Ubuntu 16 Graylog 2.4

12

when i set in etc/default /graylog-server i think it is for Graylog and JVM together. Older Version !
I see this when i make bigger

Screenshot 2021-02-17 153310
Screenshot 2021-02-17 1533101260×785 168 KB

13

he @RalfThomas

is that a new installation? if yes, you might want to change to a Version that gets security updates and has some bug fixes and works with newer elasticsearch versions.

Or did you have that version on purpose?

What Elasticsearch version are you using?

Jan

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.