Graylog is not searching

Dear Sirs,

if i make an search nothing will happens

Screenshot 2021-02-17 102307

In the Server.log i have this errors
021-02-17T10:12:56.601+01:00 WARN [IndexRotationThread] Deflector is pointing to [graylog_729], not the newest one: [graylog_730]. Re-pointing.
2021-02-17T10:13:32.276+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #12).
2021-02-17T10:13:56.657+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #2).
2021-02-17T10:14:03.396+01:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Couldn’t read health status for index graylog_730

    at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:95) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
    at org.graylog2.indexer.indices.Indices.waitForStatus(Indices.java:642) ~[graylog.jar:?]
    at org.graylog2.indexer.indices.Indices.waitForRecovery(Indices.java:633) ~[graylog.jar:?]
    at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:163) ~[graylog.jar:?]
    at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
    at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_161]
    at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
    at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_161]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

2021-02-17T10:14:56.716+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #15).
2021-02-17T10:15:32.893+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #13).
2021-02-17T10:15:56.777+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #3).
2021-02-17T10:16:34.421+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #16).
2021-02-17T10:17:07.205+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #14).
2021-02-17T10:17:36.753+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #4).
2021-02-17T10:18:05.464+01:00 WARN [IndexRotationThread] Deflector is pointing to [graylog_729], not the newest one: [graylog_730]. Re-pointing.
2021-02-17T10:19:05.498+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #17).
2021-02-17T10:19:08.513+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #15).
2021-02-17T10:19:36.665+01:00 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2021-02-17T10:20:08.567+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #5).
2021-02-17T10:20:08.716+01:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Couldn’t read health status for index graylog_730

    at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:95) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
    at org.graylog2.indexer.indices.Indices.waitForStatus(Indices.java:642) ~[graylog.jar:?]
    at org.graylog2.indexer.indices.Indices.waitForRecovery(Indices.java:633) ~[graylog.jar:?]
    at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:163) ~[graylog.jar:?]
    at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
    at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_161]
    at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
    at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_161]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

2021-02-17T10:21:08.625+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #18).
2021-02-17T10:21:12.146+01:00 ERROR [IndexerClusterCheckerThread] Uncaught exception in periodical
org.graylog2.indexer.ElasticsearchException: Unable to read Elasticsearch node information
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.Cluster.catNodes(Cluster.java:121) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.Cluster.getFileDescriptorStats(Cluster.java:126) ~[graylog.jar:?]
at org.graylog2.periodical.IndexerClusterCheckerThread.doRun(IndexerClusterCheckerThread.java:58) ~[graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_161]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_161]
at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_161]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_161]
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[graylog.jar:?]
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[graylog.jar:?]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[graylog.jar:?]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[graylog.jar:?]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[graylog.jar:?]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[graylog.jar:?]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[graylog.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[graylog.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
… 12 more
2021-02-17T10:22:08.677+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #16).
2021-02-17T10:22:12.197+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #6).

What does that means ?

Greets
Ralf

he @RalfThomas

it looks like your elasticsearch server is not up or reachable by Graylog.

Hallo,

ok thank you. Yes i read Elasticsearch . Now i make “sudo service elasticsearch start”

But Problem still exist.
qa-log:/var/log/graylog-server$ tail -n 100 server.log
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:77) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_161]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_161]
at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
… 28 more
2021-02-17T13:50:46.477+01:00 WARN [ProxiedResource] Unable to call http://127.0.0.1:12900/system/metrics/multiple on node <65e329a3-5a60-45b2-8460-9bd37051b1a8>
java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:230) ~[graylog.jar:?]
at okio.AsyncTimeout

What i see often : Unable to call http://127.0.0.1:12900/system/metrics/multiple on node <65e329a3-5a60-45b2-8460-9bd37051b1a8>
Greets
Ralf


By Elastic Search Status i see it is on Index 729 not 730. How can i move to the correct one ?

Greets
Ralf

When i check /var/log/elasticsearch$ tail -n 100 graylog.log

i geht many things : without Memory !

org.elasticsearch.index.shard.TranslogRecoveryPerformer.performRecoveryOperation(TranslogRecoveryPerformer.java:194)
at org.elasticsearch.index.shard.TranslogRecoveryPerformer.recoveryFromSnapshot(TranslogRecoveryPerformer.java:107)
at org.elasticsearch.index.shard.IndexShard$1.recoveryFromSnapshot(IndexShard.java:1582)
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:235)
… 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
[2021-02-17 13:59:27,138][WARN ][cluster.action.shard ] [Rush] [graylog_729][0] received shard failed for target shard [[graylog_729][0], node[un–O4QERTKBaLyuwqkUXQ], [P], v[1337], s[INITIALIZING], a[id=J1kNA6QDTcKRqbSug–BMA], unassigned_info[[reason=ALLOCATION_FAILED], at[2021-02-17T12:57:44.710Z], details[engine failure, reason [out of memory (source: [index])], failure OutOfMemoryError[Java heap space]]]], indexUUID [DpGZd52cScm21T1t6AXkMA], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [message#e60cf3e2-67a7-11eb-8e34-0050560202c7]]; nested: OutOfMemoryError[Java heap space]; ]
[graylog_729][[graylog_729][0]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [message#e60cf3e2-67a7-11eb-8e34-0050560202c7]]; nested: OutOfMemoryError[Java heap space];
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: [graylog_729][[graylog_729][0]] EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [message#e60cf3e2-67a7-11eb-8e34-0050560202c7]]; nested: OutOfMemoryError[Java heap space];
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:174)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1513)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1497)
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:970)
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:942)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
… 5 more
Caused by: [graylog_729][[graylog_729][0]] EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [message#e60cf3e2-67a7-11eb-8e34-0050560202c7]]; nested: OutOfMemoryError[Java heap space];
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:237)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:171)
… 11 more
Caused by: [graylog_729][[graylog_729][0]] IndexFailedEngineException[Index failed for [message#e60cf3e2-67a7-11eb-8e34-0050560202c7]]; nested: OutOfMemoryError[Java heap space];
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:459)
at org.elasticsearch.index.shard.TranslogRecoveryPerformer.performRecoveryOperation(TranslogRecoveryPerformer.java:194)
at org.elasticsearch.index.shard.TranslogRecoveryPerformer.recoveryFromSnapshot(TranslogRecoveryPerformer.java:107)
at org.elasticsearch.index.shard.IndexShard$1.recoveryFromSnapshot(IndexShard.java:1582)
at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:235)
… 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
[2021-02-17 13:59:27,195][WARN ][cluster.action.shard ] [Rush] [graylog_729][0] received shard failed for target shard [[graylog_729][0], node[un–O4QERTKBaLyuwqkUXQ], [P], v[1337], s[INITIALIZING], a[id=g_VeTSjoRKW_OjCwV2jVVA], unassigned_info[[reason=ALLOCATION_FAILED], at[2021-02-17T12:59:27.125Z], details[engine failure, reason [out of memory (source: [index])], failure OutOfMemoryError[Java heap s

I want change the Heap Size. But i not see this Entry

i set now the double size

After change is now Error away wit HEAP Size.
But i have still in Server.log
2021-02-17T14:31:30.068+01:00 WARN [IndexRotationThread] Deflector is pointing to [graylog_729], not the newest one: [graylog_730]. Re-pointing.
2021-02-17T14:31:50.100+01:00 ERROR [Messages] Caught exception during bulk indexing: java.net.SocketTimeoutException: Read timed out, retrying (attempt #3).
2021-02-17T14:32:25.029+01:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Couldn’t read health status for index graylog_730

    at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:95) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
    at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
    at org.graylog2.indexer.indices.Indices.waitForStatus(Indices.java:642) ~[graylog.jar:?]

How can i fix this ?

he @RalfThomas

your problem is Elasticsearch, not Graylog - at the moment.

So you need to raise the HEAP of Elasticsearch and not Graylog.

ok where can i change the HEAP of Elasticsearch ?

I not have under etc/elasticsearch the file jvm.options I have Ubuntu 16 Graylog 2.4

when i set in etc/default /graylog-server i think it is for Graylog and JVM together. Older Version !
I see this when i make bigger

he @RalfThomas

is that a new installation? if yes, you might want to change to a Version that gets security updates and has some bug fixes and works with newer elasticsearch versions.

Or did you have that version on purpose?

What Elasticsearch version are you using?

Jan

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.