Graylog Search keeps failing

[Justin19 (Justin C)]

Hello again, all.
So I seem to continue having the same problem.
I attempt to search my message logs, and it seems to randomly fail.
I checked Elasticsearch overview and everything is showing as

Here’s a screenshot of my overview page:
I am noticing that my admin user and graylog-server are on different time zones. NOTE: I left the graylog-server time zone at UTC

When my search fails, it gives me the following error: cannot GET http://192.168.2.131:9000/api/search/universal/relative?query=logon&range=604800&limit=150&sort=timestamp%3Adesc (500)

I’ve ran updates on my Ubuntu server, I’ve ran updates for java - my version is current, I’ve checked Elasticsearch index health. The only thing that looks like it may be a problem is the timezone, but I could be wrong about that. What’s strange is that the search and log indexing will work fine for a while and then fail. Also the firewall is off on our server that’s running the Graylog sidecar.

Here’s my Graylog config info:
is_master: true
node_id_file = /etc/graylog/server/node-id
password_secret =xxxxxxxx
root_password_sha2 =xxxxxxxx
root_timezone = UTC (by the way if anyone knows how to change this to California or Pacific time that would be awesome)
bin_dir = /usr/share-graylog-server/bin
http_bind_address = 192.168.2.131:9000
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

Here’s my Elasticsearch.yml info:
cluster.name: greylog
action.auto_create_index: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

Finally, here’s what’s listed in the Graylog server.log file:

019-05-17T16:17:28.106-07:00 ERROR [IndexFieldTypePollerPeriodical] Couldn’t update field types for index set <Default index set/5c9$
org.graylog2.indexer.ElasticsearchException: Couldn’t collect indices for alias graylog_deflector
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) $
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1$
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_21$
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
… 11 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connecti$
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[gray$
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylo$
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
… 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_212]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_212]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_212]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_212]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_212]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_212]
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar$
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[gray$
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylo$
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
… 11 more
2019-05-17T16:17:28.826-07:00 ERROR [Cluster] Couldn’t read cluster health for indices [graylog_*] (Could not connect to http://127.0$
2019-05-17T16:17:28.827-07:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.

Here’s what I get when I check the Elasticsearch status when the search fails
root@ladmin-GreylogDell-3020:/var/log/graylog-server# service elasticsearch statuss

• Usage: /etc/init.d/elasticsearch {start|stop|restart|force-reload|status}
  root@ladmin-GreylogDell-3020:/var/log/graylog-server# service elasticsearch status
  ● elasticsearch.service - Elasticsearch
  Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
  Active: failed (Result: exit-code) since Fri 2019-05-17 16:12:50 PDT; 29min ago
  Docs: http://www.elastic.co
  Process: 989 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=127)
  Main PID: 989 (code=exited, status=127)

May 17 15:48:46 ladmin-GreylogDell-3020 systemd[1]: Started Elasticsearch.
May 17 15:48:46 ladmin-GreylogDell-3020 elasticsearch[989]: warning: Falling back to java on path. This behavior is deprecated. Specif
May 17 16:12:50 ladmin-GreylogDell-3020 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=127/n/a
May 17 16:12:50 ladmin-GreylogDell-3020 systemd[1]: elasticsearch.service: Failed with result ‘exit-code’.

Thanks for any help

[michael.ehrenreich]

Graylog cannot connect Elasticsearch, because Elasticsearch is not running. I’m sure, the logs inside /var/log/elasticsearch will give you a better insight than the brief systemd status.

[Justin19 (Justin C)]

I will check the log and post back the log readout. The weird thing is that Elasticsearch will run after I reboot the system, I can search message/view messages…but then, almost randomly I will do a search and the search will fail and I’ll get that (500) error. So something is causing Elasticsearch to either crash or fail in it’s connection to Graylog.

[Tarequl (Md.Tarequl Islam)]

Hello Dear All,
I configured gray-log server , Mongodb, Elasticsearch and java for running graylog server. i am facing problem when i gave input, after that when i try to search in search bar that time it’s loading continuously. what should i do now?

[jan (Jan Doberstein)]

please check all Graylog parts if they work proper and if Graylog is able to communicate with Elasticsearch and error in your server.log.

[Justin19 (Justin C)]

Hello Michael. BTW…thanks for the quick reply on my previous question.

So I checked the elasticsearch log for one of the days I ran into problems, and here’s what I found – Basically it’s showing “Java overhead spent, Java.lang.OutOfMemoryError: Java heap space”
Which makes sense…because the crashing and disconnection from the database is abrupt and seems a bit random. Is the next step to add more memory? Or should I can we control what’s allocated to the Java heap space?

Below are the actual log messagesk for var/log/elasticsearch/elasticsearch-2019-05-17-1.log

[2019-05-17T16:12:50,083][WARN ][o.e.m.j.JvmGcMonitorService] [Utejpt2] [gc][1423] overhead, spent [1.3s] collecting in the last [1.3s]
[2019-05-17T16:12:48,778][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [Utejpt2] fatal error in thread [Thread-5], exiting
java.lang.OutOfMemoryError: Java heap space
** at io.netty.util.internal.PlatformDependent.allocateUninitializedArray(PlatformDependent.java:204) ~[?:?]**
** at io.netty.buffer.PoolArena$HeapArena.newByteArray(PoolArena.java:676) ~[?:?]**
** at io.netty.buffer.PoolArena$HeapArena.newChunk(PoolArena.java:686) ~[?:?]**
** at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[?:?]**
** at io.netty.buffer.PoolArena.allocate(PoolArena.java:226) ~[?:?]**
** at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[?:?]**
** at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:307) ~[?:?]**
** at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:166) ~[?:?]**
** at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:157) ~[?:?]**
** at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139) ~[?:?]**
** at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) ~[?:?]**
** at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147) ~[?:?]**
** at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[?:?]**
** at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]**
** at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]**
** at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]**
** at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]**
** at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]**

[jan (Jan Doberstein)]

how much RAM does your server has and what is the configured JVM HEAP size?

http://docs.graylog.org/en/3.0/pages/configuration/file_location.html

[Justin19 (Justin C)]

Hey Jan.
I reset the heap size defaults to 4GB for both initial and total heap space.
My system is running a total of 16GB of RAM

Here’s what I set the heap size to in the /etc/elasticsearch/java.options:

Xms represents the initial size of total heap space

Xmx represents the maximum size of total heap space

-Xms4g
-Xmx4g

Here’s what it shows as the default settings for java in /etc/default/graylog-server

Path to the java executable.

JAVA=/usr/bin/java

Default Java options for heap and garbage collection.

GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:$

Pass some extra args to graylog-server. (i.e. “-d” to enable debug mode)

GRAYLOG_SERVER_ARGS=“”

Program that will be used to wrap the graylog-server command. Useful to

support programs like authbind.

GRAYLOG_COMMAND_WRAPPER=“”

[jan (Jan Doberstein)]

if you get the OOM error from Elasticsearch, you need to add more RAM to the Server and Raise the HEAP for Elasticsearch.