Graylog Search keeps failing

Graylog Central (peer support)
1

Hello again, all.
So I seem to continue having the same problem.
I attempt to search my message logs, and it seems to randomly fail.
I checked Elasticsearch overview and everything is showing as

Here’s a screenshot of my overview page:
I am noticing that my admin user and graylog-server are on different time zones. NOTE: I left the graylog-server time zone at UTC

image
image.png1269×304 23.3 KB

When my search fails, it gives me the following error: cannot GET http://192.168.2.131:9000/api/search/universal/relative?query=logon&range=604800&limit=150&sort=timestamp%3Adesc (500)

I’ve ran updates on my Ubuntu server, I’ve ran updates for java - my version is current, I’ve checked Elasticsearch index health. The only thing that looks like it may be a problem is the timezone, but I could be wrong about that. What’s strange is that the search and log indexing will work fine for a while and then fail. Also the firewall is off on our server that’s running the Graylog sidecar.

Here’s my Graylog config info:
is_master: true
node_id_file = /etc/graylog/server/node-id
password_secret =xxxxxxxx
root_password_sha2 =xxxxxxxx
root_timezone = UTC (by the way if anyone knows how to change this to California or Pacific time that would be awesome)
bin_dir = /usr/share-graylog-server/bin
http_bind_address = 192.168.2.131:9000
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

Here’s my Elasticsearch.yml info:
cluster.name: greylog
action.auto_create_index: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

Finally, here’s what’s listed in the Graylog server.log file:

019-05-17T16:17:28.106-07:00 ERROR [IndexFieldTypePollerPeriodical] Couldn’t update field types for index set <Default index set/5c9$
org.graylog2.indexer.ElasticsearchException: Couldn’t collect indices for alias graylog_deflector
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:335) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:204) ~[graylog.jar:?]
at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:201) $
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1$
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_21$
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:80) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
… 11 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:9200 [/127.0.0.1] failed: Connection refused (Connecti$
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[gray$
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylo$
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
… 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_212]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_212]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_212]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_212]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_212]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_212]
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[graylog.jar$
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[gray$
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[graylo$
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[graylog.jar:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[graylog.jar:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[graylog.jar:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
… 11 more
2019-05-17T16:17:28.826-07:00 ERROR [Cluster] Couldn’t read cluster health for indices [graylog_*] (Could not connect to http://127.0$
2019-05-17T16:17:28.827-07:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.

Here’s what I get when I check the Elasticsearch status when the search fails
root@ladmin-GreylogDell-3020:/var/log/graylog-server# service elasticsearch statuss

  • Usage: /etc/init.d/elasticsearch {start|stop|restart|force-reload|status}
    root@ladmin-GreylogDell-3020:/var/log/graylog-server# service elasticsearch status
    ● elasticsearch.service - Elasticsearch
    Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
    Active: failed (Result: exit-code) since Fri 2019-05-17 16:12:50 PDT; 29min ago
    Docs: http://www.elastic.co
    Process: 989 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=127)
    Main PID: 989 (code=exited, status=127)

May 17 15:48:46 ladmin-GreylogDell-3020 systemd[1]: Started Elasticsearch.
May 17 15:48:46 ladmin-GreylogDell-3020 elasticsearch[989]: warning: Falling back to java on path. This behavior is deprecated. Specif
May 17 16:12:50 ladmin-GreylogDell-3020 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=127/n/a
May 17 16:12:50 ladmin-GreylogDell-3020 systemd[1]: elasticsearch.service: Failed with result ‘exit-code’.

Thanks for any help :slight_smile:

2

Graylog cannot connect Elasticsearch, because Elasticsearch is not running. I’m sure, the logs inside /var/log/elasticsearch will give you a better insight than the brief systemd status.

3

I will check the log and post back the log readout. The weird thing is that Elasticsearch will run after I reboot the system, I can search message/view messages…but then, almost randomly I will do a search and the search will fail and I’ll get that (500) error. So something is causing Elasticsearch to either crash or fail in it’s connection to Graylog.

4

Hello Dear All,
I configured gray-log server , Mongodb, Elasticsearch and java for running graylog server. i am facing problem when i gave input, after that when i try to search in search bar that time it’s loading continuously. what should i do now?

Graylog
Graylog.PNG1328×589 35.7 KB

5

please check all Graylog parts if they work proper and if Graylog is able to communicate with Elasticsearch and error in your server.log.

6

Hello Michael. BTW…thanks for the quick reply on my previous question.

So I checked the elasticsearch log for one of the days I ran into problems, and here’s what I found – Basically it’s showing “Java overhead spent, Java.lang.OutOfMemoryError: Java heap space”
Which makes sense…because the crashing and disconnection from the database is abrupt and seems a bit random. Is the next step to add more memory? Or should I can we control what’s allocated to the Java heap space?

Below are the actual log messagesk for var/log/elasticsearch/elasticsearch-2019-05-17-1.log

[2019-05-17T16:12:50,083][WARN ][o.e.m.j.JvmGcMonitorService] [Utejpt2] [gc][1423] overhead, spent [1.3s] collecting in the last [1.3s]
[2019-05-17T16:12:48,778][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [Utejpt2] fatal error in thread [Thread-5], exiting
java.lang.OutOfMemoryError: Java heap space
** at io.netty.util.internal.PlatformDependent.allocateUninitializedArray(PlatformDependent.java:204) ~[?:?]**
** at io.netty.buffer.PoolArena$HeapArena.newByteArray(PoolArena.java:676) ~[?:?]**
** at io.netty.buffer.PoolArena$HeapArena.newChunk(PoolArena.java:686) ~[?:?]**
** at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[?:?]**
** at io.netty.buffer.PoolArena.allocate(PoolArena.java:226) ~[?:?]**
** at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[?:?]**
** at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:307) ~[?:?]**
** at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:166) ~[?:?]**
** at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:157) ~[?:?]**
** at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139) ~[?:?]**
** at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) ~[?:?]**
** at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147) ~[?:?]**
** at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[?:?]**
** at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]**
** at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]**
** at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]**
** at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]**
** at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]**

7

how much RAM does your server has and what is the configured JVM HEAP size?

http://docs.graylog.org/en/3.0/pages/configuration/file_location.html

8

Hey Jan.
I reset the heap size defaults to 4GB for both initial and total heap space.
My system is running a total of 16GB of RAM

Here’s what I set the heap size to in the /etc/elasticsearch/java.options:

Xms represents the initial size of total heap space

Xmx represents the maximum size of total heap space

-Xms4g
-Xmx4g

Here’s what it shows as the default settings for java in /etc/default/graylog-server

Path to the java executable.

JAVA=/usr/bin/java

Default Java options for heap and garbage collection.

GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:$

Pass some extra args to graylog-server. (i.e. “-d” to enable debug mode)

GRAYLOG_SERVER_ARGS=“”

Program that will be used to wrap the graylog-server command. Useful to

support programs like authbind.

GRAYLOG_COMMAND_WRAPPER=“”

9

if you get the OOM error from Elasticsearch, you need to add more RAM to the Server and Raise the HEAP for Elasticsearch.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.