Server Gateway slow down the Graylog GUI

Graylog Central (peer support)
1

Hi dear community!

I’ve been installing graylog system on our production environnement recently but I met a problem that I met on the development plateform.
Actually, I’m gathering log using rsyslog and collector-sidecar from 2 differents subnets :
1.1.1.0/24 and 2.2.2.0/24. My graylog server is set only on the 1.1.1.0 subnet.
I got only 1 node of Graylog server 2.2 and 1 node elasticsearch 2.4.4.
They are installed on 2 differents VM with centos 7.1 minimal. Network manager, Firwalld and Selinux are disabled as shown below:

[root@graylog ~]# systemctl status NetworkManager
NetworkManager.service - Network Manager
   Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; disabled)
   Active: inactive (dead)

[root@graylog ~]# systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled)
Active: inactive (dead)

[root@graylog ~]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

IP server configuration graylog et elasticsearch:

[root@graylog network-scripts]# cat ifcfg-eno*
TYPE=Ethernet
BOOTPROTO=none
DEVICE=eno16000028
ONBOOT=yes
IPADDR=1.1.1.6
PREFIX=24

[root@elasticsearch network-scripts]# cat ifcfg-eno*
TYPE=Ethernet
BOOTPROTO=none
NAME=eno16000028
DEVICE=eno16000028
ONBOOT=yes
IPADDR=1.1.1.5
PREFIX=24

To gather log from both subnet I need to set up the gateway on the graylog server but when I set it up, the GUI get very very slow, almost unusable. Here is how I set the gateway on graylog server:

    [root@graylog sysconfig]# more /etc/sysconfig/network
    # Created by anaconda
    NOZEROCONF=yes 
    GATEWAY=1.1.1.254

Also, (in the graylog GUI) the heap space bar monitoring of the elasticsearch node doesn’t show up anymore as shown on the following picture:

NODE2
NODE21797×380 24.9 KB

When I remove the gateway the node is reachable:
ok_sans_gw_erreur_graylog2
ok_sans_gw_erreur_graylog21920×1040 82.5 KB

Also I got the following error when I try to reach some pages:
erreur_graylog2
erreur_graylog21635×764 448 KB

Nevertheless, the job is partially done. Message from both subnet are stored in the elasticsearch base. But the web interface got some pages not working and the interface get very slow.

Here is the server configuration, following by the elasticsearch configuration and then the log from the server:

Server configuration:

`############################
# GRAYLOG CONFIGURATION FILE
############################
is_master = true

node_id_file = /etc/graylog/server/node-id
password_secret = sfksdfjsdfjsdfljsdfjsdf
root_password_sha2 = 755dedsfsdf59sdf3172f3945266c59450692a56fsd56f4sf5649f9
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://1.1.1.6:9000/api/
web_listen_uri = http://1.1.1.6:9000/
web_endpoint_uri = http://1.1.1.6:9000/api/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_cluster_name = graylog
elasticsearch_discovery_zen_ping_unicast_hosts = 1.1.1.5:9300
elasticsearch_node_master = false
elasticsearch_node_data = false
elasticsearch_network_host = 1.1.1.6
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 5
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
http://api.mongodb.com/java/current/com/mongodb/MongoOptions.html#threadsAllowedToBlockForConnectionMultiplier
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

Elasticsearch yml on elasticsearch node

# ======================== Elasticsearch Configuration =========================
cluster.name: graylog
path.data: /home/elastic
network.host: 1.1.1.5
http.port: 9200
discovery.zen.ping.unicast.hosts: ["1.1.1.5:9300"]

Elasticsearch yml on graylog node:

> # ======================== Elasticsearch Configuration =========================
> cluster.name: graylog
> network.host: 1.1.1.6
> http.port: 9200
> discovery.zen.ping.unicast.hosts: ["1.1.1.6:9300"]

LOG server graylog:

2017-12-13T17:04:39.543+01:00 WARN  [ProxiedResource] Unable to call 
http://1.1.1.6:9000/api/system/metrics/multiple on node <67956847-4dsfsdf-4f32-913c-66e3e5fg6teb5>
java.net.SocketTimeoutException: timeout
	at okio.Okio$4.newTimeoutException(Okio.java:227) ~[graylog.jar:?]
	at okio.AsyncTimeout.exit(AsyncTimeout.java:284) ~[graylog.jar:?]
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:240) ~[graylog.jar:?]
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) ~[graylog.jar:?]
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) ~[graylog.jar:?]
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) ~[graylog.jar:?]
	at okhttp3.internal.http1.Http1Codec.readResponse(Http1Codec.java:191) ~[graylog.jar:?]
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:132) ~[graylog.jar:?]
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:54) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:179) ~[graylog.jar:?]
	at okhttp3.RealCall.execute(RealCall.java:63) ~[graylog.jar:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?]
	at org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76) ~[graylog.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_121]
	at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_121]
	at okio.Okio$2.read(Okio.java:138) ~[graylog.jar:?]
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:236) ~[graylog.jar:?]
	... 29 more
2017-12-13T17:04:39.547+01:00 WARN  [ProxiedResource] Unable to call http://1.1.1.6:9000/api/system/jobs on node <67956847-4dsfsdf-4f32-913c-66e3e5fg6teb5>
java.net.SocketTimeoutException: timeout
	at okio.Okio$4.newTimeoutException(Okio.java:227) ~[graylog.jar:?]
	at okio.AsyncTimeout.exit(AsyncTimeout.java:284) ~[graylog.jar:?]
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:240) ~[graylog.jar:?]
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) ~[graylog.jar:?]
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) ~[graylog.jar:?]
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) ~[graylog.jar:?]
	at okhttp3.internal.http1.Http1Codec.readResponse(Http1Codec.java:191) ~[graylog.jar:?]
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:132) ~[graylog.jar:?]
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:54) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:179) ~[graylog.jar:?]
	at okhttp3.RealCall.execute(RealCall.java:63) ~[graylog.jar:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?]
	at org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76) ~[graylog.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_121]
	at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_121]
	at okio.Okio$2.read(Okio.java:138) ~[graylog.jar:?]
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:236) ~[graylog.jar:?]
	... 29 more
2017-12-13T17:04:59.515+01:00 WARN  [ProxiedResource] Unable to call http://1.1.1.6:9000/api/system on node <67956847-4dsfsdf-4f32-913c-66e3e5fg6teb5>
java.net.SocketTimeoutException: timeout
	at okio.Okio$4.newTimeoutException(Okio.java:227) ~[graylog.jar:?]
	at okio.AsyncTimeout.exit(AsyncTimeout.java:284) ~[graylog.jar:?]
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:240) ~[graylog.jar:?]
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) ~[graylog.jar:?]
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) ~[graylog.jar:?]
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) ~[graylog.jar:?]
	at okhttp3.internal.http1.Http1Codec.readResponse(Http1Codec.java:191) ~[graylog.jar:?]
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:132) ~[graylog.jar:?]
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:54) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:179) ~[graylog.jar:?]
	at okhttp3.RealCall.execute(RealCall.java:63) ~[graylog.jar:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?]
	at org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76) ~[graylog.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_121]
	at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_121]
	at okio.Okio$2.read(Okio.java:138) ~[graylog.jar:?]
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:236) ~[graylog.jar:?]
	... 29 more`Preformatted text`

Thanks in advance for your help!

Arnaud

2

@arnaud

  1. the heap usage is not of Elasticsearch, it is Graylog heap usage.
  2. the main issue is that your network settings - if the gateway is added - disable the ability that the Graylog node can’t reach it self on the configured API URI http://1.1.1.6:9000/api/
  3. Your browser where you access Graylog is also not able to reach the API URI http://1.1.1.6:9000/api/

As I do not know your Network diagram I can’t give you a way to fix it - just the above reasons for the error.

3

I Jan, Thanks for the quick answer. I wanted to be sure that my graylog configuration is correct before going investigating on the network. Thanks for the correction on the heap usage. I was wrong since a while. It might be something with the router that the only difference between both platform…

4

Actually I resolved my problem by unsetting the hostname of the server. A simple systemctl set-hostname “”…
I do not understand why the server hostname is interfering with the gateway. Does Graylog use the server hostname to communicate with api or web interface?

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.