Hi,
We have a 3 node graylog cluster that I upgraded from 2.2.3 to 2.3.1. After the upgrade the web interface was noticeably slower. Especially so on the Search page and the Sources page.
Last time I tried loading the Search page it took 12 seconds but the search result is saying it found Found 39,207 messages in 342 ms, searched in 231 indices for the last 5 minutes. The Sources page is taking roughly 6 seconds to load and the two graphs have the spinning icon until it loads. Both these pages loaded almost instantaneously before the upgrade.
I thought it might be due to the load balancer haproxy that points to nginx. I took both out of the equation and the speeds remained the same.
I then upgraded elasticsearch from 2 to 5. Still the same result.
Our setup is as follows:
3 graylog VMs running Graylog and Mongodb
20 CPUs
14GB of ram
Centos 7
Graylog 2.3.1+9f2c6ef o
Linux 3.10.0-693.2.2.el7.x86_64)
Oracle Corporation 1.8.0_144
openjdk version "1.8.0_144"
OpenJDK Runtime Environment (build 1.8.0_144-b01)
OpenJDK 64-Bit Server VM (build 25.144-b01, mixed mode)
3 elasticsearch VM nodes
20 CPUs
25 GB of ram
12 GB java heap
Centos 7
Linux 3.10.0-693.2.2.el7.x86_64)
elasticsearch
"number" : “5.6.2”,
“build_hash” : “57e20f3”,
“build_date” : “2017-09-23T13:16:45.703Z”,
“build_snapshot” : false,
“lucene_version” : “6.6.1”
The Indices/Index are set to
Shards: 4
Replicas: 2
Each elasticsearch node has roughly 250GB of data.
The hardware behind this setup is brand new and not being taxed at all. The SAN iops are hardly being touched. This setup is only receiving roughly 100 messages/sec.
Here is a link to our config for the master graylog node. The other 2 are identical except where node1 needs to be node2, etc… And only node1 is master.
Any ideas on what to do?
Thanks,
Ryan