Streams and Indices Pages are very slow to load

Hi Community

We have graylog 3.3.5 installation.
Mongodb - 4.2 (iops optimized disk)
Elasticsearch - 6.8.12 (iops optimized disk)

We have currently 339 total streams. Our problem is that Streams and System -> Indices Pages are loading very slow. We manually checked the api http://graylog-url/api/streams and it takes around 3s. Same time it shows on network console too when loading the page, i.e. api is successful after 3s but it keeps on waiting for about 10s on frontend before returning the data. It keeps on saying Loading for long time and then loads the data even the /streams api is successful in 3s. Same happens for our indices page.

We have given good amount of CPU, and memory and we can see that there are no resource bottleneck on resources side. Also as api response time is good, I suspect something on frontend ?

Can someone please suggest.

This is what we see on frontend for >10s:

Hello @ankit, welcome!

You mention you have 339 streams. How many indices do you have? How many total shards? Also please describe your elasticsearch deployment and hardware allocation.

Hi @ttsandrew ,

Thank you.
We have 177 indices and 354 shards.
Elasticsearch cluster version 6.8.12 , health green and no logs being ingested yet into it. (i.e. we haven’t started any input as of now)
Configuration - using ECK for elasticsearch with 2 data and 3 master nodes.
3gb Xmx for master nodes and 6gb for data nodes. CPU limits for eck pods is set to 1.6 . On our monitoring we can see that neither CPU nor memory is the problem for ES cluster as of now.

Still not able to figure out what I can upscale to fix this. All the apis as per the network console on browser completes within 2-3 sec and still page takes >10s to load up.

@ttsandrew, any suggestion here. : (

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.