Troubleshooting Dash Speed Issues: Maybe Shards/Indices?

Graylog 5.2.2+8eab621

Diagram:

Server Host: ProxMox 8.0.3
GuestOS: Debian 12
Nodes: 16 vcpus 32GB ram 16GB heap (x3)
Servers: 8 vcpus 16GB ram (x3)

Greetings; I am trying to trouble shoot extremely slow performance (witnessed when using the server [all aspects of the web gui]).

I figured I would start with opensearch but, honestly, am not sure how to go about tweaking .

Currently, my indices are as follows:

Note: the settings are new and have only been in effect for about 2 hours; previous settings had index rotation at 30d; 12 kept; delete. Shards/replicas have always been set at 1/1.

curl -k -X GET "https://<username>:<password>@graynode-1.foo.barl:9200/_cat/shards?v=true&h=index,prirep,shard,store&s=prirep,store&bytes=gb"

produces:
indices.pdf (53.8 KB)

curl -k -X GET "https://<username>:<password> @graynode-1.foo.bar:9200/_cluster/health?pretty"

produces:
health.pdf (19.7 KB)

Now, when I say slow, I mean I’ve had it take in excess of 60s (and then I quit).

Any ideas? I can supply details as requested (I did not know what to provide)

How much data are you ingesting a day (you can see from system>overview? So you were keeping data for a year, did you actually get up to that full year of retention, or on average how far back does your data go as of today, but you just changed these all to store no more than 40 days right?

The company I support is project based so its feast or famine when it comes to data ingestion. Busy period we tip the scales at 20GB a day. Slow its down below 10GB

About 6 months, now (on how long its been up and running)

Yes, so that, as I understand it, it will not rotate the index until, at most, 40 days while keeping it between 20-50GB in size (am I understanding that correctly?) and then will keep all indices not older than “12” (i.e. based on 30-40 day period, approximately a year)

No, if you are using time size optimized it ignores that keep X number of indices. So it will keep data for 40 days max, that’s it.

It will rotate the index when the size gets to an ideal size, so in a busy environment maybe it rotates a few times a day and there are dozens on indices in the 40 days, in a small environment maybe it only rotates every two weeks and there are 2 or 3 indices.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.