Performance optimizing

[Mattias]

Hello!
Sorry that i messed up the preformatted text.

1. How can I increase performance against the web interface?:

2. I have a graylog cluster in VMware. Two RedHat 8 machines with apache-web/graylog/elastic on both. I have realized now that this was a bad choice on my part, so now I need help to increase performance. It takes ages to load the search page and dashboards:

• OS Information:
  2 x RedHat ES 8 with latest patches.
  Per machine, 8 vCPU and 32GB memory and disks on SAN

• Package Version:
  graylog-server-4.2.6-1
  elasticsearch-oss-7.10.2-1
  mongodb-org-4.2.18-1
  java 8

• Service logs, configurations, and environment variables:
  Everything i pretty much default settings that comes with the packages.

jvm.options for elasticsearch is
-Xms8g
-Xmx8g

And override file is configured with:
[Service]
LimitMEMLOCK=infinity

For graylog
GRAYLOG_SERVER_JAVA_OPTS="-Xms8g -Xmx8g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow"

3. What steps have you already taken to try and solve the problem?
I have tried to have the web server on another machine.
I have tried with more and less cpu/ram
I have tried with different values in the files mentioned above.

Current,

4. How can the community help?
How can i increase web access performance so that i can work with the system?
I have read some articles here in the forum how to optimize but I dont get any wiser except i now realize I should have put elasticsearch and graylog on separate machines due to both elastic and graylog will fight for available resources.

Im no expert on graylog or elastic so if anyone have the time to point me in the right direction i will be very happy!

Thanks!

[Arie (Arie van den Heuvel)]

Not knowing what exactly causes your problem it is kind of need to know if elastic is
configured a cluster to.

8GB for graylog sounds to much for me, try lowering that to the default.

Things one could look at is tuned (to better suet for performance), and if one sends in a lot of messages change the refresh interfal for your indexes and to continue that create a templet for it with cerebro.

{
  "order": 0,
  "index_patterns": [
    "graylog_*"
  ],
  "settings": {
    "index": {
      "refresh_interval": "5s"
    }
  },
  "mappings": {},
  "aliases": {}
}

Configure ulimit the right way, by default in this setup the limits can create a bottleneck
this is in /etc/security/limits.conf

root       - nofile  65535
*            - nofile  65535
elasticsearch   - memlock unlimited

Best option is to migrate elasticsearch to its own cluster, you could expand your
cluster to extra nodes an shrink it by sutting down the first ones whaen all data
is replicated.

[Mattias]

@Arie Thanks!

Yes, i missed to mention that elastic is in a cluster and yes, my plan now i think is to expand the elastic cluster to two new nodes on separate machines and them remove elastic from the graylog nodes.

I have used 2/4/6/8 GB for graylog. Cant see any performance changes so i will set it to default again.

I hade the follwoing in /etc/security/limits.d/elasticsearch.conf

elasticsearch  -  nofile  65536
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
elasticsearch - memlock unlimited

I will have a look at tuned! Super!

Thanks!!!

[HansWurst]

When do you have performance problems? How many messages per second do you process?

I have a 4 node system. 2 of the nodes host Graylog. All 4 nodes host MongoDB. All 4 nodes host Elasticsearch. LB before to distribute incoming Traffic (Beats, Syslog, Raw UDP).

8 Cores per VM and 64Gb RAM each. Usually no real load on the system, processing 20-30GB per day. At the peak they processed aprox. 7.000 messages per second and GL node as I rolled out some sidecars and they send all their old logs at the same time.

[Mattias]

@HansWurst The performance problem is when hit “search” and land on the /search page. It takes a minute or so to load that page and if i try searching for something i takes another minute or so.
I have around 100-200 messages per second and only 6GB per day.

So its only when im “working” with the system i experience performance issues.
Do you have a apache or nginx fronting Graylog? Is it fast to work with? If i access graylog directly or via apache in front, i cant notice any performance differences.
I thought it would be enough for me to have a “master/slave” setup with everything on both nodes but it looks like a need to exapand it.

Thanks!