Error while retrieving data for widget [helm chart]

[hamzasaeed5594]

This question is similar to this question
But I am using (version Graylog 4.2.3) helm chart on my Kubernetes EKS cluster for Graylog cluster installation kongz like this:
kubectl create namespace graylog
helm install --namespace "graylog" graylog kongz/graylog
so I can’t modify the memory or anything. I have tried to check the files in graylog and elasticsearch pod but didn’t find them there.
/etc/elasticsearch/jvm.options
/var/log/elasticsearch/hs_err_pid495.log
/etc/default/graylog-server

To avoid this in the first place, yesterday I set the retention as follows but it didn’t do much:

Right now the memory/heap usage is as follows:

Can anyone please help me out to increase memory/cpu for my Graylog cluster using helm or manually in Kubernetes? Thanks in advance

[gsmith]

Hello,

@hamzasaeed5594

so I can’t modify the memory or anything. I

Well… that sucks. Only suggestion I can think of reducing the amount of resource you Graylog server needs.

As for the screen shot you provided with the error.Elasticsearch circuit breaking exeception Dtat to large, data for http_request . I did a quick search.

https://cylab.be/blog/165/fixing-circuit-breaking-exception-parent-data-too-large-data-for-http-request-elk-stack-error

Conclusion:

If this server is unable to add resources that it needs then you will have to adjust your environment to adapt to the resource that it can handle.

I see you have 1.8 JAVA heap but have you tried to increase elasticsearch?
Found in this file.
/etc/elasticsearch/jvm.options

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms4g
-Xmx4g

Or docker-compose.yaml file, perhaps add this line and increase the amount needed.

environment:

 - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

[hamzasaeed5594]

I couldn’t find the file /etc/elasticsearch/jvm.options in Graylog or Elasticsearch cluster. I am using helm chart for Graylog cluster installation, I tried to set -Xmx4g -Xms4g like this:
helm install --namespace "graylog" graylog kongz/graylog \ --set graylog.heapSize="-Xms4g -Xmx4g"
Now I am getting crashloopbackoff and errors on graylog pod.

when I checked the error logs I got this:

[hamzasaeed5594]

I got the solution. After checking the above error log, I came to know that -Xms is already defined and all I have to do is enter the value like this:

helm install --namespace "graylog" graylog kongz/graylog \
  --set graylog.heapSize=4g

The 4g will be allocated to Xms

[hamzasaeed5594]

@gsmith I increased the heap memory to 4GB.

Now I have observed the error is still persisting

[gsmith]

Hello,

What I tried to show you was increase memory for Elasticsearch/Graylog. Try to increase GL Heap and see if you get the same results.
Not sure how your going to do it in helm

• Java Heap sizes in Graylog and ElasticSearch | Graylog

Example:
Two completely different settings

GRAYLOG_SERVER_JAVA_OPTS: '-Xms4g -Xmx4'
 - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

Perhaps this will help

• https://cylab.be/blog/165/fixing-circuit-breaking-exception-parent-data-too-large-data-for-http-request-elk-stack-error#:~:text=There%20are%20different%20ways%20of,detailed%20in%20the%20Elasticsearch%20documentation.

[gsmith]

Did some research for ya.

Since Helm is being used, I believe this option need to be executed esJavaOpts

Java options for Elasticsearch. This is where you could configure the jvm heap size

You will see this in the description from the link below.

• https://github.com/elastic/helm-charts/blob/main/elasticsearch/README.md

Hope that helps