Graylog cluster with Nginx not working

Graylog Central (peer support)
1

1. Describe your incident:
Access Graylog via Nginx load balancer not loading content, getting error “Loading component failed: Loading chunk 74e0f23a-689 failed. (error: http://192.xxx.1.50:9000/assets/LoggedInPage.ef4331dbd405f3caf4d0.js)”

I have 2 graylog nodes with the following conf file pointing to the same 1x opensearch (wz indexer) node. they are running fine individually via their ip (http://192.xxx.1.80:9000 and http://192.xxx.1.81:9000).

I then added a nginx load balancer in front the 2 nodes. the following were observed:

  • the nginx url (http://192.xxx.1.50:9000/) return blank page
  • Login page http://192.xxx.1.50:9000/login was able to load and I was able to login but landed at another blank page with the loading component and chuck error
  • Static assets (e.g., menu bar) starts to show up after refresh the browser
  • static asset (e.g., system) can be displayed, the log counter on the top right is showing log coming in, but other pages (search, streams) return the same load error.
  • if the worker node is removed from the up_stream block, then all problems go away, and I can access the master node via nginx url

2. Describe your environment:

nginx conf file:
http {
upstream graylog-cluster {
server 192.xxx.1.80:9000;
server 192.xxx.1.81:9000;
}
server {
listen 9000;
listen [::]:9000;
location / {
proxy_pass http://graylog-cluster;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}

3. What steps have you already taken to try and solve the problem?
We have tried out multiple suggestions online, like:

  • enable gzip option in nginx
  • uncomment http_external_uri
    there are limited documentation on graylog cluster with load balancer, most blogs related to cluster are back in 2020 or older and most of their “resolved” configuration does not work in our version. We followed this doc (https://graylog.org/post/load-balancing-graylog-with-nginx-ultimate-guide/) closely but we are stuck at the current situation.

4. How can the community help?
could you please guide us to any resources related to graylog cluster and load balancer?

Any help are much appreciated.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

are you sure to have the same graylog version on both nodes? I have this error usually if there is a missmatch

3

Thanks for the response, I have been researching along the idea of session persistency and in fact, I added ip_host in Nginx to “solve” the problem. With you advice and after checking, indeed, the 2 nodes are of a minor version different, 6.1.8 and 6.1.7. after upgrading both to 6.1.10 and removing ip_host from Nginx config, the loading issue is resolved. However, I am getting “Could not load last items” and sometimes “Could not load favorite” errors. will research more