Search failing to load when adding nodes to a cluster

Graylog Central (peer support)
1

I have an issue searching when running multi-node setup. If I am running just a single node, things are okay. As soon as I spin up a second or third node, I get the spinning “Updating search results” which changes to “This is taking a bit longer, please hold on” and ultimately nothing. If I delete the additional node(s), it all starts working perfectly again. From what I can tell, nothing is wrong with mongodb or opensearch. I have don’t see any errors or issues in my logs. I am completely baffled as to what would even cause this, and with no errors being reported, I can’t think what to do next. I normally try to bang my head until I figure out my problem, but I’ve tried that and am getting nowhere. I figure I will break down and ask the question here.

Running k3s (v1.31.2+k3s1) version of kubernetes on Ubuntu 22.04
Graylog server 6.1.2+c8b654f
JRE: Eclipse Adoptium 17.0.13 on Linux 6.8.0-48-generic
Deployment: docker

mongodb-agent:12.0.24.7719-1
opensearch:2.18.0

Here is the startup log from my master node:

And then here is the first additional node in the cluster:

Data seems to be collecting fine, no issue with input.
Opensearch health is fine.
All nodes on the status page in graylog seem fine, no errors or warnings.
No issues reported under any of the indexes.

Really at a lost and very confused. I appreciate in advance any assistance anyone can provide on what I might look at next to try to figure this one out. The good news, if I just run one node, it works okay, but not the setup I want.

2

I have a bad feeling I’m going to feel really stupid when someone points out the obvious problem that I missed :frowning:

3

If you lookup online how to use curl and directly run a query against the opensearch api, do those results show correctly?

Are you listing all the opensearch nodes in your server.conf file and in the opensearch configs?

4

Thanks @Joel_Duffield

Yes, Opensearch seems fine. That is actually where I started. But that uses a load balancer that sits in front of opensearch.

I’ve configured graylog with the following to reach opensearch:

elasticsearch_hosts = http://opensearch-cluster-master.opensearch.svc.cluster.local.:9200

Oh, I think you just figured it out! I’m wondering if this is a cross node issue. I’m going to check if it’s because graylog-0 works and is on the same k3s node as the opensearch master, but fails when a graylog-1 node is on a different k3s node…I’ll come back with more test results, but that would make sense to describe the problem. Then I just would need to figure out why :slight_smile:

5

Nope, it was a good theory, but I don’t think that is the problem. When I connect to both pods directly, I can curl opensearch from both nodes using that URL no problem, and it hit each of the different opensearch nodes.

╰─○ kubectl describe service/opensearch-cluster-master | grep Endp
Endpoints:                192.168.125.102:9200,192.168.125.106:9200,192.168.96.173:9200

I’ll keep looking. This is definitely something strange in how I have Graylog setup though. I’m pretty sure Opensearch is fine. This did used to work, and I’m not exactly sure what I did in the environment that broke it. I’m sure it’s something I did, just trying to figure out what.

6

I have confirmed further it’s not an issue with communications across nodes. I ran the active graylog node on my -2 and -3 servers, and they can query fine, as long as no other graylog servers are running. As soon as I have 2 or more nodes in my graylog cluster, I start having issues with querying. Strange right!? :slight_smile:

7

Wait, your putting a load balancer between graylog and opensearch, thats not how its built to work?

8

No, sorry, let me clarify. When running the manual test it goes through a load balancer. That’s how I normally expose services on my network. However the config in Graylog is the kubernetes service address, which responds with the three different opensearch nodes. So if I manually run curl from the graylog pod, I see each of the different servers. For example:

graylog@graylog-0:~$ curl http://opensearch-cluster-master.opensearch.svc.cluster.local.:9200 -s | grep name
  "name" : "opensearch-cluster-master-0",
  "cluster_name" : "opensearch-cluster",
graylog@graylog-0:~$ curl http://opensearch-cluster-master.opensearch.svc.cluster.local.:9200 -s | grep name
  "name" : "opensearch-cluster-master-1",
  "cluster_name" : "opensearch-cluster",
graylog@graylog-0:~$ curl http://opensearch-cluster-master.opensearch.svc.cluster.local.:9200 -s | grep name
  "name" : "opensearch-cluster-master-1",
  "cluster_name" : "opensearch-cluster",
graylog@graylog-0:~$ curl http://opensearch-cluster-master.opensearch.svc.cluster.local.:9200 -s | grep name
  "name" : "opensearch-cluster-master-2",
  "cluster_name" : "opensearch-cluster",
graylog@graylog-0:~$ curl http://opensearch-cluster-master.opensearch.svc.cluster.local.:9200 -s | grep name
  "name" : "opensearch-cluster-master-1",
  "cluster_name" : "opensearch-cluster",
graylog@graylog-0:~$ curl http://opensearch-cluster-master.opensearch.svc.cluster.local.:9200 -s | grep name
  "name" : "opensearch-cluster-master-2",
  "cluster_name" : "opensearch-cluster",
graylog@graylog-0:~$ curl http://opensearch-cluster-master.opensearch.svc.cluster.local.:9200 -s | grep name
  "name" : "opensearch-cluster-master-2",
  "cluster_name" : "opensearch-cluster",

I could change this value to test, but I haven’t yet. Reason being, I don’t see how that would affect things if more than one node is running, it’s the same value either way, and works regardless of the number of nodes running. That’s the part I can’t figure, is what graylog does differently, when running a query, when it has a partner live. Also, I know graylog is seeing all nodes, as they show up in the interface too.

9

So to clarify adding and removing opensearch nodes isnt the issue, the issue is when you add additional graylog nodes?

If so, if you have say 3 graylog nodes running and you connect to the web interface of each directly bypassing the load balancer, does a query work from one but not the others, say the leader works but not the two member nodes?

10

That is correct. Opensearch doesn’t seem to be a problem, it’s just adding graylog nodes. My LB does round robin, and I can see which node I’m on when I run queries. I don’t have direct access to them from my desktop currently as they are only in the kubernetes network, so I only reach them through the LB. I can make that change later though and play around more. But, I have confirmed as I see which one I’m connected to that they all end up having problems as I move through them. But…queries will sometimes work, so, it’s intermittent, and I have not found a pattern for when then work. It’s a good point though, so let me do that and provide the results so I can be specific. Thanks.

11

So strange…yes, I can confirm that each node stops working correctly as soon as it has a sibling. Almost just want to start over :frowning:

What I find frustrating is just not knowing why. Something in my recent upgrades did this, but I did a few so I can’t tell what would cause this. Everything else seems to be working just fine. Oh well.

I have completely wiped out all persistent volumes and nodes, and booted fresh, same thing. Maybe I wipe out all my settings and mongo next.

12

I think I might know what is happening here. And now what I can’t understand is how this ever worked for me. I’ve been running like this for ages.

Here is my theory. When running with a single node, an LB is no problem, all requests go to one place. When there are multiple nodes, the page will load up no problem from one server, but when the request for a query goes out for a search, it will be a search job on one node, but then follow up requests might get directed to different nodes.

When I look at the console, I see a 404 on the search job. This makes me think the search job is on the server originally loaded, but the follow up request was set to a different node.

image
image2910×136 44.5 KB

I dunno, just a working theory right now, where requests and search jobs are not sticky to the same node.

Continuing my investigation.

13

I have confirmed the issue. I tried to get sticky sessions setup, but I’m not doing something right there. I just forced all traffic to a single node, and that works. All three nodes can be up, but all requests are just sent to the first node. So I believe this means my theory is right. The search is getting sent to a different node than originally visited, causing the search job to be unknown.

Now to figure out a good solution. Folks must run Graylog behind traffic managers, do they need to be stateful sessions?

Time to head back to the docs, but I do welcome any thoughts folks have on their setups if they use traefik or another LB.

Thanks.

14

It should actually work with the api requests hitting different nodes, IF everything is setup correctly.

Can you post a sanitized version of your server.conf config, or in this case your k8 config for thr server.conf settings?

15

Thanks Joel! Yes, and it was working for a long time :slight_smile: So that would make sense. Happy to share my whole config if you have ideas!

╰─⠠⠵ cat graylog.conf | egrep -v '^#|^$'
is_master = false
node_id_file = /usr/share/graylog/data/node-id
password_secret = ****
root_password_sha2 = *****
root_timezone = America/New_York
bin_dir = /usr/share/graylog/bin
data_dir = /usr/share/graylog/data
plugin_dir = /usr/share/graylog/plugin
http_bind_address = 0.0.0.0:9000
http_publish_uri = https://graylog-prod.goepp.net/
http_external_uri = https://graylog-prod.goepp.net/
http_enable_cors = true
trusted_proxies = 10.43.0.0/8
elasticsearch_hosts = http://opensearch-cluster-master.opensearch.svc.cluster.local.:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = data/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://graylog:****@mongodb-0.mongodb-svc.mongodb.svc.cluster.local:27017,mongodb-1.mongodb-svc.mongodb.svc.cluster.local:27017,mongodb-2.mongodb-svc.mongodb.svc.cluster.local:27017/graylog?replicaSet=mongodb&ssl=false
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = true
transport_email_hostname = mail.goepp.net.
transport_email_port = 587
transport_email_use_auth = false
transport_email_subject_prefix = [graylog]
transport_email_from_email = graylog@goepp.net
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_web_interface_url = https://graylog-prod.goepp.net/
proxied_requests_thread_pool_size = 32

If it helps I’ll include my manifest file for kubernetes here too. I know that might not be your area, but in case it has something interesting.

---
apiVersion: v1
kind: Namespace
metadata:
  name: graylog

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: graylog
  namespace: graylog
spec:
  serviceName: "graylog"
  replicas: 3
  selector:
    matchLabels:
      app: graylog
  template:
    metadata:
      labels:
        app: graylog
    spec:
      dnsConfig:
        options:
          - name: ndots
            value: "1"
      containers:
      - name: graylog
        image: graylog/graylog:6.1
        resources:
          requests:
            cpu: "100m"
            memory: "2Gi"
          limits:
            cpu: "2"
            memory: "4Gi"
        imagePullPolicy: Always
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        - name: graylog-data
          mountPath: /usr/share/graylog/data
        - name: config
          mountPath: /usr/share/graylog/data/config
        ports:
        - name: http
          containerPort: 9000
        - name: syslog-udp
          containerPort: 1514
          protocol: UDP
        - name: gelf-udp
          containerPort: 12201
          protocol: UDP
        readinessProbe:
          httpGet:
            path: /
            port: 9000
          initialDelaySeconds: 0 
          periodSeconds: 20
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /
            port: 9000
          initialDelaySeconds: 0
          periodSeconds: 20
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
      volumes:
      - name: config
        configMap:
          name: graylog-config
  volumeClaimTemplates:
  - metadata:
      name: graylog-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

---
apiVersion: v1
kind: Service
metadata:
  name: graylog-udp
  namespace: graylog
  labels:
    app: graylog
spec:
  selector:
    app: graylog
  type: ClusterIP
  clusterIP: None
  ports:
  - name: syslog-udp
    port: 1514
    targetPort: 1514
    protocol: UDP
  - name: gelf-udp
    port: 12201
    targetPort: 12201
    protocol: UDP

---
apiVersion: v1
kind: Service
metadata:
  name: graylog-tcp
  namespace: graylog
  labels:
    app: graylog
spec:
  selector:
    app: graylog
    graylog-active: "true"
  type: ClusterIP
  clusterIP: None
  ports:
  - name: http
    port: 9000
    targetPort: 9000
    protocol: TCP

---
apiVersion: traefik.io/v1alpha1
kind: IngressRouteUDP
metadata:
  name: graylog-syslog-udp
  namespace: graylog
spec:
  entryPoints:
    - syslog-udp
  routes:
  - services:
    - name: graylog-udp
      port: 1514

---
apiVersion: traefik.io/v1alpha1
kind: IngressRouteUDP
metadata:
  name: graylog-gelf-udp
  namespace: graylog
spec:
  entryPoints:
    - gelf-udp
  routes:
  - services:
    - name: graylog-udp
      port: 12201

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: graylog
  namespace: graylog
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`graylog-prod.goepp.net`) && PathPrefix(`/`)
      kind: Rule
      services:
        - name: graylog-tcp
          port: 9000
16

I have read the docs too, but I’m not finding anything obvious that I’m doing wrong.

https://go2docs.graylog.org/5-0/setting_up_graylog/web_interface.htm#making-the-web-interface-work-with-load-balancersproxies

17

When you run multiple node Graylog, can you check if you can see the health status of all nodes in the system/nodes menu?

If you click into them all, do you get the party gorilla on any of them?

18

Your publish uri is the load balancer address, but it needs to be the address of that node that the other nodes use to communicate with it. Only external uri should be the load balancer address.

19

Thank you @Joel_Duffield and @kpearson! I was not reading that config option correctly, that was exactly the problem. What I can’t figure out is how it was working at all before??? I went back through my git history, that hasn’t changed ever. Anyway, it’s working now. Great…moving on. Really appreciate the help and patience. Sure enough, my original comment came true, I’m going to feel stupid when I learn what the issue was :frowning:

20

On the bright side, maybe someone else will come along some day with this same problem and this post will help them so they don’t have to feel as stupid.