Graylog init pod not completing (Kubernetes Helm Chart)

Graylog Central (peer support)
1

Hi, all,
My team and I have been using Graylog for a while to monitor our services and are now working on migrating to a kubernetes cluster. However, I’ve run into a problem in deploying the Helm Chart.

After following the Manual Install Dependencies + Install Chart guide, mongodb and elasticsearch seem to be starting correctly but the Graylog init pod is stuck in a CrashLoop. The same thing happens when I run the Quick Installation.

graylog-0                               0/1     Init:0/1                12         38m
graylog-0                               0/1     Init:Error              12         38m
graylog-0                               0/1     Init:CrashLoopBackOff   12         38m

Commands used:

  • helm install --namespace "graylog" -n "mongodb" stable/mongodb-replicaset
  • helm install --namespace "graylog" -n "elasticsearch" stable/elasticsearch
  • helm install --namespace "graylog" -n "graylog" stable/graylog --set tags.install-mongodb=false --set tags.install-elasticsearch=false --set graylog.mongodb.uri=mongodb://mongodb-mongodb-replicaset-0.mongodb-mongodb-replicaset.graylog.svc.cluster.local:27017/graylog?replicaSet=rs0 --set graylog.elasticsearch.hosts=http://elasticsearch-client.graylog.svc.cluster.local:9200

The logs seem to indicate the graylog server not starting, but I can’t figure out why this is the case.

λ kubectl logs -n graylog graylog-0
Error from server (BadRequest): container "graylog-server" in pod "graylog-0" is waiting to start: PodInitializing

λ kubectl logs -n graylog graylog-0 -c setup
Connecting to storage.googleapis.com (172.217.20.48:443)
wget: note: TLS certificate validation not implemented
saving to '/k8s/kubectl'
kubectl                7% |**                              | 3278k  0:00:11 ETA
kubectl               24% |*******                         |  9.9M  0:00:06 ETA
kubectl               40% |************                    | 16.6M  0:00:04 ETA
kubectl               57% |******************              | 23.4M  0:00:03 ETA
kubectl               73% |***********************         | 30.1M  0:00:01 ETA
kubectl               89% |****************************    | 36.8M  0:00:00 ETA
kubectl              100% |********************************| 41.1M  0:00:00 ETA
'/k8s/kubectl' saved
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1205: Read-only file system
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1205: Resource temporarily unavailable
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1305: Read-only file system
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1305: Resource temporarily unavailable
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1105: Read-only file system
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1105: Resource temporarily unavailable
chown: /usr/share/graylog/data/journal/.snapshot: Read-only file system
chown: /usr/share/graylog/data/journal/.snapshot: Read-only file system

I’ve also brought this up on the Helm Chart GitHub page, and since this might primarily be a Kubernetes issue it is no sweat if it is outside the scope of this forum, always worth a shot though.

Please let me know if there is more information required.
Cheers,
Oscar

2

sorry I can’t help with that. I had never need to run Graylog on Kubernetes.

3

No worries, it feels like there’s a very specific fault happening.

The .snapshot directory it complains about when trying to run chown -R 1100:1100 ... already is owned by 1100:1100 as far as I can see, but for some reason the .snapshot directory complains as it for some reason seems to be set to Read Only… I don’t understand why the .snapshot directory is even there, since afaik it should be and empty volume.

I am running an nfs storage for my cluster, where .snapshots seem to be made on a root level, but they shouldn’t be mounted into the volumes since these are placed as sub-directories… Yup, this eludes me.

4

TL;DR: Adding subPath: mount to the volumeMount “journal” moves the .snapshot file out of the container, avoiding the problem.

...
        volumeMounts:
        - mountPath: /usr/share/graylog/data/journal
          name: journal
          subPath: mount
...

The problem was indeed the .snapshot file, after applying a workaround detailed detailed in my StackOverflow question (adding a subPath to the mounted volume, thus moving the .snapshot file “up one level” and keeping it out of the actual container), Graylog finally started!

Now, my problem is I haven’t managed to route traffic from my external computer to the server where the service is running, but that is not a Graylog issue.

@jan, I’ll look around and see if there’s a standard for changing titles to [solved] or something like that, and leave it up in case someone else runs into something similar, but if you want to clean up feel free to delete the thread. Cheers!

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.