Graylog init pod not completing (Kubernetes Helm Chart)

Hi, all,
My team and I have been using Graylog for a while to monitor our services and are now working on migrating to a kubernetes cluster. However, I’ve run into a problem in deploying the Helm Chart.

After following the Manual Install Dependencies + Install Chart guide, mongodb and elasticsearch seem to be starting correctly but the Graylog init pod is stuck in a CrashLoop. The same thing happens when I run the Quick Installation.

graylog-0                               0/1     Init:0/1                12         38m
graylog-0                               0/1     Init:Error              12         38m
graylog-0                               0/1     Init:CrashLoopBackOff   12         38m

Commands used:

  • helm install --namespace "graylog" -n "mongodb" stable/mongodb-replicaset
  • helm install --namespace "graylog" -n "elasticsearch" stable/elasticsearch
  • helm install --namespace "graylog" -n "graylog" stable/graylog --set tags.install-mongodb=false --set tags.install-elasticsearch=false --set graylog.mongodb.uri=mongodb://mongodb-mongodb-replicaset-0.mongodb-mongodb-replicaset.graylog.svc.cluster.local:27017/graylog?replicaSet=rs0 --set graylog.elasticsearch.hosts=http://elasticsearch-client.graylog.svc.cluster.local:9200

The logs seem to indicate the graylog server not starting, but I can’t figure out why this is the case.

λ kubectl logs -n graylog graylog-0
Error from server (BadRequest): container "graylog-server" in pod "graylog-0" is waiting to start: PodInitializing
λ kubectl logs -n graylog graylog-0 -c setup
Connecting to (
wget: note: TLS certificate validation not implemented
saving to '/k8s/kubectl'
kubectl                7% |**                              | 3278k  0:00:11 ETA
kubectl               24% |*******                         |  9.9M  0:00:06 ETA
kubectl               40% |************                    | 16.6M  0:00:04 ETA
kubectl               57% |******************              | 23.4M  0:00:03 ETA
kubectl               73% |***********************         | 30.1M  0:00:01 ETA
kubectl               89% |****************************    | 36.8M  0:00:00 ETA
kubectl              100% |********************************| 41.1M  0:00:00 ETA
'/k8s/kubectl' saved
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1205: Read-only file system
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1205: Resource temporarily unavailable
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1305: Read-only file system
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1305: Resource temporarily unavailable
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1105: Read-only file system
chown: /usr/share/graylog/data/journal/.snapshot/hourly.2019-06-25_1105: Resource temporarily unavailable
chown: /usr/share/graylog/data/journal/.snapshot: Read-only file system
chown: /usr/share/graylog/data/journal/.snapshot: Read-only file system

I’ve also brought this up on the Helm Chart GitHub page, and since this might primarily be a Kubernetes issue it is no sweat if it is outside the scope of this forum, always worth a shot though.

Please let me know if there is more information required.

sorry I can’t help with that. I had never need to run Graylog on Kubernetes.

No worries, it feels like there’s a very specific fault happening.

The .snapshot directory it complains about when trying to run chown -R 1100:1100 ... already is owned by 1100:1100 as far as I can see, but for some reason the .snapshot directory complains as it for some reason seems to be set to Read Only… I don’t understand why the .snapshot directory is even there, since afaik it should be and empty volume.

I am running an nfs storage for my cluster, where .snapshots seem to be made on a root level, but they shouldn’t be mounted into the volumes since these are placed as sub-directories… Yup, this eludes me.

TL;DR: Adding subPath: mount to the volumeMount “journal” moves the .snapshot file out of the container, avoiding the problem.

        - mountPath: /usr/share/graylog/data/journal
          name: journal
          subPath: mount

The problem was indeed the .snapshot file, after applying a workaround detailed detailed in my StackOverflow question (adding a subPath to the mounted volume, thus moving the .snapshot file “up one level” and keeping it out of the actual container), Graylog finally started!

Now, my problem is I haven’t managed to route traffic from my external computer to the server where the service is running, but that is not a Graylog issue.

@jan, I’ll look around and see if there’s a standard for changing titles to [solved] or something like that, and leave it up in case someone else runs into something similar, but if you want to clean up feel free to delete the thread. Cheers!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.