Grylog taking much more time to start

Graylog version 3
Elastic search 6.5

I am running graylog helm chart to setup graylog in kubernetes. but i am getting error in when i do kubectl get describe pod <>

error is : Readiness probe failed: Get http://10.8.2.67:9000/api/system/lbstatus: dial tcp 10.8.2.67:9000: connect: connection refused

After very long time pod is starting but still slave if not working and in master log i have found one issue

2019-05-03 06:27:29,576 ERROR: org.graylog2.indexer.fieldtypes.IndexFieldTypePoller - Couldn't get mapping for index <graylog_0>: null
2019-05-03 06:27:31,277 ERROR: org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical - Couldn't update field types for index set <Default index set/5ccaa268d1c68b000e51e3f4>
com.mongodb.MongoInterruptedException: Interrupted acquiring a permit to retrieve an item from the pool

while my slave is not staring only.

containers:
                    - name: graylog-master
                      image: "graylog/graylog:3.0.1"
                      securityContext:
                        privileged: true
                        runAsUser: 1100
                      env:
                      - name: GRAYLOG_PASSWORD_SECRET
                        value: {{required "A valid .Values.graylog.passwordSecret entry required!" .Values.graylog.passwordSecret }}
                      - name: GRAYLOG_ROOT_PASSWORD_SHA2
                        value: {{ .Values.graylog.passwordSecret | sha256sum }}
                      - name: GRAYLOG_MONGODB_URI
                        value: "mongodb://{{ $mongodbServiceName }}-0.{{ $mongodbServiceName }}:27017/graylog?replicaSet=rs0"
                      - name: GRAYLOG_IS_MASTER
                        value: "true"
                      - name: GRAYLOG_ELASTICSEARCH_DISCOVERY_ZEN_PING_UNICAST_HOSTS
                        value: "{{ $elasticsearchServiceName }}-0.{{ $elasticsearchServiceName }}:9300"
                      - name: GRAYLOG_ELASTICSEARCH_HOSTS
                        value: "http://{{ $elasticsearchServiceName }}-0.{{ $elasticsearchServiceName }}:9200"
                      - name: GRAYLOG_HTTP_PUBLISH_URI  
                        value: "https://<MY URL>"
                      - name: GRAYLOG_HTTP_EXTERNAL_URI  
                        value: "https://<MY URL>"
                      - name: GRAYLOG_SERVER_JAVA_OPTS
                        value: "-Xms1g -Xmx1g -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow"
                      ports:
                      - containerPort: {{ .Values.graylog.web.service.internalPort }}
                        name: http
                      - containerPort: 12201
                        name: udp-input
                      - containerPort: 1514
                        name: tcp-input
                      volumeMounts:
                      - name: graylog-persistent-storage
                        mountPath: /usr/share/graylog/data/journal
                        subPath: graylog
                      resources:
            {{ toYaml .Values.graylog.master.containers.resources | indent 12 }}
                      readinessProbe:
                        httpGet:
                          path: /api/system/lbstatus
                          port: {{ .Values.graylog.web.service.internalPort }}
                        initialDelaySeconds: 120
                        periodSeconds: 3
                        timeoutSeconds: 3
                        successThreshold: 1
                        failureThreshold: 4

In my slave just it is not master config change is there.

It is also giving error when it start ’ Index Rotation Problem!’ and slave is not working.

just with your few log lines it is hard to tell.

but i guess something with the connection between Graylog and Elasticsearch and/or Graylog and MongoDB might be the issue. But as I do not have Kubernetes experience someone else might be better help.

1 Like

Thankyou so much for reply can you please help me i this case

2019-05-06 07:36:24,598 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE] 2019-05-06 07:36:24,695 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running. 2019-05-06 07:48:10,198 ERROR: org.glassfish.jersey.server.ServerRuntime$Responder - An I/O error has occurred while writing a response message entity to the container output stream. org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed

this is the error i am getting inside my pod and pod got restarted again in kubernetes if you have any idea.

Thanks once again

do you have mongodb and elasticsearch running and reachable for Graylog?

Thankyou for reply Yes it is running in kubernetes cluster i have setup single pod for mongodb and single pod for elastic search.

the question - can Graylog connect to them? How did you check that this communication is possible.

yes graylog is running now but when my kubernetes pod restart i checked logs and it was giving that error i have shared.

ok - but that wasn’t readable from your message.

It wasn’t clear from your decription. As I do not have Kubernetes I can’t look into that. Maybe someone else can.

Thankyou for reply no worries some how i will be able figure out. thanks once again.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.