Grylog taking much more time to start

Graylog Central (peer support)
1

Graylog version 3
Elastic search 6.5

I am running graylog helm chart to setup graylog in kubernetes. but i am getting error in when i do kubectl get describe pod <>

error is : Readiness probe failed: Get http://10.8.2.67:9000/api/system/lbstatus: dial tcp 10.8.2.67:9000: connect: connection refused

After very long time pod is starting but still slave if not working and in master log i have found one issue

2019-05-03 06:27:29,576 ERROR: org.graylog2.indexer.fieldtypes.IndexFieldTypePoller - Couldn't get mapping for index <graylog_0>: null
2019-05-03 06:27:31,277 ERROR: org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical - Couldn't update field types for index set <Default index set/5ccaa268d1c68b000e51e3f4>
com.mongodb.MongoInterruptedException: Interrupted acquiring a permit to retrieve an item from the pool

while my slave is not staring only.

containers:
                    - name: graylog-master
                      image: "graylog/graylog:3.0.1"
                      securityContext:
                        privileged: true
                        runAsUser: 1100
                      env:
                      - name: GRAYLOG_PASSWORD_SECRET
                        value: {{required "A valid .Values.graylog.passwordSecret entry required!" .Values.graylog.passwordSecret }}
                      - name: GRAYLOG_ROOT_PASSWORD_SHA2
                        value: {{ .Values.graylog.passwordSecret | sha256sum }}
                      - name: GRAYLOG_MONGODB_URI
                        value: "mongodb://{{ $mongodbServiceName }}-0.{{ $mongodbServiceName }}:27017/graylog?replicaSet=rs0"
                      - name: GRAYLOG_IS_MASTER
                        value: "true"
                      - name: GRAYLOG_ELASTICSEARCH_DISCOVERY_ZEN_PING_UNICAST_HOSTS
                        value: "{{ $elasticsearchServiceName }}-0.{{ $elasticsearchServiceName }}:9300"
                      - name: GRAYLOG_ELASTICSEARCH_HOSTS
                        value: "http://{{ $elasticsearchServiceName }}-0.{{ $elasticsearchServiceName }}:9200"
                      - name: GRAYLOG_HTTP_PUBLISH_URI  
                        value: "https://<MY URL>"
                      - name: GRAYLOG_HTTP_EXTERNAL_URI  
                        value: "https://<MY URL>"
                      - name: GRAYLOG_SERVER_JAVA_OPTS
                        value: "-Xms1g -Xmx1g -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow"
                      ports:
                      - containerPort: {{ .Values.graylog.web.service.internalPort }}
                        name: http
                      - containerPort: 12201
                        name: udp-input
                      - containerPort: 1514
                        name: tcp-input
                      volumeMounts:
                      - name: graylog-persistent-storage
                        mountPath: /usr/share/graylog/data/journal
                        subPath: graylog
                      resources:
            {{ toYaml .Values.graylog.master.containers.resources | indent 12 }}
                      readinessProbe:
                        httpGet:
                          path: /api/system/lbstatus
                          port: {{ .Values.graylog.web.service.internalPort }}
                        initialDelaySeconds: 120
                        periodSeconds: 3
                        timeoutSeconds: 3
                        successThreshold: 1
                        failureThreshold: 4

In my slave just it is not master config change is there.

It is also giving error when it start ’ Index Rotation Problem!’ and slave is not working.

2

just with your few log lines it is hard to tell.

but i guess something with the connection between Graylog and Elasticsearch and/or Graylog and MongoDB might be the issue. But as I do not have Kubernetes experience someone else might be better help.

1 Like
3

Thankyou so much for reply can you please help me i this case

2019-05-06 07:36:24,598 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE] 2019-05-06 07:36:24,695 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running. 2019-05-06 07:48:10,198 ERROR: org.glassfish.jersey.server.ServerRuntime$Responder - An I/O error has occurred while writing a response message entity to the container output stream. org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed

this is the error i am getting inside my pod and pod got restarted again in kubernetes if you have any idea.

Thanks once again

4

do you have mongodb and elasticsearch running and reachable for Graylog?

5

Thankyou for reply Yes it is running in kubernetes cluster i have setup single pod for mongodb and single pod for elastic search.

6

the question - can Graylog connect to them? How did you check that this communication is possible.

7

yes graylog is running now but when my kubernetes pod restart i checked logs and it was giving that error i have shared.

8

ok - but that wasn’t readable from your message.

It wasn’t clear from your decription. As I do not have Kubernetes I can’t look into that. Maybe someone else can.

9

Thankyou for reply no worries some how i will be able figure out. thanks once again.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.