Multinode graylog connectivity error

Graylog Central (peer support)
1

Hi ,
I am using graylog -3 multinode cluster ,1 master 2 datanodes , Once the service started , the cluster is running fine with all 3 nodes. After few minutes, 2 nodes went down and we are continuously getting below error . Help me sort out this.

2019-07-23 12:17:36,491 WARN : org.graylog2.buffers.processors.OutputBufferProcessor - Timeout reached. Not waiting any
longer for stream output writer threads to complete.
2019-07-23 12:17:18,881 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://10.24.1.26:91
00/api/system/metrics/multiple on node <53d0d47f-b1b9-43ba-8768-941fdb99590e>
java.net.SocketTimeoutException: timeout
        at okio.Okio$4.newTimeoutException(Okio.java:232) ~[graylog.jar:?]
        at okio.AsyncTimeout.exit(AsyncTimeout.java:285) ~[graylog.jar:?]
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) ~[graylog.jar:?]
        at okio.RealBufferedSource.indexOf(RealBufferedSource.java:355) ~[graylog.jar:?]
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227) ~[graylog.jar:?]
        at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215) ~[graylog.jar:?]
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) ~[graylog.j
ar:?]

Thanks,
Jay’

2

I guess you have configured the same IP in the http_* settings for all 3 nodes?

As you can see - guessing is the only option we have as you did share only your problem but not your configuration.

3

Hi Jan,
same ip for HTTP_EXTERNAL_URI or "GRAYLOG_HTTP_BIND_ADDRESS?
Please find my configuration for master and data node

- name: GRAYLOG_HTTP_EXTERNAL_URI
          value: http://abc.cloudapp.azure.com:30033/
        
        - name: "GRAYLOG_HTTP_BIND_ADDRESS"
          value: "0.0.0.0:9000"

Thanks,
Jay’

4 
2019-07-24 13:06:10,216 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://10.244.1.46:90
00/api/system/inputstates on node <b8eceadc-6da8-4d41-b74d-8a2ee205ac06>
java.net.SocketTimeoutException: timeout
        at okio.Okio$4.newTimeoutException(Okio.java:232) ~[graylog.jar:?]
        at okio.AsyncTimeout.exit(AsyncTimeout.java:275) ~[graylog.jar:?]
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:243) ~[graylog.jar:?]
        at okio.RealBufferedSource.indexOf(RealBufferedSource.java:355) ~[graylog.jar:?]
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227) ~[graylog.jar:?]
        at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215) ~[graylog.jar:?]
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]

Above error i am getting.
Thanks,
Jay’

5

are the Graylog Nodes able to communicate with each other?

That is what the publish URI is used for - internal cluster communication.

6

Yes , they are communicating with each others .Only problem is for very less data also after few minutes heap slowly reaching the max level and stopped the cluster. I increased cpu 4 core memory till 8 GB. But no use. Need u r help to fix.
Thanks

7

Below is my configuration

      - name: graylog3
        image: graylog/graylog:3.0      
        env:
        - name: GRAYLOG_IS_MASTER
          value: "true"
        - name: node_id_file
          value: MASTER
        - name: GRAYLOG_PASSWORD_SECRET
          value: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
        - name: GRAYLOG_ROOT_PASSWORD_SHA2
          value: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
        - name: GRAYLOG_HTTP_EXTERNAL_URI
          value: http://abc.cloudapp.azure.com:30033/
        - name: GRAYLOG_ELASTICSEARCH_HOSTS
          value: http://elasticsearch-master:9200
        - name: GRAYLOG_MONGODB_URI
          value: mongodb://graylog-mongodb-mongodb-replicaset:27017/graylog
       - name: "GRAYLOG_SERVER_JAVA_OPTS"
          value: "-Xms8g -Xmx8g"
        - name: "GRAYLOG_HTTP_BIND_ADDRESS"
          value: "0.0.0.0:9000"
        ports:
        - containerPort: 9000
        - containerPort: 12201
        - containerPort: 5044
        volumeMounts:
        - name: graylog-plugins
          mountPath: /usr/share/graylog/plugin
        resources:
          limits:
            cpu: 4.0
            memory: "9Gi"
          requests:
            cpu: 0.5
            memory: "4Gi"

I am processing very less volume .
Some issue with heap.

Thanks,

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.