Unable to call on node caused by Socket closed

Graylog Central (peer support)
1

Hi,

Lately I have been getting this WARN very often, is it something normal?

2019-06-05 17:19:00,634 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://apps.dev.ca/graylog/api/system/jobs on node <c3201551-f5d9-41c4-8b89-20b4c843c2ca>
java.net.SocketTimeoutException: timeout
        at okio.Okio$4.newTimeoutException(Okio.java:232) ~[graylog.jar:?]
        at okio.AsyncTimeout.exit(AsyncTimeout.java:285) ~[graylog.jar:?]
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) ~[graylog.jar:?]
        at okio.RealBufferedSource.indexOf(RealBufferedSource.java:355) ~[graylog.jar:?]
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227) ~[graylog.jar:?]
        at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215) ~[graylog.jar:?]
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:61) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) ~[graylog.jar:?]
        at okhttp3.RealCall.execute(RealCall.java:77) ~[graylog.jar:?]
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
        at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:78) ~[graylog.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
Caused by: java.net.SocketException: Socket closed
        at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_192]
        at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_192]
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[?:1.8.0_192]
        at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[?:1.8.0_192]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) ~[?:1.8.0_192]
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933) ~[?:1.8.0_192]
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[?:1.8.0_192]
        at okio.Okio$2.read(Okio.java:140) ~[graylog.jar:?]
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
        ... 28 more

When connecting to the url https://apps.dev.ca/graylog/api/system/jobs using curl:

[root@log1 graylog]# curl -v https://apps.dev.ca/graylog/api/system/jobs
* About to connect() to apps.dev.ca port 443 (#0)
*   Trying 40.85.226.67...
* Connected to apps.dev.ca (40.85.226.67) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Server certificate:
*       subject: CN=*.dev.ca
*       start date: May 08 22:49:50 2019 GMT
*       expire date: Aug 06 22:49:50 2019 GMT
*       common name: *.dev.ca
*       issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
* NSS error -8179 (SEC_ERROR_UNKNOWN_ISSUER)
* Peer's Certificate issuer is not recognized.
* Closing connection 0
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Then I proceeded to import the certificates in cacerts as follows:

wget "https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt" --output-document=letsencryptauthorityx3.pem

openssl x509 -outform der -in letsencryptauthorityx3.pem -out letsencryptauthorityx3.der

[root@log1 certs]# keytool -import -alias letsencryptauthorityx3 -keystore cacerts -file letsencryptauthorityx3.der

openssl x509 -outform der -in trustid-x3-root.pem -out trustid-x3-root.der

keytool -import -alias trustid-x3-root -keystore cacerts -file trustid-x3-root.der

However, I continue getting the WARN message. Do you know what else would cause this problem? I have been digging the logs in Graylog because I have problems in getting messages of the logs in Graylog. Filebeat is shipping the logs, but Graylog is not displaying them when searching in the UI and just this Warning message I am getting in server.log.

Thanks for your help

2

Graylog might have to much load that is why it can’t answer to its own keepalive questions (if you have a single host setup).

3

Thank you so much for your reply Jan.

Graylog is running on a single server with 4GB in memory and 32GB in memory for ES. The current status of the node is as follows:

nodes
nodes.jpg1872×648 57.1 KB

The server has 4 process with Intel® Xeon® CPU E5-2673 v4 @ 2.30GHz.

I have the following metrics in the node:

**Incoming Messages:**
Total:
191,100 events
Mean:
3.08 events/second
1 minute avg:
21.86 events/second
5 minute avg:
21.44 events/second
15 minute avg:
15.44 events/second

**Processed Messages:**
Total:
2,871 events
Mean:
0.05 events/second
1 minute avg:
0 events/second
5 minute avg:
0 events/second
15 minute avg:
0 events/second

**Graylog throughput input:**
Counter
Value:
191,100

**Graylog throughput Output:**
Counter
Value:
2,866

**Traffic input:**
Counter
Value:
516,601,242

**Traffic output:**
Counter
Value:
2,982,736

Please let me know your insights about how to alleviate the performance. I will start by doubling the memory assigned to Graylog to have 8GB.

Thanks a lot

4

How much RAM does this Computer have?

It is not always a RAM problem - it could be I/O, it could be CPU.

If you have Elasticsearch and Graylog on the same box they fight for the CPU. it could be a better to limit the number of processors that are available for Elasticsearch, because it will take all available cores if not limits are given.

5

Thanks for your reply Jan,

This server has 62 GB in RAM. I will check how to limit number of processors available for ES. Considering 4 processors as the following:

Architecture:          x86_64
Processors:           4
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Stepping:              1
CPU MHz:               2294.683
BogoMIPS:              4589.36
Virtualization:        VT-x
Hypervisor vendor:     Microsoft
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              51200K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq vmx ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch tpr_shadow vnmi ept vpid fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt

How many processors should be desirable to have for ES?

Thanks a lot

Cheers,
Arturo

6

I would stick ES to 1 core …

as you have Graylog running on that machine too

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.