Hello all.
Graylog particulars:
Single 8 core 16 GB RAM instance running Graylog-server 2.5.2
Nginx reverse proxy on 443 to 9000 for TLS
MongoDB cluster on different hardware
ES cluster on different hardware
I am currently experiencing a very strange and new issue with my Graylog instance in which the REST/web APIs seem to just die. A restart of the service brings it back to life and it keeps on chugging. The HTTP and GELF endpoints are still up and processing logs and I can not for the life of me find out what the issue is as the graylog-server.log yields nothing when it fails and in DEBUG I can not seem to narrow it down.
What I have found is that there seems to be a difference with “SSLConnectionContext” in that when the REST/web API is down, I see a lot of lines like such:
2021-02-25T17:39:11.273Z DEBUG [SSLConnectionContext] wrap engine: sun.security.ssl.SSLEngineImpl@1d36918f input: ByteBufferWrapper (1897816865) [visible=[java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]]] output: ByteBufferWrapper (1851544420) [visible=[java.nio.HeapByteBuffer[pos=0 lim=33418 cap=33418]]]
2021-02-25T17:39:11.273Z DEBUG [SSLConnectionContext] wrap done engine: sun.security.ssl.SSLEngineImpl@1d36918f result: Status = OK HandshakeStatus = NEED_WRAP
bytesConsumed = 0 bytesProduced = 0 input: ByteBufferWrapper (1897816865) [visible=[java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]]] output: ByteBufferWrapper (1851544420) [visible=[java.nio.HeapByteBuffer[pos=0 lim=33418 cap=33418]]]
But after a restart of the service, I see these entries for SSLConnectionContext:
2021-02-25T19:35:58.796Z DEBUG [SSLConnectionContext] wrap engine: sun.security.ssl.SSLEngineImpl@4bf72ef input: HeapBuffer (968133336) [pos=0 lim=203 cap=2048] output: ByteBufferWrapper (961877094) [visible=[java.nio.HeapByteBuffer[pos=0 lim=25063 cap=25063]]]
2021-02-25T19:35:58.796Z DEBUG [SSLConnectionContext] wrap done engine: sun.security.ssl.SSLEngineImpl@4bf72ef result: Status = OK HandshakeStatus = NOT_HANDSHAKING
Scratching my head on this one. Any assistance would be greatly appreciated. Pasting in my webconfig bits as well as Nginx config for clarity:
rest_listen_uri = https://fqdn.of.server:9000/api/
web_listen_uri = https://fqdn.of.server:9000/
web_endpoint_uri = https://fqdn.of.server:9000/api
server {
listen 443 ssl;
ssl_certificate /path/to/cert.pem;
ssl_certificate_key /path/to/key.pem;
server_name graylog-server;
location / {
proxy_pass https://fqdn.of.server:9000/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass_request_headers on;
proxy_connect_timeout 150;
proxy_send_timeout 100;
proxy_read_timeout 100;
proxy_buffers 4 32k;
client_max_body_size 8m;
client_body_buffer_size 128k;
}
}
server {
listen 12202 ssl;
ssl_certificate /path/to/cert.pem;
ssl_certificate_key /path/to/key.pem;
server_name graylog-server-gelf;
location /gelf {
proxy_pass http://127.0.0.1:12201/gelf;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass_request_headers on;
proxy_connect_timeout 150;
proxy_send_timeout 100;
proxy_read_timeout 100;
proxy_buffers 4 32k;
client_max_body_size 8m;
client_body_buffer_size 128k;
}
}