Hi,
I got an issue when connecting to an RabbitMQ cluster from graylog.
The connection is lost and the graylog is making a new connection every 1 second.
I do not know what could cause this, but I got in rabbitmq admin interface 800 connections and the rabbitmq cluster does not respoding anymore.
I try to stop the inputs from graylog interface,but in the graylog logs I see that the graylog node keep trying connecting to rabbitmq. A restart it may fix this, but the graylog automticaly start the amqp input, and I have to move the input to put it to an local gl-node and stop it , and then restart the other GL nodes, and then handle the rabbitmq cluster.
So in the first place, whe the graylog keeps trying to connect to graylog cluster if I stop the amqp input
And can the reconnect interval can be modified as I want? 1 second retry is very often.
Here are some logs
2018-04-22T06:48:45.283Z ERROR [AmqpConsumer] AMQP connection lost! Trying reconnect in 1 second.
2018-04-22T06:48:45.283Z ERROR [AmqpConsumer] AMQP connection lost! Trying reconnect in 1 second.
2018-04-22T06:48:45.283Z ERROR [AmqpConsumer] Could not re-connect to AMQP broker.
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002) ~[?:1.8.0_152]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:1.8.0_152]
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757) ~[?:1.8.0_152]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[?:1.8.0_152]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_152]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_152]
at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[?:1.8.0_152]
at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:147) ~[graylog.jar:?]
at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:153) ~[graylog.jar:?]
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:292) ~[graylog.jar:?]
at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:62) ~[graylog.jar:?]
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:99) ~[graylog.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:911) ~[graylog.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:870) ~[graylog.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:828) ~[graylog.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:966) ~[graylog.jar:?]
at org.graylog2.inputs.transports.AmqpConsumer.connect(AmqpConsumer.java:176) ~[graylog.jar:?]
at org.graylog2.inputs.transports.AmqpConsumer$3.shutdownCompleted(AmqpConsumer.java:206) [graylog.jar:?]
at com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(ShutdownNotifierComponent.java:77) [graylog.jar:?]
at com.rabbitmq.client.impl.AMQConnection.doFinalShutdown(AMQConnection.java:689) [graylog.jar:?]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:584) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505) ~[?:1.8.0_152]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ~[?:1.8.0_152]
… 21 more
2018-04-22T06:48:45.283Z ERROR [AmqpConsumer] AMQP connection lost! Trying reconnect in 1 second.
2018-04-22T06:48:45.283Z ERROR [AmqpConsumer] Could not re-connect to AMQP broker.
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002) ~[?:1.8.0_152]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:1.8.0_152]
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757) ~[?:1.8.0_152]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[?:1.8.0_152]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_152]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_152]
at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[?:1.8.0_152]
at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:147) ~[graylog.jar:?]
at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:153) ~[graylog.jar:?]
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:292) ~[graylog.jar:?]
at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:62) ~[graylog.jar:?]
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:99) ~[graylog.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:911) ~[graylog.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:870) ~[graylog.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:828) ~[graylog.jar:?]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:966) ~[graylog.jar:?]
at org.graylog2.inputs.transports.AmqpConsumer.connect(AmqpConsumer.java:176) ~[graylog.jar:?]
at org.graylog2.inputs.transports.AmqpConsumer$3.shutdownCompleted(AmqpConsumer.java:206) [graylog.jar:?]
at com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(ShutdownNotifierComponent.java:77) [graylog.jar:?]
at com.rabbitmq.client.impl.AMQConnection.doFinalShutdown(AMQConnection.java:689) [graylog.jar:?]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:584) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505) ~[?:1.8.0_152]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ~[?:1.8.0_152]
… 21 more