Hi!
I have installed Graylog 2.2.3+7adc951 on a Linux 3.13.0-115-generic from the AWS AMI. I’m having an issue with the graylog web interface constantly resetting that the server is unavailable when I hit certain pages like /search. Note: This is a single node setup.
Server currently unavailable:
“We are experiencing problems connecting to the Graylog server running on https://graylog.example.org/api." please verify that the server is healthy and working correctly. You will be automatically redirected to the previous page once we can connect to the server.”
Then it redirects back to the /search page only to redirect back to this unavailable page.
Here are the logs when I run sudo graylog-ctl tail server
2017-07-28_00:41:03.57473 Caused by: java.io.IOException: Connection reset by peer
2017-07-28_00:41:03.57521 at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_101]
2017-07-28_00:41:03.57648 at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_101]
2017-07-28_00:41:03.57723 at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_101]
2017-07-28_00:41:03.57814 at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_101]
2017-07-28_00:41:03.57853 at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_101]
2017-07-28_00:41:03.57990 at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) ~[graylog.jar:?]
2017-07-28_00:41:03.58099 at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeCompositeBuffer(TCPNIOUtils.java:86) ~[graylog.jar:?]
2017-07-28_00:41:03.58131 at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:129) ~[graylog.jar:?]
2017-07-28_00:41:03.58193 at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) ~[graylog.jar:?]
2017-07-28_00:41:03.58355 at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:260) ~[graylog.jar:?]
2017-07-28_00:41:03.58431 at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:169) ~[graylog.jar:?]
2017-07-28_00:41:03.58556 at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:71) ~[graylog.jar:?]
2017-07-28_00:41:03.58600 at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleWrite(TCPNIOTransportFilter.java:126) ~[graylog.jar:?]
2017-07-28_00:41:03.58791 at org.glassfish.grizzly.filterchain.TransportFilter.handleWrite(TransportFilter.java:191) ~[graylog.jar:?]
2017-07-28_00:41:03.58869 at org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111) ~[graylog.jar:?]
2017-07-28_00:41:03.58901 at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) ~[graylog.jar:?]
2017-07-28_00:41:03.59153 at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) ~[graylog.jar:?]
2017-07-28_00:41:03.59205 at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) ~[graylog.jar:?]
2017-07-28_00:41:03.59262 at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) ~[graylog.jar:?]
2017-07-28_00:41:03.59307 at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[graylog.jar:?]
2017-07-28_00:41:03.59346 at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:890) ~[graylog.jar:?]
2017-07-28_00:41:03.59428 at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:858) ~[graylog.jar:?]
2017-07-28_00:41:03.59467 at org.glassfish.grizzly.http.io.OutputBuffer.flushBuffer(OutputBuffer.java:1059) ~[graylog.jar:?]
2017-07-28_00:41:03.59682 at org.glassfish.grizzly.http.io.OutputBuffer.flushBinaryBuffers(OutputBuffer.java:1046) ~[graylog.jar:?]
2017-07-28_00:41:03.59721 at org.glassfish.grizzly.http.io.OutputBuffer.flushAllBuffers(OutputBuffer.java:1017) ~[graylog.jar:?]
2017-07-28_00:41:03.59764 at org.glassfish.grizzly.http.io.OutputBuffer.flush(OutputBuffer.java:766) ~[graylog.jar:?]
2017-07-28_00:41:03.59983 at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.flush(NIOOutputStreamImpl.java:90) ~[graylog.jar:?]
2017-07-28_00:41:03.60191 at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:292) ~[graylog.jar:?]
2017-07-28_00:41:03.60223 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.flush(WriterInterceptorExecutor.java:304) ~[graylog.jar:?]
2017-07-28_00:41:03.60272 at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1048) ~[graylog.jar:?]
2017-07-28_00:41:03.60526 at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:953) ~[graylog.jar:?]
2017-07-28_00:41:03.60645 at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
2017-07-28_00:41:03.60764 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
2017-07-28_00:41:03.60790 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
2017-07-28_00:41:03.61076 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
2017-07-28_00:41:03.61182 at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
2017-07-28_00:41:03.61257 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
2017-07-28_00:41:03.61274 at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
2017-07-28_00:41:03.61336 ... 20 more
2017-07-28_00:41:18.97213 2017-07-28 00:41:18,967 ERROR: org.glassfish.jersey.server.ServerRuntime$Responder - An I/O error has occurred while writing a response message entity to the container output stream.
2017-07-28_00:41:18.97251 org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed
2017-07-28_00:41:18.97309 at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) ~[graylog.jar:?]
2017-07-28_00:41:18.97396 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
2017-07-28_00:41:18.97442 at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) ~[graylog.jar:?]
2017-07-28_00:41:18.97521 at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) [graylog.jar:?]
2017-07-28_00:41:18.97577 at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) [graylog.jar:?]
2017-07-28_00:41:18.97649 at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) [graylog.jar:?]
2017-07-28_00:41:18.97692 at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) [graylog.jar:?]
2017-07-28_00:41:18.97966 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
2017-07-28_00:41:18.98016 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
2017-07-28_00:41:18.98107 at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
2017-07-28_00:41:18.98169 at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
2017-07-28_00:41:18.98273 at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
2017-07-28_00:41:18.98342 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
2017-07-28_00:41:18.98425 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
2017-07-28_00:41:18.98458 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
2017-07-28_00:41:18.98553 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
2017-07-28_00:41:18.98594 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
2017-07-28_00:41:18.98730 at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
2017-07-28_00:41:18.98767 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
2017-07-28_00:41:18.98812 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
2017-07-28_00:41:18.98851 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
2017-07-28_00:41:18.98921 Caused by: java.io.IOException: Connection is closed
2017-07-28_00:41:18.98962 at org.glassfish.grizzly.nio.NIOConnection.assertOpen(NIOConnection.java:445) ~[graylog.jar:?]
2017-07-28_00:41:18.99048 at org.glassfish.grizzly.http.io.OutputBuffer.flush(OutputBuffer.java:763) ~[graylog.jar:?]
2017-07-28_00:41:18.99089 at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.flush(NIOOutputStreamImpl.java:90) ~[graylog.jar:?]
2017-07-28_00:41:18.99155 at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:292) ~[graylog.jar:?]
2017-07-28_00:41:18.99195 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.flush(WriterInterceptorExecutor.java:304) ~[graylog.jar:?]
2017-07-28_00:41:18.99296 at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1048) ~[graylog.jar:?]
2017-07-28_00:41:18.99357 at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:953) ~[graylog.jar:?]
2017-07-28_00:41:18.99427 at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
2017-07-28_00:41:18.99547 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
2017-07-28_00:41:18.99584 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
2017-07-28_00:41:18.99612 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
2017-07-28_00:41:18.99637 at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
2017-07-28_00:41:18.99673 at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
2017-07-28_00:41:18.99707 at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]