Graylog blinking server unavailable


(Sam) #1

Hi!

I have installed Graylog 2.2.3+7adc951 on a Linux 3.13.0-115-generic from the AWS AMI. I’m having an issue with the graylog web interface constantly resetting that the server is unavailable when I hit certain pages like /search. Note: This is a single node setup.

Server currently unavailable:
“We are experiencing problems connecting to the Graylog server running on https://graylog.example.org/api." please verify that the server is healthy and working correctly. You will be automatically redirected to the previous page once we can connect to the server.”

Then it redirects back to the /search page only to redirect back to this unavailable page.

Here are the logs when I run sudo graylog-ctl tail server

2017-07-28_00:41:03.57473 Caused by: java.io.IOException: Connection reset by peer
2017-07-28_00:41:03.57521 	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_101]
2017-07-28_00:41:03.57648 	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_101]
2017-07-28_00:41:03.57723 	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_101]
2017-07-28_00:41:03.57814 	at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_101]
2017-07-28_00:41:03.57853 	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_101]
2017-07-28_00:41:03.57990 	at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) ~[graylog.jar:?]
2017-07-28_00:41:03.58099 	at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeCompositeBuffer(TCPNIOUtils.java:86) ~[graylog.jar:?]
2017-07-28_00:41:03.58131 	at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:129) ~[graylog.jar:?]
2017-07-28_00:41:03.58193 	at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) ~[graylog.jar:?]
2017-07-28_00:41:03.58355 	at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:260) ~[graylog.jar:?]
2017-07-28_00:41:03.58431 	at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:169) ~[graylog.jar:?]
2017-07-28_00:41:03.58556 	at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:71) ~[graylog.jar:?]
2017-07-28_00:41:03.58600 	at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleWrite(TCPNIOTransportFilter.java:126) ~[graylog.jar:?]
2017-07-28_00:41:03.58791 	at org.glassfish.grizzly.filterchain.TransportFilter.handleWrite(TransportFilter.java:191) ~[graylog.jar:?]
2017-07-28_00:41:03.58869 	at org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111) ~[graylog.jar:?]
2017-07-28_00:41:03.58901 	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) ~[graylog.jar:?]
2017-07-28_00:41:03.59153 	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) ~[graylog.jar:?]
2017-07-28_00:41:03.59205 	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) ~[graylog.jar:?]
2017-07-28_00:41:03.59262 	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) ~[graylog.jar:?]
2017-07-28_00:41:03.59307 	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[graylog.jar:?]
2017-07-28_00:41:03.59346 	at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:890) ~[graylog.jar:?]
2017-07-28_00:41:03.59428 	at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:858) ~[graylog.jar:?]
2017-07-28_00:41:03.59467 	at org.glassfish.grizzly.http.io.OutputBuffer.flushBuffer(OutputBuffer.java:1059) ~[graylog.jar:?]
2017-07-28_00:41:03.59682 	at org.glassfish.grizzly.http.io.OutputBuffer.flushBinaryBuffers(OutputBuffer.java:1046) ~[graylog.jar:?]
2017-07-28_00:41:03.59721 	at org.glassfish.grizzly.http.io.OutputBuffer.flushAllBuffers(OutputBuffer.java:1017) ~[graylog.jar:?]
2017-07-28_00:41:03.59764 	at org.glassfish.grizzly.http.io.OutputBuffer.flush(OutputBuffer.java:766) ~[graylog.jar:?]
2017-07-28_00:41:03.59983 	at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.flush(NIOOutputStreamImpl.java:90) ~[graylog.jar:?]
2017-07-28_00:41:03.60191 	at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:292) ~[graylog.jar:?]
2017-07-28_00:41:03.60223 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.flush(WriterInterceptorExecutor.java:304) ~[graylog.jar:?]
2017-07-28_00:41:03.60272 	at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1048) ~[graylog.jar:?]
2017-07-28_00:41:03.60526 	at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:953) ~[graylog.jar:?]
2017-07-28_00:41:03.60645 	at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
2017-07-28_00:41:03.60764 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
2017-07-28_00:41:03.60790 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
2017-07-28_00:41:03.61076 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
2017-07-28_00:41:03.61182 	at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
2017-07-28_00:41:03.61257 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
2017-07-28_00:41:03.61274 	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
2017-07-28_00:41:03.61336 	... 20 more
2017-07-28_00:41:18.97213 2017-07-28 00:41:18,967 ERROR: org.glassfish.jersey.server.ServerRuntime$Responder - An I/O error has occurred while writing a response message entity to the container output stream.
2017-07-28_00:41:18.97251 org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed
2017-07-28_00:41:18.97309 	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) ~[graylog.jar:?]
2017-07-28_00:41:18.97396 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
2017-07-28_00:41:18.97442 	at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) ~[graylog.jar:?]
2017-07-28_00:41:18.97521 	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) [graylog.jar:?]
2017-07-28_00:41:18.97577 	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) [graylog.jar:?]
2017-07-28_00:41:18.97649 	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) [graylog.jar:?]
2017-07-28_00:41:18.97692 	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) [graylog.jar:?]
2017-07-28_00:41:18.97966 	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
2017-07-28_00:41:18.98016 	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
2017-07-28_00:41:18.98107 	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
2017-07-28_00:41:18.98169 	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
2017-07-28_00:41:18.98273 	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
2017-07-28_00:41:18.98342 	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
2017-07-28_00:41:18.98425 	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
2017-07-28_00:41:18.98458 	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
2017-07-28_00:41:18.98553 	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
2017-07-28_00:41:18.98594 	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
2017-07-28_00:41:18.98730 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
2017-07-28_00:41:18.98767 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
2017-07-28_00:41:18.98812 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
2017-07-28_00:41:18.98851 	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
2017-07-28_00:41:18.98921 Caused by: java.io.IOException: Connection is closed
2017-07-28_00:41:18.98962 	at org.glassfish.grizzly.nio.NIOConnection.assertOpen(NIOConnection.java:445) ~[graylog.jar:?]
2017-07-28_00:41:18.99048 	at org.glassfish.grizzly.http.io.OutputBuffer.flush(OutputBuffer.java:763) ~[graylog.jar:?]
2017-07-28_00:41:18.99089 	at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.flush(NIOOutputStreamImpl.java:90) ~[graylog.jar:?]
2017-07-28_00:41:18.99155 	at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:292) ~[graylog.jar:?]
2017-07-28_00:41:18.99195 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.flush(WriterInterceptorExecutor.java:304) ~[graylog.jar:?]
2017-07-28_00:41:18.99296 	at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1048) ~[graylog.jar:?]
2017-07-28_00:41:18.99357 	at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:953) ~[graylog.jar:?]
2017-07-28_00:41:18.99427 	at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
2017-07-28_00:41:18.99547 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
2017-07-28_00:41:18.99584 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
2017-07-28_00:41:18.99612 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
2017-07-28_00:41:18.99637 	at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
2017-07-28_00:41:18.99673 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
2017-07-28_00:41:18.99707 	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]

(Jochen) #2

What are the specs of the virtual machine which is running the Graylog AMI?


(Sam) #3

It’s a t2.large instance, 8GB mem, 32GB ssd, 2 CPUs. Do you think it’s an issue with the machine?


(Jochen) #4

No, that should be fine. I wondered if you were using too little memory for all components to run on the same machine, but 8 GB memory is fine.


(Sam) #5

Any ideas why this would be happening? I tried installing a fresh version of graylog as well on a new machine and even with no inputs (except the default nginx, graylog-server, etcd) it is constantly crashing and refreshing.


(system) #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.