Hi,
I’ve had some issues with Graylog in last couple of days. At some point it stops processing messages. New messages are being added to process buffer, so it fills up quickly, then the same with yournal.
Once I remove journal and restart Graylog, it works fine for a while.
Graylog 3.2.1
ES 6.5.4
Logstash GELF UPD input
Here are Graylog errors in logs that I found:
2020-02-12 08:52:10,144 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=f47a6af0-4d74-11ea-9f76-0242ac110006, journalOffset=117014, codec=gelf, payloadSize=507, timestamp=2020-02-12T08:52:10.143Z, remoteAddress=/<IP>:38753}
java.lang.IllegalArgumentException: GELF message <f47a6af0-4d74-11ea-9f76-0242ac110006> (received from < <IP>:38753>) has invalid "short_message":
at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:255) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66-internal]
2020-02-12 08:52:10,145 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Unable to decode raw message RawMessage{id=f47a6af1-4d74-11ea-9f76-0242ac110006, journalOffset=117015, codec=gelf, payloadSize=515, timestamp=2020-02-12T08:52:10.143Z, remoteAddress=/<IP>:38753} on input <5b90f0f54cedfd00145d2ebd>.
2020-02-24 10:07:20,308 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=71b35f00-56ed-11ea-af1d-0242ac110006, journalOffset=1620208, codec=gelf, payloadSize=5510, timestamp=2020-02-24T10:07:20.304Z, remoteAddress=/<IP>:34946}
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'u was expecting ('true', 'false' or 'null')
{A��ew�wU��ZF2��sriF�|����i�梿>�ӷ>*�����7��>*�%rŒA2SkOW
������&}2kP܆�~g�+lC���jZ�+-�_I]�Qqu��u�<�pA�b�3ô��:x^�����44����Vf�n�M��l����3j���j)0�H���W�`����M=EP�8;�2`��v^����X���\� ҹ�h+�$_���{Nx��~\B������-5(I6_%yͮ��A��J5?_I���gg���x��Ez�<Y|��v��R�8����3I��O��V鵧�UI'��?����C�4�▒▒ڜ��X���}s�>��n����x�w��
&JJ%���[`�$�1Y�,�A�d��1���8�<�MM
ʜ�h�~�q��Q$/�R��t�.�ەB���y��cYhX��%�!�+��`Hrɀ��\Iw��u!Η��1��AT���,�`E
t������${�x�0��B�L��"[truncated 4735 chars]; line: 1, column: 3]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
xterm-256colorxterm-256color at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
xterm-256color at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66-internal]
2020-02-24 10:13:15,599 ERROR: org.glassfish.jersey.server.ServerRuntime$Responder - An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection closed
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_66-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_66-internal]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66-internal]
Caused by: java.io.IOException: Connection closed
at org.glassfish.grizzly.asyncqueue.TaskQueue.onClose(TaskQueue.java:331) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.onClose(AbstractNIOAsyncQueueWriter.java:501) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.closeConnection(TCPNIOTransport.java:402) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.NIOConnection.doClose(NIOConnection.java:647) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.NIOConnection$6.run(NIOConnection.java:613) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.DefaultSelectorHandler.execute(DefaultSelectorHandler.java:238) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:607) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:291) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:136) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:108) ~[graylog.jar:?]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:526) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) ~[graylog.jar:?]
... 1 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_66-internal]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_66-internal]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_66-internal]
at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_66-internal]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_66-internal]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeCompositeBuffer(TCPNIOUtils.java:86) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:129) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:108) ~[graylog.jar:?]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:526) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) ~[graylog.jar:?]
... 1 more
2020-02-24 10:01:34,342 ERROR: org.graylog2.inputs.codecs.GelfChunkAggregator - Received duplicate chunk 86 for message 2b985c60d7f868ec from null
2020-02-24 10:01:34,343 ERROR: org.graylog2.inputs.codecs.GelfChunkAggregator - Received duplicate chunk 95 for message 2b985c60d7f868ec from null
2020-02-24 10:01:34,343 ERROR: org.graylog2.inputs.codecs.GelfChunkAggregator - Received duplicate chunk 98 for message 2b985c60d7f868ec from null
Can I get some help, please?
If any config needed, I’ll add it, just let know what.
Thanks,
Daniel