Hi
I’m getting error with graylog cluster.My graylog and elasticsesarch work correctly. input and output message on graylog sill work. But I can search anything in graylog search.
I got this error when tail graylog log:
" Caused by: java.io.IOException: Broken pipe"
How can I fix this?
Thanks
More logs “ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed”
Please post the complete logs of your Graylog and Elasticsearch nodes.
Here is the full log:
2018-03-06T17:06:57.789+07:00 ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: java.io.IOException: Connection is closed
at org.glassfish.grizzly.nio.NIOConnection.assertOpen(NIOConnection.java:445) ~[graylog.jar:?]
at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:677) ~[graylog.jar:?]
at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.write(NIOOutputStreamImpl.java:83) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1354) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1301) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:457) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:49) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951) ~[graylog.jar:?]
at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
... 20 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_144]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_144]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_144]
at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_144]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_144]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeCompositeBuffer(TCPNIOUtils.java:86) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:129) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:260) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:169) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:71) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleWrite(TCPNIOTransportFilter.java:126) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.TransportFilter.handleWrite(TransportFilter.java:191) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) ~[graylog.jar:?]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:890) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:858) ~[graylog.jar:?]
at org.glassfish.grizzly.http.io.OutputBuffer.flushBuffer(OutputBuffer.java:1059) ~[graylog.jar:?]
at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:709) ~[graylog.jar:?]
at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.write(NIOOutputStreamImpl.java:83) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1354) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1301) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:457) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:49) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951) ~[graylog.jar:?]
at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
... 20 more
Is this really the complete content of the logs produces by your Graylog nodes?
It looks like a single message to me.
Yep, there are all error log from graylog master node.
Please post the complete logs, not only the messages on the ERROR log level.
Body is limited to 32000 characters; you entered 40592.
Can I split them in 2 times.
2018-03-06T16:57:47.117+07:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 2.4.3 [org.graylog.aws.plugin.AWSPlugin]
2018-03-06T16:57:47.123+07:00 INFO [CmdLineTool] Loaded plugin: Elastic Beats Input 2.4.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-03-06T16:57:47.124+07:00 INFO [CmdLineTool] Loaded plugin: CEF Input 2.4.3 [org.graylog.plugins.cef.CEFInputPlugin]
2018-03-06T16:57:47.126+07:00 INFO [CmdLineTool] Loaded plugin: Collector 2.4.3 [org.graylog.plugins.collector.CollectorPlugin]
2018-03-06T16:57:47.128+07:00 INFO [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.4.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2018-03-06T16:57:47.130+07:00 INFO [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.4.3 [org.graylog.plugins.map.MapWidgetPlugin]
2018-03-06T16:57:47.131+07:00 INFO [CmdLineTool] Loaded plugin: NetFlow Plugin 2.4.3 [org.graylog.plugins.netflow.NetFlowPlugin]
2018-03-06T16:57:47.148+07:00 INFO [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.4.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-03-06T16:57:47.150+07:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.4.3 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2018-03-06T16:57:47.151+07:00 INFO [CmdLineTool] Loaded plugin: Slack 3.0.0 [org.graylog2.plugins.slack.callback.SlackAlarmCallback]
2018-03-06T16:57:48.077+07:00 INFO [CmdLineTool] Running with JVM arguments: -Djava.net.preferIPv4Stack=true -Xms4g -Xmx4g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2018-03-06T16:57:48.399+07:00 INFO [Version] HV000001: Hibernate Validator 5.1.3.Final
2018-03-06T16:57:51.707+07:00 INFO [InputBufferImpl] Message journal is enabled.
2018-03-06T16:57:51.736+07:00 INFO [NodeId] Node ID: b90ec773-0032-4bae-bc40-37e969c6018c
2018-03-06T16:57:51.986+07:00 INFO [LogManager] Loading logs.
2018-03-06T16:57:52.112+07:00 INFO [LogManager] Logs loading complete.
2018-03-06T16:57:52.112+07:00 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2018-03-06T16:57:52.142+07:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <131072> and wait strategy <BlockingWaitStrategy>, running 4 parallel message handlers.
2018-03-06T16:57:52.175+07:00 INFO [cluster] Cluster created with settings {hosts=[xxxxxx:27017, xxx:27017, xxx:27017], mode=MULTIPLE, requiredClusterType=REPLICA_SET, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000, requiredReplicaSetName='sector5-rs0'}
2018-03-06T16:57:52.177+07:00 INFO [cluster] Adding discovered server xxx:27017 to client view of cluster
2018-03-06T16:57:52.219+07:00 INFO [cluster] Adding discovered server xxx:27017 to client view of cluster
2018-03-06T16:57:52.250+07:00 INFO [cluster] Adding discovered server xxx:27017 to client view of cluster
2018-03-06T16:57:52.317+07:00 INFO [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=xxx:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=xxx:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=xxx:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2018-03-06T16:57:52.985+07:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:463554}] to xxx:27017
2018-03-06T16:57:52.998+07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=xxx:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 4]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=3674632, setName='sector5-rs0', canonicalAddress=xxx:27017, hosts=[xxx:27017, xxx:27017], passives=[xxx:27017], arbiters=[], primary='xxx:27017', tagSet=TagSet{[]}, electionId=null, setVersion=34, lastWriteDate=null, lastUpdateTimeNanos=4053199318463233}
2018-03-06T16:57:53.019+07:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:288147}] to xxx:27017
2018-03-06T16:57:53.021+07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=xxx:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 4]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=966202, setName='sector5-rs0', canonicalAddress=xxx:27017, hosts=[xxx:27017, xxx:27017], passives=[xxx:27017], arbiters=[], primary='xxx:27017', tagSet=TagSet{[]}, electionId=null, setVersion=34, lastWriteDate=null, lastUpdateTimeNanos=4053199342579108}
2018-03-06T16:57:53.022+07:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:1103452}] to xxx:27017
2018-03-06T16:57:53.024+07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=xxx:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 4]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=1121058, setName='sector5-rs0', canonicalAddress=xxx:27017, hosts=[xxx:27017, xxx:27017], passives=[xxx:27017], arbiters=[], primary='xxx:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000012, setVersion=34, lastWriteDate=null, lastUpdateTimeNanos=4053199345208760}
2018-03-06T16:57:53.024+07:00 INFO [cluster] Setting max election id to 7fffffff0000000000000012 from replica set primary xxx:27017
2018-03-06T16:57:53.024+07:00 INFO [cluster] Setting max set version to 34 from replica set primary xxx:27017
2018-03-06T16:57:53.025+07:00 INFO [cluster] Discovered replica set primary xxx:27017
2018-03-06T16:57:53.165+07:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:1103453}] to xxx:27017
2018-03-06T16:57:53.853+07:00 INFO [AbstractJestClient] Setting server pool to a list of 3 servers: [http://elas1:9200,http://elas2:9200,http://elas3:9200]
2018-03-06T16:57:53.856+07:00 INFO [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2018-03-06T16:57:54.011+07:00 INFO [JestClientFactory] Using custom ObjectMapper instance
2018-03-06T16:57:54.012+07:00 INFO [JestClientFactory] Node Discovery enabled...
2018-03-06T16:57:54.049+07:00 INFO [JestClientFactory] Idle connection reaping disabled...
2018-03-06T16:57:54.154+07:00 INFO [AbstractJestClient] Setting server pool to a list of 3 servers: [http://10.130.57.108:9200,http://10.130.42.18:9200,http://10.130.46.220:9200]
2018-03-06T16:57:54.487+07:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <131072> and wait strategy <BlockingWaitStrategy>.
2018-03-06T16:57:57.604+07:00 INFO [RulesEngineProvider] No static rules file loaded.
2018-03-06T16:57:57.906+07:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <131072> and wait strategy <BlockingWaitStrategy>.
2018-03-06T16:57:57.993+07:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:1103456}] to xxx:27017
2018-03-06T16:57:58.381+07:00 INFO [ServerBootstrap] Graylog server 2.4.3+2c41897 starting up
2018-03-06T16:57:58.382+07:00 INFO [ServerBootstrap] JRE: Oracle Corporation 1.8.0_144 on Linux 4.4.0-109-generic
2018-03-06T16:57:58.382+07:00 INFO [ServerBootstrap] Deployment: deb
2018-03-06T16:57:58.382+07:00 INFO [ServerBootstrap] OS: Ubuntu 16.04.3 LTS (xenial)
2018-03-06T16:57:58.382+07:00 INFO [ServerBootstrap] Arch: amd64
2018-03-06T16:57:58.385+07:00 WARN [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2018-03-06T16:57:58.418+07:00 INFO [PeriodicalsService] Starting 25 periodicals ...
2018-03-06T16:57:58.429+07:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2018-03-06T16:57:58.443+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.464+07:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2018-03-06T16:57:58.465+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.465+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.466+07:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2018-03-06T16:57:58.474+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.474+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRetentionThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.474+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRotationThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.474+07:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2018-03-06T16:57:58.475+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.VersionCheckThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.553+07:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2018-03-06T16:57:58.563+07:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2018-03-06T16:57:58.563+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.566+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.568+07:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2018-03-06T16:57:58.569+07:00 INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2018-03-06T16:57:58.569+07:00 INFO [PeriodicalsService] Not starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.569+07:00 INFO [PeriodicalsService] Not starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.644+07:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:1103457}] to xxx:27017
2018-03-06T16:57:58.680+07:00 INFO [connection] Opened connection [connectionId{localValue:7, serverValue:1103458}] to xxx:27017
2018-03-06T16:57:58.699+07:00 INFO [LookupTableService] Data Adapter abuse-ch-ransomware-domains/5a40782b84e00e7f34f46e87 [ 74df32be] STARTING
2018-03-06T16:57:58.699+07:00 ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-domains/5a40782b84e00e7f34f46e87/ 74df32be>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2018-03-06T16:57:58.712+07:00 INFO [LookupTableService] Data Adapter abuse-ch-ransomware-domains/5a40782b84e00e7f34f46e87 [ 74df32be] RUNNING
2018-03-06T16:57:58.723+07:00 INFO [LookupTableService] Data Adapter otx-api-domain/5a40782b84e00e7f34f46e8c [ 1b80df02] STARTING
2018-03-06T16:57:58.724+07:00 WARN [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2018-03-06T16:57:58.749+07:00 INFO [LookupTableService] Data Adapter whois/5a40782b84e00e7f34f46e8a [ 29abb3d8] STARTING
2018-03-06T16:57:58.777+07:00 INFO [LookupTableService] Data Adapter whois/5a40782b84e00e7f34f46e8a [ 29abb3d8] RUNNING
2018-03-06T16:57:58.777+07:00 INFO [LookupTableService] Data Adapter test/5992cfe567ec180b2e1a8145 [ 61f57df1] STARTING
2018-03-06T16:57:58.778+07:00 INFO [LookupTableService] Data Adapter otx-api-domain/5a40782b84e00e7f34f46e8c [ 1b80df02] RUNNING
2018-03-06T16:57:58.783+07:00 WARN [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2018-03-06T16:57:58.783+07:00 ERROR [LookupDataAdapter] Couldn't start data adapter <tor-exit-node/5a40782b84e00e7f34f46e88/ 7973b9c4>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: TOR service is disabled, not starting TOR exit addresses adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.tor.TorExitNodeDataAdapter.doStart(TorExitNodeDataAdapter.java:73) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2018-03-06T16:57:58.792+07:00 ERROR [LookupDataAdapter] Couldn't start data adapter <spamhaus-drop/5a40782b84e00e7f34f46e8d/ 731a4770>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Spamhaus service is disabled, not starting (E)DROP adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.spamhaus.SpamhausEDROPDataAdapter.doStart(SpamhausEDROPDataAdapter.java:68) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2018-03-06T16:57:58.799+07:00 INFO [LookupTableService] Data Adapter otx-api-ip/5a40782b84e00e7f34f46e8b [ 1efad89] STARTING
2018-03-06T16:57:58.799+07:00 ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-ip/5a40782b84e00e7f34f46e89/ 68b652d9>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2018-03-06T16:57:58.804+07:00 INFO [LookupTableService] Data Adapter tor-exit-node/5a40782b84e00e7f34f46e88 [ 7973b9c4] STARTING
2018-03-06T16:57:58.804+07:00 INFO [LookupTableService] Data Adapter otx-api-ip/5a40782b84e00e7f34f46e8b [ 1efad89] RUNNING
2018-03-06T16:57:58.805+07:00 INFO [LookupTableService] Data Adapter tor-exit-node/5a40782b84e00e7f34f46e88 [ 7973b9c4] RUNNING
2018-03-06T16:57:58.805+07:00 INFO [LookupTableService] Data Adapter spamhaus-drop/5a40782b84e00e7f34f46e8d [ 731a4770] STARTING
2018-03-06T16:57:58.806+07:00 INFO [LookupTableService] Data Adapter abuse-ch-ransomware-ip/5a40782b84e00e7f34f46e89 [ 68b652d9] STARTING
2018-03-06T16:57:58.807+07:00 INFO [LookupTableService] Data Adapter spamhaus-drop/5a40782b84e00e7f34f46e8d [ 731a4770] RUNNING
2018-03-06T16:57:58.812+07:00 INFO [LookupTableService] Data Adapter abuse-ch-ransomware-ip/5a40782b84e00e7f34f46e89 [ 68b652d9] RUNNING
2018-03-06T16:57:58.812+07:00 INFO [LookupTableService] Data Adapter test/5992cfe567ec180b2e1a8145 [ 61f57df1] RUNNING
2018-03-06T16:57:58.813+07:00 INFO [LookupDataAdapterRefreshService] Adding job for <test/5992cfe567ec180b2e1a8145/ 61f57df1> [interval=60000ms]
2018-03-06T16:57:58.835+07:00 INFO [LookupTableService] Cache test/5992d24ceb22840b7ba729fa [ 4ad261c2] STARTING
2018-03-06T16:57:58.845+07:00 INFO [LookupTableService] Cache threat-intel-uncached-adapters/5a40782b84e00e7f34f46e83 [ 9c0dca0] STARTING
2018-03-06T16:57:58.845+07:00 INFO [LookupTableService] Cache test/5992d24ceb22840b7ba729fa [ 4ad261c2] RUNNING
2018-03-06T16:57:58.845+07:00 INFO [LookupTableService] Cache otx-api-ip-cache/5a40782b84e00e7f34f46e85 [ 686113f0] STARTING
2018-03-06T16:57:58.855+07:00 INFO [LookupTableService] Cache threat-intel-uncached-adapters/5a40782b84e00e7f34f46e83 [ 9c0dca0] RUNNING
2018-03-06T16:57:58.857+07:00 INFO [LookupTableService] Cache spamhaus-e-drop-cache/5a40782b84e00e7f34f46e7f [ 634178c5] STARTING
2018-03-06T16:57:58.858+07:00 INFO [LookupTableService] Cache otx-api-ip-cache/5a40782b84e00e7f34f46e85 [ 686113f0] RUNNING
2018-03-06T16:57:58.877+07:00 INFO [LookupTableService] Cache whois-cache/5a40782b84e00e7f34f46e80 [ 44aac0c7] STARTING
2018-03-06T16:57:58.867+07:00 INFO [LookupTableService] Cache otx-api-domain-cache/5a40782b84e00e7f34f46e82 [ 5f67868e] STARTING
2018-03-06T16:57:58.879+07:00 INFO [LookupTableService] Cache otx-api-domain-cache/5a40782b84e00e7f34f46e82 [ 5f67868e] RUNNING
2018-03-06T16:57:58.881+07:00 INFO [LookupTableService] Cache whois-cache/5a40782b84e00e7f34f46e80 [ 44aac0c7] RUNNING
2018-03-06T16:57:58.881+07:00 INFO [LookupTableService] Cache spamhaus-e-drop-cache/5a40782b84e00e7f34f46e7f [ 634178c5] RUNNING
2018-03-06T16:57:58.905+07:00 INFO [LookupTableService] Starting lookup table hostname/5992d28467ec180b2e1a83e7 [ 3d8b237] using cache test/5992d24ceb22840b7ba729fa [ 4ad261c2], data adapter test/5992cfe567ec180b2e1a8145 [ 61f57df1]
2018-03-06T16:57:58.906+07:00 INFO [LookupTableService] Starting lookup table otx-api-domain/5a40782b84e00e7f34f46e8f [ 6e936627] using cache otx-api-domain-cache/5a40782b84e00e7f34f46e82 [ 5f67868e], data adapter otx-api-domain/5a40782b84e00e7f34f46e8c [ 1b80df02]
2018-03-06T16:57:58.906+07:00 INFO [LookupTableService] Starting lookup table tor-exit-node-list/5a40782b84e00e7f34f46e90 [ 70d758e0] using cache threat-intel-uncached-adapters/5a40782b84e00e7f34f46e83 [ 9c0dca0], data adapter tor-exit-node/5a40782b84e00e7f34f46e88 [ 7973b9c4]
2018-03-06T16:57:58.907+07:00 INFO [LookupTableService] Starting lookup table otx-api-ip/5a40782b84e00e7f34f46e91 [ 50ab82fd] using cache otx-api-ip-cache/5a40782b84e00e7f34f46e85 [ 686113f0], data adapter otx-api-ip/5a40782b84e00e7f34f46e8b [ 1efad89]
2018-03-06T16:57:58.907+07:00 INFO [LookupTableService] Starting lookup table abuse-ch-ransomware-domains/5a40782b84e00e7f34f46e92 [ 1624cbd5] using cache threat-intel-uncached-adapters/5a40782b84e00e7f34f46e83 [ 9c0dca0], data adapter abuse-ch-ransomware-domains/5a40782b84e00e7f34f46e87 [ 74df32be]
2018-03-06T16:57:58.907+07:00 INFO [LookupTableService] Starting lookup table whois/5a40782b84e00e7f34f46e93 [ 63f32568] using cache whois-cache/5a40782b84e00e7f34f46e80 [ 44aac0c7], data adapter whois/5a40782b84e00e7f34f46e8a [ 29abb3d8]
2018-03-06T16:57:58.908+07:00 INFO [LookupTableService] Starting lookup table spamhaus-drop/5a40782b84e00e7f34f46e94 [ 292971ae] using cache spamhaus-e-drop-cache/5a40782b84e00e7f34f46e7f [ 634178c5], data adapter spamhaus-drop/5a40782b84e00e7f34f46e8d [ 731a4770]
2018-03-06T16:57:58.908+07:00 INFO [LookupTableService] Starting lookup table abuse-ch-ransomware-ip/5a40782b84e00e7f34f46e95 [ 2599a21c] using cache threat-intel-uncached-adapters/5a40782b84e00e7f34f46e83 [ 9c0dca0], data adapter abuse-ch-ransomware-ip/5a40782b84e00e7f34f46e89 [ 68b652d9]
2018-03-06T16:57:59.504+07:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2018-03-06T16:58:15.165+07:00 INFO [NetworkListener] Started listener bound to [graylog2.abc.com:9000]
2018-03-06T16:58:15.167+07:00 INFO [HttpServer] [HttpServer] Started.
2018-03-06T16:58:15.167+07:00 INFO [JerseyService] Started REST API at <http://graylog2.abc.com:9000/api/>
2018-03-06T16:58:15.167+07:00 INFO [JerseyService] Started Web Interface at <http://graylog2.abc.com:9000/>
2018-03-06T16:58:15.168+07:00 INFO [ServiceManagerListener] Services are healthy
2018-03-06T16:58:15.169+07:00 INFO [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=12, BufferSynchronizerService [RUNNING]=46, OutputSetupService [RUNNING]=113, KafkaJournal [RUNNING]=148, PeriodicalsService [RUNNING]=162, ConfigurationEtagService [RUNNING]=206, JournalReader [RUNNING]=219, StreamCacheService [RUNNING]=249, LookupTableService [RUNNING]=482, JerseyService [RUNNING]=16751}
2018-03-06T16:58:15.169+07:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2018-03-06T16:58:15.179+07:00 INFO [ServerBootstrap] Graylog server up and running.
2018-03-06T16:58:15.216+07:00 INFO [InputStateListener] Input [Beats/59d1fce884e00e4ce3981255] is now STARTING
2018-03-06T16:58:15.221+07:00 INFO [InputStateListener] Input [Beats/59d216a784e00e51a71af7f6] is now STARTING
2018-03-06T16:58:15.223+07:00 INFO [InputStateListener] Input [GELF UDP/58e703c7ecd007561209ea53] is now STARTING
2018-03-06T16:58:15.335+07:00 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=WAN, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=e0fab262-c283-4352-9b18-5b0ec671aaec} should be 2000000 but is 212992.
2018-03-06T16:58:15.354+07:00 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=VPS-DO, type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2018-03-06T16:58:15.355+07:00 INFO [InputStateListener] Input [GELF UDP/58e703c7ecd007561209ea53] is now RUNNING
2018-03-06T16:58:15.359+07:00 INFO [InputStateListener] Input [Beats/59d1fce884e00e4ce3981255] is now RUNNING
2018-03-06T16:58:15.369+07:00 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=K8s-pods, type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2018-03-06T16:58:15.373+07:00 INFO [InputStateListener] Input [Beats/59d216a784e00e51a71af7f6] is now RUNNING
Here is the last:
2018-03-06T16:58:38.419+07:00 INFO [connection] Opened connection [connectionId{localValue:9, serverValue:1103460}] to xxx:27017
2018-03-06T16:58:38.422+07:00 INFO [connection] Opened connection [connectionId{localValue:10, serverValue:1103461}] to xxx:27017
2018-03-06T16:58:38.432+07:00 INFO [connection] Opened connection [connectionId{localValue:8, serverValue:1103459}] to xxx:27017
2018-03-06T17:03:19.181+07:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=988e4f91-2125-11e8-a965-00163e000000, journalOffset=6141511245, codec=gelf, payloadSize=196, timestamp=2018-03-06T10:03:19.177Z, remoteAddress=/103.253.145.218:59806} on input <58e703c7ecd007561209ea53>.
2018-03-06T17:03:19.184+07:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=988e4f91-2125-11e8-a965-00163e000000, journalOffset=6141511245, codec=gelf, payloadSize=196, timestamp=2018-03-06T10:03:19.177Z, remoteAddress=/103.253.145.218:59806}
java.lang.IllegalArgumentException: GELF message <988e4f91-2125-11e8-a965-00163e000000> (received from <103.253.145.218:59806>) has empty mandatory "short_message" field.
at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:252) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:134) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2018-03-06T17:04:29.585+07:00 ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: java.io.IOException: Connection is closed
at org.glassfish.grizzly.nio.NIOConnection.assertOpen(NIOConnection.java:445) ~[graylog.jar:?]
at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:677) ~[graylog.jar:?]
at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.write(NIOOutputStreamImpl.java:83) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1354) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1301) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:457) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:49) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951) ~[graylog.jar:?]
at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
... 20 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_144]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_144]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_144]
at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_144]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_144]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeCompositeBuffer(TCPNIOUtils.java:86) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:129) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:260) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:169) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:71) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleWrite(TCPNIOTransportFilter.java:126) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.TransportFilter.handleWrite(TransportFilter.java:191) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) ~[graylog.jar:?]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:890) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:858) ~[graylog.jar:?]
at org.glassfish.grizzly.http.io.OutputBuffer.flushBuffer(OutputBuffer.java:1059) ~[graylog.jar:?]
at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:709) ~[graylog.jar:?]
at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.write(NIOOutputStreamImpl.java:83) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeNumber(UTF8JsonGenerator.java:853) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.NumberSerializers$IntegerSerializer.serialize(NumberSerializers.java:137) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951) ~[graylog.jar:?]
at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
... 20 more
Or you’ll use a pastebin service like https://0bin.net/ or https://gist.github.com/ next time.
The time range maybe not is the cause. After the last restarting graylog server, I haven’t see that error logs, but I can’t still search anything.
Try rotating the active write index (System/Indices/Index Set/Maintenance).
Oh It work. Can u tell me why?
Thanks.
Probably something with your Elasticsearch cluster.
You didn’t provide any information on your Elasticsearch nodes, so you’ll have to find out yourself, why rotating the index fixed your issue.
If you do not mind, Can I describe my scenario?
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.