Graylog can not search message


(Kieulam141) #1

Hi
I’m getting error with graylog cluster.My graylog and elasticsesarch work correctly. input and output message on graylog sill work. But I can search anything in graylog search.
I got this error when tail graylog log:
" Caused by: java.io.IOException: Broken pipe"
How can I fix this?
Thanks


(Kieulam141) #2

More logs “ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed”


(Jochen) #3

Please post the complete logs of your Graylog and Elasticsearch nodes.


(Kieulam141) #4

Here is the full log:

2018-03-06T17:06:57.789+07:00 ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) ~[graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: java.io.IOException: Connection is closed
	at org.glassfish.grizzly.nio.NIOConnection.assertOpen(NIOConnection.java:445) ~[graylog.jar:?]
	at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:677) ~[graylog.jar:?]
	at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.write(NIOOutputStreamImpl.java:83) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1354) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1301) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:457) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:49) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951) ~[graylog.jar:?]
	at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
	at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
	... 20 more
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_144]
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_144]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_144]
	at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_144]
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_144]
	at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeCompositeBuffer(TCPNIOUtils.java:86) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:129) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:260) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:169) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:71) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleWrite(TCPNIOTransportFilter.java:126) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.TransportFilter.handleWrite(TransportFilter.java:191) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) ~[graylog.jar:?]
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:890) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:858) ~[graylog.jar:?]
	at org.glassfish.grizzly.http.io.OutputBuffer.flushBuffer(OutputBuffer.java:1059) ~[graylog.jar:?]
	at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:709) ~[graylog.jar:?]
	at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.write(NIOOutputStreamImpl.java:83) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1354) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1301) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:457) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:49) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951) ~[graylog.jar:?]
	at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
	at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
	... 20 more

(Jochen) #5

Is this really the complete content of the logs produces by your Graylog nodes?
It looks like a single message to me.


(Kieulam141) #6

Yep, there are all error log from graylog master node.


(Jochen) #7

Please post the complete logs, not only the messages on the ERROR log level.


(Kieulam141) #8

Body is limited to 32000 characters; you entered 40592.
Can I split them in 2 times.


(Kieulam141) #9
2018-03-06T16:57:47.117+07:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 2.4.3 [org.graylog.aws.plugin.AWSPlugin]
2018-03-06T16:57:47.123+07:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.4.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-03-06T16:57:47.124+07:00 INFO  [CmdLineTool] Loaded plugin: CEF Input 2.4.3 [org.graylog.plugins.cef.CEFInputPlugin]
2018-03-06T16:57:47.126+07:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.4.3 [org.graylog.plugins.collector.CollectorPlugin]
2018-03-06T16:57:47.128+07:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.4.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2018-03-06T16:57:47.130+07:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.4.3 [org.graylog.plugins.map.MapWidgetPlugin]
2018-03-06T16:57:47.131+07:00 INFO  [CmdLineTool] Loaded plugin: NetFlow Plugin 2.4.3 [org.graylog.plugins.netflow.NetFlowPlugin]
2018-03-06T16:57:47.148+07:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.4.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-03-06T16:57:47.150+07:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.4.3 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2018-03-06T16:57:47.151+07:00 INFO  [CmdLineTool] Loaded plugin: Slack 3.0.0 [org.graylog2.plugins.slack.callback.SlackAlarmCallback]
2018-03-06T16:57:48.077+07:00 INFO  [CmdLineTool] Running with JVM arguments: -Djava.net.preferIPv4Stack=true -Xms4g -Xmx4g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2018-03-06T16:57:48.399+07:00 INFO  [Version] HV000001: Hibernate Validator 5.1.3.Final
2018-03-06T16:57:51.707+07:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-03-06T16:57:51.736+07:00 INFO  [NodeId] Node ID: b90ec773-0032-4bae-bc40-37e969c6018c
2018-03-06T16:57:51.986+07:00 INFO  [LogManager] Loading logs.
2018-03-06T16:57:52.112+07:00 INFO  [LogManager] Logs loading complete.
2018-03-06T16:57:52.112+07:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2018-03-06T16:57:52.142+07:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <131072> and wait strategy <BlockingWaitStrategy>, running 4 parallel message handlers.
2018-03-06T16:57:52.175+07:00 INFO  [cluster] Cluster created with settings {hosts=[xxxxxx:27017, xxx:27017, xxx:27017], mode=MULTIPLE, requiredClusterType=REPLICA_SET, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000, requiredReplicaSetName='sector5-rs0'}
2018-03-06T16:57:52.177+07:00 INFO  [cluster] Adding discovered server xxx:27017 to client view of cluster
2018-03-06T16:57:52.219+07:00 INFO  [cluster] Adding discovered server xxx:27017 to client view of cluster
2018-03-06T16:57:52.250+07:00 INFO  [cluster] Adding discovered server xxx:27017 to client view of cluster
2018-03-06T16:57:52.317+07:00 INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=xxx:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=xxx:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=xxx:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2018-03-06T16:57:52.985+07:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:463554}] to xxx:27017
2018-03-06T16:57:52.998+07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=xxx:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 4]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=3674632, setName='sector5-rs0', canonicalAddress=xxx:27017, hosts=[xxx:27017, xxx:27017], passives=[xxx:27017], arbiters=[], primary='xxx:27017', tagSet=TagSet{[]}, electionId=null, setVersion=34, lastWriteDate=null, lastUpdateTimeNanos=4053199318463233}
2018-03-06T16:57:53.019+07:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:288147}] to xxx:27017
2018-03-06T16:57:53.021+07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=xxx:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 4]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=966202, setName='sector5-rs0', canonicalAddress=xxx:27017, hosts=[xxx:27017, xxx:27017], passives=[xxx:27017], arbiters=[], primary='xxx:27017', tagSet=TagSet{[]}, electionId=null, setVersion=34, lastWriteDate=null, lastUpdateTimeNanos=4053199342579108}
2018-03-06T16:57:53.022+07:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:1103452}] to xxx:27017
2018-03-06T16:57:53.024+07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=xxx:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 4]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=1121058, setName='sector5-rs0', canonicalAddress=xxx:27017, hosts=[xxx:27017, xxx:27017], passives=[xxx:27017], arbiters=[], primary='xxx:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000012, setVersion=34, lastWriteDate=null, lastUpdateTimeNanos=4053199345208760}
2018-03-06T16:57:53.024+07:00 INFO  [cluster] Setting max election id to 7fffffff0000000000000012 from replica set primary xxx:27017
2018-03-06T16:57:53.024+07:00 INFO  [cluster] Setting max set version to 34 from replica set primary xxx:27017
2018-03-06T16:57:53.025+07:00 INFO  [cluster] Discovered replica set primary xxx:27017
2018-03-06T16:57:53.165+07:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:1103453}] to xxx:27017
2018-03-06T16:57:53.853+07:00 INFO  [AbstractJestClient] Setting server pool to a list of 3 servers: [http://elas1:9200,http://elas2:9200,http://elas3:9200]
2018-03-06T16:57:53.856+07:00 INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2018-03-06T16:57:54.011+07:00 INFO  [JestClientFactory] Using custom ObjectMapper instance
2018-03-06T16:57:54.012+07:00 INFO  [JestClientFactory] Node Discovery enabled...
2018-03-06T16:57:54.049+07:00 INFO  [JestClientFactory] Idle connection reaping disabled...
2018-03-06T16:57:54.154+07:00 INFO  [AbstractJestClient] Setting server pool to a list of 3 servers: [http://10.130.57.108:9200,http://10.130.42.18:9200,http://10.130.46.220:9200]
2018-03-06T16:57:54.487+07:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <131072> and wait strategy <BlockingWaitStrategy>.
2018-03-06T16:57:57.604+07:00 INFO  [RulesEngineProvider] No static rules file loaded.
2018-03-06T16:57:57.906+07:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <131072> and wait strategy <BlockingWaitStrategy>.
2018-03-06T16:57:57.993+07:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:1103456}] to xxx:27017
2018-03-06T16:57:58.381+07:00 INFO  [ServerBootstrap] Graylog server 2.4.3+2c41897 starting up
2018-03-06T16:57:58.382+07:00 INFO  [ServerBootstrap] JRE: Oracle Corporation 1.8.0_144 on Linux 4.4.0-109-generic
2018-03-06T16:57:58.382+07:00 INFO  [ServerBootstrap] Deployment: deb
2018-03-06T16:57:58.382+07:00 INFO  [ServerBootstrap] OS: Ubuntu 16.04.3 LTS (xenial)
2018-03-06T16:57:58.382+07:00 INFO  [ServerBootstrap] Arch: amd64
2018-03-06T16:57:58.385+07:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2018-03-06T16:57:58.418+07:00 INFO  [PeriodicalsService] Starting 25 periodicals ...
2018-03-06T16:57:58.429+07:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2018-03-06T16:57:58.443+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.464+07:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2018-03-06T16:57:58.465+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.465+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.466+07:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2018-03-06T16:57:58.474+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.474+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRetentionThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.474+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRotationThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.474+07:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2018-03-06T16:57:58.475+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.VersionCheckThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.553+07:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2018-03-06T16:57:58.563+07:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2018-03-06T16:57:58.563+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.566+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.567+07:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.568+07:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2018-03-06T16:57:58.569+07:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2018-03-06T16:57:58.569+07:00 INFO  [PeriodicalsService] Not starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.569+07:00 INFO  [PeriodicalsService] Not starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical. Only started on Graylog master nodes.
2018-03-06T16:57:58.644+07:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:1103457}] to xxx:27017
2018-03-06T16:57:58.680+07:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:1103458}] to xxx:27017
2018-03-06T16:57:58.699+07:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-domains/5a40782b84e00e7f34f46e87 [ 74df32be] STARTING
2018-03-06T16:57:58.699+07:00 ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-domains/5a40782b84e00e7f34f46e87/ 74df32be>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
	at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
	at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
	at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2018-03-06T16:57:58.712+07:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-domains/5a40782b84e00e7f34f46e87 [ 74df32be] RUNNING
2018-03-06T16:57:58.723+07:00 INFO  [LookupTableService] Data Adapter otx-api-domain/5a40782b84e00e7f34f46e8c [ 1b80df02] STARTING
2018-03-06T16:57:58.724+07:00 WARN  [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2018-03-06T16:57:58.749+07:00 INFO  [LookupTableService] Data Adapter whois/5a40782b84e00e7f34f46e8a [ 29abb3d8] STARTING
2018-03-06T16:57:58.777+07:00 INFO  [LookupTableService] Data Adapter whois/5a40782b84e00e7f34f46e8a [ 29abb3d8] RUNNING
2018-03-06T16:57:58.777+07:00 INFO  [LookupTableService] Data Adapter test/5992cfe567ec180b2e1a8145 [ 61f57df1] STARTING
2018-03-06T16:57:58.778+07:00 INFO  [LookupTableService] Data Adapter otx-api-domain/5a40782b84e00e7f34f46e8c [ 1b80df02] RUNNING
2018-03-06T16:57:58.783+07:00 WARN  [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2018-03-06T16:57:58.783+07:00 ERROR [LookupDataAdapter] Couldn't start data adapter <tor-exit-node/5a40782b84e00e7f34f46e88/ 7973b9c4>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: TOR service is disabled, not starting TOR exit addresses adapter. To enable it please go to System / Configurations.
	at org.graylog.plugins.threatintel.adapters.tor.TorExitNodeDataAdapter.doStart(TorExitNodeDataAdapter.java:73) ~[?:?]
	at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
	at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2018-03-06T16:57:58.792+07:00 ERROR [LookupDataAdapter] Couldn't start data adapter <spamhaus-drop/5a40782b84e00e7f34f46e8d/ 731a4770>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Spamhaus service is disabled, not starting (E)DROP adapter. To enable it please go to System / Configurations.
	at org.graylog.plugins.threatintel.adapters.spamhaus.SpamhausEDROPDataAdapter.doStart(SpamhausEDROPDataAdapter.java:68) ~[?:?]
	at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
	at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2018-03-06T16:57:58.799+07:00 INFO  [LookupTableService] Data Adapter otx-api-ip/5a40782b84e00e7f34f46e8b [ 1efad89] STARTING
2018-03-06T16:57:58.799+07:00 ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-ip/5a40782b84e00e7f34f46e89/ 68b652d9>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
	at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
	at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
	at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2018-03-06T16:57:58.804+07:00 INFO  [LookupTableService] Data Adapter tor-exit-node/5a40782b84e00e7f34f46e88 [ 7973b9c4] STARTING
2018-03-06T16:57:58.804+07:00 INFO  [LookupTableService] Data Adapter otx-api-ip/5a40782b84e00e7f34f46e8b [ 1efad89] RUNNING
2018-03-06T16:57:58.805+07:00 INFO  [LookupTableService] Data Adapter tor-exit-node/5a40782b84e00e7f34f46e88 [ 7973b9c4] RUNNING
2018-03-06T16:57:58.805+07:00 INFO  [LookupTableService] Data Adapter spamhaus-drop/5a40782b84e00e7f34f46e8d [ 731a4770] STARTING
2018-03-06T16:57:58.806+07:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-ip/5a40782b84e00e7f34f46e89 [ 68b652d9] STARTING
2018-03-06T16:57:58.807+07:00 INFO  [LookupTableService] Data Adapter spamhaus-drop/5a40782b84e00e7f34f46e8d [ 731a4770] RUNNING
2018-03-06T16:57:58.812+07:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-ip/5a40782b84e00e7f34f46e89 [ 68b652d9] RUNNING
2018-03-06T16:57:58.812+07:00 INFO  [LookupTableService] Data Adapter test/5992cfe567ec180b2e1a8145 [ 61f57df1] RUNNING
2018-03-06T16:57:58.813+07:00 INFO  [LookupDataAdapterRefreshService] Adding job for <test/5992cfe567ec180b2e1a8145/ 61f57df1> [interval=60000ms]
2018-03-06T16:57:58.835+07:00 INFO  [LookupTableService] Cache test/5992d24ceb22840b7ba729fa [ 4ad261c2] STARTING
2018-03-06T16:57:58.845+07:00 INFO  [LookupTableService] Cache threat-intel-uncached-adapters/5a40782b84e00e7f34f46e83 [ 9c0dca0] STARTING
2018-03-06T16:57:58.845+07:00 INFO  [LookupTableService] Cache test/5992d24ceb22840b7ba729fa [ 4ad261c2] RUNNING
2018-03-06T16:57:58.845+07:00 INFO  [LookupTableService] Cache otx-api-ip-cache/5a40782b84e00e7f34f46e85 [ 686113f0] STARTING
2018-03-06T16:57:58.855+07:00 INFO  [LookupTableService] Cache threat-intel-uncached-adapters/5a40782b84e00e7f34f46e83 [ 9c0dca0] RUNNING
2018-03-06T16:57:58.857+07:00 INFO  [LookupTableService] Cache spamhaus-e-drop-cache/5a40782b84e00e7f34f46e7f [ 634178c5] STARTING
2018-03-06T16:57:58.858+07:00 INFO  [LookupTableService] Cache otx-api-ip-cache/5a40782b84e00e7f34f46e85 [ 686113f0] RUNNING
2018-03-06T16:57:58.877+07:00 INFO  [LookupTableService] Cache whois-cache/5a40782b84e00e7f34f46e80 [ 44aac0c7] STARTING
2018-03-06T16:57:58.867+07:00 INFO  [LookupTableService] Cache otx-api-domain-cache/5a40782b84e00e7f34f46e82 [ 5f67868e] STARTING
2018-03-06T16:57:58.879+07:00 INFO  [LookupTableService] Cache otx-api-domain-cache/5a40782b84e00e7f34f46e82 [ 5f67868e] RUNNING
2018-03-06T16:57:58.881+07:00 INFO  [LookupTableService] Cache whois-cache/5a40782b84e00e7f34f46e80 [ 44aac0c7] RUNNING
2018-03-06T16:57:58.881+07:00 INFO  [LookupTableService] Cache spamhaus-e-drop-cache/5a40782b84e00e7f34f46e7f [ 634178c5] RUNNING
2018-03-06T16:57:58.905+07:00 INFO  [LookupTableService] Starting lookup table hostname/5992d28467ec180b2e1a83e7 [ 3d8b237] using cache test/5992d24ceb22840b7ba729fa [ 4ad261c2], data adapter test/5992cfe567ec180b2e1a8145 [ 61f57df1]
2018-03-06T16:57:58.906+07:00 INFO  [LookupTableService] Starting lookup table otx-api-domain/5a40782b84e00e7f34f46e8f [ 6e936627] using cache otx-api-domain-cache/5a40782b84e00e7f34f46e82 [ 5f67868e], data adapter otx-api-domain/5a40782b84e00e7f34f46e8c [ 1b80df02]
2018-03-06T16:57:58.906+07:00 INFO  [LookupTableService] Starting lookup table tor-exit-node-list/5a40782b84e00e7f34f46e90 [ 70d758e0] using cache threat-intel-uncached-adapters/5a40782b84e00e7f34f46e83 [ 9c0dca0], data adapter tor-exit-node/5a40782b84e00e7f34f46e88 [ 7973b9c4]
2018-03-06T16:57:58.907+07:00 INFO  [LookupTableService] Starting lookup table otx-api-ip/5a40782b84e00e7f34f46e91 [ 50ab82fd] using cache otx-api-ip-cache/5a40782b84e00e7f34f46e85 [ 686113f0], data adapter otx-api-ip/5a40782b84e00e7f34f46e8b [ 1efad89]
2018-03-06T16:57:58.907+07:00 INFO  [LookupTableService] Starting lookup table abuse-ch-ransomware-domains/5a40782b84e00e7f34f46e92 [ 1624cbd5] using cache threat-intel-uncached-adapters/5a40782b84e00e7f34f46e83 [ 9c0dca0], data adapter abuse-ch-ransomware-domains/5a40782b84e00e7f34f46e87 [ 74df32be]
2018-03-06T16:57:58.907+07:00 INFO  [LookupTableService] Starting lookup table whois/5a40782b84e00e7f34f46e93 [ 63f32568] using cache whois-cache/5a40782b84e00e7f34f46e80 [ 44aac0c7], data adapter whois/5a40782b84e00e7f34f46e8a [ 29abb3d8]
2018-03-06T16:57:58.908+07:00 INFO  [LookupTableService] Starting lookup table spamhaus-drop/5a40782b84e00e7f34f46e94 [ 292971ae] using cache spamhaus-e-drop-cache/5a40782b84e00e7f34f46e7f [ 634178c5], data adapter spamhaus-drop/5a40782b84e00e7f34f46e8d [ 731a4770]
2018-03-06T16:57:58.908+07:00 INFO  [LookupTableService] Starting lookup table abuse-ch-ransomware-ip/5a40782b84e00e7f34f46e95 [ 2599a21c] using cache threat-intel-uncached-adapters/5a40782b84e00e7f34f46e83 [ 9c0dca0], data adapter abuse-ch-ransomware-ip/5a40782b84e00e7f34f46e89 [ 68b652d9]
2018-03-06T16:57:59.504+07:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2018-03-06T16:58:15.165+07:00 INFO  [NetworkListener] Started listener bound to [graylog2.abc.com:9000]
2018-03-06T16:58:15.167+07:00 INFO  [HttpServer] [HttpServer] Started.
2018-03-06T16:58:15.167+07:00 INFO  [JerseyService] Started REST API at <http://graylog2.abc.com:9000/api/>
2018-03-06T16:58:15.167+07:00 INFO  [JerseyService] Started Web Interface at <http://graylog2.abc.com:9000/>
2018-03-06T16:58:15.168+07:00 INFO  [ServiceManagerListener] Services are healthy
2018-03-06T16:58:15.169+07:00 INFO  [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=12, BufferSynchronizerService [RUNNING]=46, OutputSetupService [RUNNING]=113, KafkaJournal [RUNNING]=148, PeriodicalsService [RUNNING]=162, ConfigurationEtagService [RUNNING]=206, JournalReader [RUNNING]=219, StreamCacheService [RUNNING]=249, LookupTableService [RUNNING]=482, JerseyService [RUNNING]=16751}
2018-03-06T16:58:15.169+07:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2018-03-06T16:58:15.179+07:00 INFO  [ServerBootstrap] Graylog server up and running.
2018-03-06T16:58:15.216+07:00 INFO  [InputStateListener] Input [Beats/59d1fce884e00e4ce3981255] is now STARTING
2018-03-06T16:58:15.221+07:00 INFO  [InputStateListener] Input [Beats/59d216a784e00e51a71af7f6] is now STARTING
2018-03-06T16:58:15.223+07:00 INFO  [InputStateListener] Input [GELF UDP/58e703c7ecd007561209ea53] is now STARTING
2018-03-06T16:58:15.335+07:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=WAN, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=e0fab262-c283-4352-9b18-5b0ec671aaec} should be 2000000 but is 212992.
2018-03-06T16:58:15.354+07:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=VPS-DO, type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2018-03-06T16:58:15.355+07:00 INFO  [InputStateListener] Input [GELF UDP/58e703c7ecd007561209ea53] is now RUNNING
2018-03-06T16:58:15.359+07:00 INFO  [InputStateListener] Input [Beats/59d1fce884e00e4ce3981255] is now RUNNING
2018-03-06T16:58:15.369+07:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=K8s-pods, type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2018-03-06T16:58:15.373+07:00 INFO  [InputStateListener] Input [Beats/59d216a784e00e51a71af7f6] is now RUNNING

(Kieulam141) #10

Here is the last:

2018-03-06T16:58:38.419+07:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:1103460}] to xxx:27017
2018-03-06T16:58:38.422+07:00 INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:1103461}] to xxx:27017
2018-03-06T16:58:38.432+07:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:1103459}] to xxx:27017
2018-03-06T17:03:19.181+07:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=988e4f91-2125-11e8-a965-00163e000000, journalOffset=6141511245, codec=gelf, payloadSize=196, timestamp=2018-03-06T10:03:19.177Z, remoteAddress=/103.253.145.218:59806} on input <58e703c7ecd007561209ea53>.
2018-03-06T17:03:19.184+07:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=988e4f91-2125-11e8-a965-00163e000000, journalOffset=6141511245, codec=gelf, payloadSize=196, timestamp=2018-03-06T10:03:19.177Z, remoteAddress=/103.253.145.218:59806}
java.lang.IllegalArgumentException: GELF message <988e4f91-2125-11e8-a965-00163e000000> (received from <103.253.145.218:59806>) has empty mandatory "short_message" field.
	at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:252) ~[graylog.jar:?]
	at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:134) ~[graylog.jar:?]
	at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
	at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2018-03-06T17:04:29.585+07:00 ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) ~[graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: java.io.IOException: Connection is closed
	at org.glassfish.grizzly.nio.NIOConnection.assertOpen(NIOConnection.java:445) ~[graylog.jar:?]
	at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:677) ~[graylog.jar:?]
	at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.write(NIOOutputStreamImpl.java:83) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1354) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1301) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:457) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:49) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951) ~[graylog.jar:?]
	at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
	at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
	... 20 more
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_144]
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_144]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_144]
	at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_144]
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_144]
	at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeCompositeBuffer(TCPNIOUtils.java:86) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:129) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:260) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:169) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:71) ~[graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleWrite(TCPNIOTransportFilter.java:126) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.TransportFilter.handleWrite(TransportFilter.java:191) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) ~[graylog.jar:?]
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:890) ~[graylog.jar:?]
	at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:858) ~[graylog.jar:?]
	at org.glassfish.grizzly.http.io.OutputBuffer.flushBuffer(OutputBuffer.java:1059) ~[graylog.jar:?]
	at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:709) ~[graylog.jar:?]
	at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.write(NIOOutputStreamImpl.java:83) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039) ~[graylog.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeNumber(UTF8JsonGenerator.java:853) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.NumberSerializers$IntegerSerializer.serialize(NumberSerializers.java:137) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951) ~[graylog.jar:?]
	at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
	at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
	... 20 more

(Jochen) #11

Or you’ll use a pastebin service like https://0bin.net/ or https://gist.github.com/ next time.


(Kieulam141) #12

I can’t understand this. Messages are increasing but can not see in chart.


(Kieulam141) #13

The time range maybe not is the cause. After the last restarting graylog server, I haven’t see that error logs, but I can’t still search anything.


(Jochen) #14

Try rotating the active write index (System/Indices/Index Set/Maintenance).


(Kieulam141) #15

Oh It work. Can u tell me why?
Thanks.


(Jochen) #16

Probably something with your Elasticsearch cluster.

You didn’t provide any information on your Elasticsearch nodes, so you’ll have to find out yourself, why rotating the index fixed your issue.


(Kieulam141) #17

If you do not mind, Can I describe my scenario?


(system) #18

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.