Hi,
I appreciate your priceless help and support in the questions we post. Currently, the configuration get the logs from filebeat and send to elasticsearch. In UI graylog I am receiving logs from filebeat, but not all of them. For instance, I cannot received the last log in tomcat container which is from Monday April 11th:
2019-03-11 06:22:48 [Thread-4 ] DEBUG: ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB
On Filebeat the log is processed:
2019-03-14T16:18:50.377-0700 DEBUG [publish] pipeline/processor.go:308 Publish event: {
"@timestamp": "2019-03-14T23:18:45.376Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.6.0"
},
"host": {
"name": "tomcat",
"architecture": "x86_64",
"os": {
"codename": "Core",
"platform": "centos",
"version": "7 (Core)",
"family": "redhat",
"name": "CentOS Linux"
},
"id": "6aaed308aa5a419f880c5e45eea65414",
"containerized": true
},
"source": "/app/logs/WEB/WEB-rest-api/WEB-rest-api.log",
"log": {
"file": {
"path": "/app/logs/WEB/WEB-rest-api/WEB-rest-api.log"
}
},
"message": "2019-03-11 06:22:48 [Thread-4 ] DEBUG: ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB",
"beat": {
"name": "tomcat",
"hostname": "tomcat",
"version": "6.6.0"
},
"offset": 6771071,
"prospector": {
"type": "log"
},
"input": {
"type": "log"
},
"meta": {
"cloud": {
"instance_name": "tomcat",
"machine_type": "Standard_D8s_v3",
"region": "CanadaCentral",
"provider": "az",
"instance_id": "6452bcf4-7f5d-4fc3-9f8e-5ea57f00724b"
}
}
}
In elasticsearch I am not getting anything related to the log from filebeat:
[2019-03-15T11:45:11,884][INFO ][o.e.g.GatewayService ] [D6DChHc] recovered [3] indices into cluster_state
[2019-03-15T11:45:12,277][INFO ][o.e.c.r.a.AllocationService] [D6DChHc] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_1][0]] ...]).
[2019-03-15T11:45:14,140][INFO ][o.e.c.m.MetaDataIndexTemplateService] [D6DChHc] adding template [filebeat-6.6.0] for index patterns [filebeat-6.6.0-*]
[2019-03-15T11:45:14,265][INFO ][o.e.c.m.MetaDataCreateIndexService] [D6DChHc] [filebeat-6.6.0-2019.03.15] creating index, cause [auto(bulk api)], templates [filebeat-6.6.0], shards [3]/[1], mappings [doc]
[2019-03-15T11:45:14,650][INFO ][o.e.c.m.MetaDataMappingService] [D6DChHc] [filebeat-6.6.0-2019.03.15/QITBikbzRISYq7QkkXXpGQ] update_mapping [doc]
In graylog I am getting this while processing the logs in filebeat:
2019-03-15 11:58:44,120 INFO : org.graylog2.inputs.InputStateListener - Input [GELF UDP/5c88357e389808ada1e8c2cd] is now STARTING
2019-03-15 11:58:44,232 INFO : org.graylog2.inputs.InputStateListener - Input [GELF UDP/5c88357e389808ada1e8c2cd] is now RUNNING
2019-03-15 11:58:44,234 WARN : org.graylog2.inputs.transports.UdpTransport - receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=GELF UDP Input, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=c3201551-f5d9-41c4-8b89-20b4c843c2ca} (channel [id: 0xaf61abfd, L:/0:0:0:0:0:0:0:0%0:12201]) should be 262144 but is 425984.
2019-03-15 11:58:44,234 WARN : org.graylog2.inputs.transports.UdpTransport - receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=GELF UDP Input, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=c3201551-f5d9-41c4-8b89-20b4c843c2ca} (channel [id: 0xfae76cc0, L:/0:0:0:0:0:0:0:0%0:12201]) should be 262144 but is 425984.
2019-03-15 11:58:44,235 WARN : org.graylog2.inputs.transports.UdpTransport - receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=GELF UDP Input, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=c3201551-f5d9-41c4-8b89-20b4c843c2ca} (channel [id: 0xc9ab7125, L:/0:0:0:0:0:0:0:0%0:12201]) should be 262144 but is 425984.
2019-03-15 11:58:44,235 WARN : org.graylog2.inputs.transports.UdpTransport - receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=GELF UDP Input, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=c3201551-f5d9-41c4-8b89-20b4c843c2ca} (channel [id: 0x918ee7ff, L:/0:0:0:0:0:0:0:0%0:12201]) should be 262144 but is 425984.
I also tried to change the configuration and I send the output from filebeat to logstash instead of sending directly to elasticsearch. Applying this change, I was able to get the same log processed by filebeat:
2019-03-15T10:32:25,982][DEBUG][logstash.outputs.gelf ] Sending GELF event {:event=>{"short_message"=>["2019-03-11 06:22:48 [Thread-4 ] DEBUG: ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB", " Connection cache monitor in thread: Thread-4 shutting down for pool: WEB"], "full_message"=>"2019-03-11 06:22:48 [Thread-4 ] DEBUG: ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB, Connection cache monitor in thread: Thread-4 shutting down for pool: WEB", "host"=>"{\"name\":\"tomcat\",\"os\":{\"name\":\"CentOS Linux\",\"version\":\"7 (Core)\",\"codename\":\"Core\"}}", "_source"=>"/app/logs/WEB/WEB-rest-api/WEB-rest-api.log", "_class"=>"ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor, %{JAVACLASS}", "_tags"=>"beats_input_codec_plain_applied", "_beat_hostname"=>"tomcat", "_beat_name"=>"tomcat", "_meta_cloud"=>{}, "_log_file"=>{"path"=>"/app/logs/WEB/WEB-rest-api/WEB-rest-api.log"}, "level"=>6}}
However, I am not getting this log in Graylog. I am confused why I receive some logs, but others not. It is worth noting that in Graylog I am getting the logs with INFO level related to this class at similar time, but not getting the ones with DEBUG level.
What are your ideas about this issue? Please let me know if you need the details of any configuration.
Thanks a lot
