Graylog3 couldn't receive messages

AmrAbdelFattah · June 18, 2019, 9:15am

Hi,
i’m using
OS RHEL7,Graylog3,elasticsearch 5.6.16,filebeat 5.6.16 and all of them configured successfully,but there is no message recieved in graylog3

Note: Filebeat output all messages to elasticsearch ,also i created input for filebeat in graylog3 and it is running with no messages recieved

Graylog3 log:

2019-06-17T23:23:14.895+02:00 INFO [Periodicals] Starting [org.graylog.plugins.enterprise.search.db.SearchesCleanUpJob] periodical in [0s], polling every [28800s].
2019-06-17T23:23:15.179+02:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2019-06-17T23:23:30.795+02:00 WARN [AuditEventModelProcessor] REST endpoint does not use a registered audit type: DELETE api/plugins/org.graylog.plugins.enterprise/views/{id}/share (type: “views:view_sharing:delete”)
2019-06-17T23:23:30.796+02:00 WARN [AuditEventModelProcessor] REST endpoint does not use a registered audit type: POST api/plugins/org.graylog.plugins.enterprise/views/{id}/share (type: “views:view_sharing:create”)
2019-06-17T23:23:34.549+02:00 INFO [NetworkListener] Started listener bound to [10.208.221.18:9000]
2019-06-17T23:23:34.551+02:00 INFO [HttpServer] [HttpServer] Started.
2019-06-17T23:23:34.551+02:00 INFO [JerseyService] Started REST API at <10.208.221.18:9000>
2019-06-17T23:23:34.551+02:00 INFO [ServiceManagerListener] Services are healthy
2019-06-17T23:23:34.552+02:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2019-06-17T23:23:34.552+02:00 INFO [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=5, OutputSetupService [RUNNING]=6, BufferSynchronizerService [RUNNING]=6, GracefulShutdownService [RUNNING]=7, KafkaJournal [RUNNING]=8, EtagService [RUNNING]=28, JournalReader [RUNNING]=29, ConfigurationEtagService [RUNNING]=30, LookupTableService [RUNNING]=32, StreamCacheService [RUNNING]=65, PeriodicalsService [RUNNING]=110, JerseyService [RUNNING]=19753}
2019-06-17T23:23:34.558+02:00 INFO [ServerBootstrap] Graylog server up and running.
2019-06-17T23:23:34.559+02:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2019-06-17T23:23:34.570+02:00 INFO [InputStateListener] Input [Beats/5d07fcf4d4e15073720cd665] is now STARTING
2019-06-17T23:23:34.690+02:00 INFO [InputStateListener] Input [Beats/5d07fcf4d4e15073720cd665] is now RUNNING
2019-06-17T23:23:34.698+02:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=filebeat, type=org.graylog.plugins.beats.Beats2Input, nodeId=a254b10c-df99-4dae-be94-1845b87e387f} (channel [id: 0xaec6260a, L:/10.208.221.18:5044]) should be 1048576 but is 2097152.
2019-06-18T00:23:14.894+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2019-06-18T01:23:14.894+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2019-06-18T02:23:14.894+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2019-06-18T03:23:14.894+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2019-06-18T04:23:14.894+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2019-06-18T05:23:14.894+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2019-06-18T06:23:14.894+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2019-06-18T07:23:14.894+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2019-06-18T08:23:14.894+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2019-06-18T09:23:14.894+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.

Elasticsearch log :

[2019-06-17T22:09:50,951][INFO ][o.e.p.PluginsService [2019-06-17T22:09:50,951][INFO ][o.e.p.PluginsService [2019-06-17T22:09:50,951][INFO ][o.e.p.PluginsService [2019-06-17T22:09:50,951][INFO ][o.e.p.PluginsService [2019-06-17T22:09:50,951][INFO ][o.e.p.PluginsService [2019-06-17T22:09:50,951][INFO ][o.e.p.PluginsService [2019-06-17T22:09:50,951][INFO ][o.e.p.PluginsService [2019-06-17T22:09:50,951][INFO ][o.e.p.PluginsService [2019-06-17T22:09:50,952][INFO ][o.e.p.PluginsService [2019-06-17T22:09:50,952][INFO ][o.e.p.PluginsService [2019-06-17T22:09:50,952][INFO ][o.e.p.PluginsService [2019-06-17T22:09:50,952][INFO ][o.e.p.PluginsService [2019-06-17T22:09:52,302][INFO ][o.e.d.DiscoveryModule [2019-06-17T22:09:52,736][INFO ][o.e.n.Node [2019-06-17T22:09:52,736][INFO ][o.e.n.Node [2019-06-17T22:09:52,895][INFO ][o.e.t.TransportService [2019-06-17T22:09:52,904][INFO ][o.e.b.BootstrapChecks [2019-06-17T22:09:55,950][INFO ][o.e.c.s.ClusterService [2019-06-17T22:09:55,964][INFO ][o.e.h.n.Netty4HttpSe [2019-06-17T22:09:55,964][INFO ][o.e.n.Node [2019-06-17T22:09:55,972][INFO ][o.e.g.GatewayService [2019-06-17T22:37:34,426][INFO ][o.e.c.m.MetaDataCrea [2019-06-17T22:37:34,674][INFO ][o.e.c.m.MetaDataMappingService] [2019-06-17T22:38:45,609][INFO ][o.e.c.m.MetaDataCrea [2019-06-17T22:38:45,793][INFO ][o.e.c.m.MetaDataCrea ] [Sf1Zaf0] loaded module [aggs-matrix-stats]
] [Sf1Zaf0] loaded module [ingest-common]
] [Sf1Zaf0] loaded module [lang-expression]
] [Sf1Zaf0] loaded module [lang-groovy]
] [Sf1Zaf0] loaded module [lang-mustache]
] [Sf1Zaf0] loaded module [lang-painless]
] [Sf1Zaf0] loaded module [parent-join]
] [Sf1Zaf0] loaded module [percolator]
] [Sf1Zaf0] loaded module [reindex]
] [Sf1Zaf0] loaded module [transport-netty3]
] [Sf1Zaf0] loaded module [transport-netty4]
] [Sf1Zaf0] no plugins loaded
] [Sf1Zaf0] using discovery type [zen]
] initialized
] [Sf1Zaf0] starting …
] [Sf1Zaf0] publish_address {10.208.221.18:9300}, bound_addresses {10.208.221.18:9300}
] [Sf1Zaf0] bound or publishing to a non-loopback address, enforcing bootstrap checks
] [Sf1Zaf0] new_master {Sf1Zaf0}{Sf1Zaf0RTPS7BEb9zYRdkw}{XYabQ0qFTsGq0XUcwKEQ6g}{10.208.221.18}{10.208.221.18:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
rverTransport] [Sf1Zaf0] publish_address {10.208.221.18:9200}, bound_addresses {10.208.221.18:9200}
] [Sf1Zaf0] started
] [Sf1Zaf0] recovered [0] indices into cluster_state
teIndexService] [Sf1Zaf0] [filebeat-2019.06.17] creating index, cause [auto(bulk api)], templates [filebeat], shards [5]/[1], mappings [default]
[Sf1Zaf0] [filebeat-2019.06.17/YFG88m60QvmgWCnCgNUm9Q] create_mapping [doc]
teIndexService] [Sf1Zaf0] [graylog_0] creating index, cause [api], templates [graylog-internal], shards [4]/[0], mappings [message]
teIndexService] [Sf1Zaf0] [filebeat_0] creating index, cause [api], templates [filebeat-template], shards [4]/[0], mappings [message]

filebeat log:

2019-06-18T11:12:50+03:00 INFO Harvester started for file: /var/log/Process order sample - Copy.log
2019-06-18T11:12:59+03:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.es.call_count.PublishEvents=1 libbeat.es.publish.read_bytes=444 libbeat.es.publish.write_bytes=14475 libbeat.es.published_and_acked_events=36 libbeat.publisher.published_events=36 publish.events=37 registrar.states.current=1 registrar.states.update=37 registrar.writes=1
2019-06-18T11:13:29+03:00 INFO No non-zero metrics in the last 30s

jan · June 18, 2019, 10:54am

Note: Filebeat output all messages to elasticsearch ,also i created input for filebeat in graylog3 and it is running with no messages recieved

Filebeat needs to output the Data to a Beats Input of Graylog - if you ingest the data direct into Elasticsearch that will not be visible in Graylog.

AmrAbdelFattah · June 18, 2019, 11:12am

Could you please send the output that i should put in filebeat.yml?

AmrAbdelFattah · June 18, 2019, 11:14am

Also,Can I use logstash instead of elasticsearch to ingest beats from filebeat?

AmrAbdelFattah · June 18, 2019, 1:46pm

issue solved after i configured filebeat to send output to logstash

jan · June 18, 2019, 1:53pm

Maybe you are confused because the naming in the Filebeat configuration is …

output:
  logstash:
    hosts: ["graylog.local.lan:5044"]

system · July 2, 2019, 1:53pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Input - Show received messages not showing logs Graylog Central (peer support)	7	6754	April 19, 2018
Graylog recieves messages but don't show them Graylog Central (peer support)	7	1556	March 24, 2018
Beats Input messages not showing in Graylog Graylog Central (peer support)	2	856	August 3, 2022
Messages being fed to Graylog and apparently indexed, but they're not visible anymore Graylog Central (peer support)	32	8512	November 19, 2019
Graylog not desplaying recent messages Graylog Central (peer support)	8	581	September 19, 2019

Graylog3 couldn't receive messages

Related Topics