How to configure multiple beats input for various applications?


(Hariom Vashisth) #1

I am using 2.4.6 graylog and installed collector-sidecar on some another server. For single configuration where only one beat input/output works! but when i add one more beats input it throws one warning but older one is still working.

    ubuntu@ip-172-50-50-52:~$ sudo tail -f /var/log/graylog/collector-sidecar/filebeat
2018-08-14T10:46:17Z INFO Loading registrar data from /var/cache/graylog/collector-sidecar/filebeat/data/registry
2018-08-14T10:46:17Z INFO States Loaded from registrar: 3
2018-08-14T10:46:17Z WARN Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-08-14T10:46:17Z INFO Loading Prospectors: 2
2018-08-14T10:46:17Z INFO Starting Registrar
2018-08-14T10:46:17Z INFO Starting prospector of type: log; ID: 9751877527429249501 
2018-08-14T10:46:17Z INFO Starting prospector of type: log; ID: 644587000511631764 
2018-08-14T10:46:17Z INFO Loading and starting Prospectors completed. Enabled prospectors: 2
2018-08-14T10:46:27Z INFO Harvester started for file: /home/ubuntu/spock-core/log/spock-access.log
2018-08-14T10:46:47Z INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30002 beat.memstats.gc_next=5878576 beat.memstats.memory_alloc=4516208 beat.memstats.memory_total=12882288 filebeat.events.active=2 filebeat.events.added=110 filebeat.events.done=108 filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.config.module.running=0 libbeat.output.events.acked=106 libbeat.output.events.batches=6 libbeat.output.events.total=106 libbeat.output.read.bytes=36 libbeat.output.type=logstash libbeat.output.write.bytes=7810 libbeat.pipeline.clients=2 libbeat.pipeline.events.active=2 libbeat.pipeline.events.filtered=2 libbeat.pipeline.events.published=108 libbeat.pipeline.events.retry=92 libbeat.pipeline.events.total=110 libbeat.pipeline.queue.acked=106 registrar.states.current=3 registrar.states.update=108 registrar.writes=8

refer below details :slight_smile:


(Jan Doberstein) #2
  1. is it not working?
  2. did you read the WARN until the end?

(Hariom Vashisth) #3

@jan
First configured beats input is working.
Second input is not working and also getting “WARN” ->

Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.

Also, can you please educate me how to fetch multiple application logs through filebeat-collector from same server.

Options:

  1. creating mupliple beats input for one beats output.
  2. seprate collector-configuration for every new application logs.

(Hariom Vashisth) #4

@jan

— SOLVED----

So, I figured out that graylog automatically genrates filebeat configuration when we add additional beats input with same beats output and on same input through graylog collector.


(system) #5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.