Sending logs from filebeat to kafka and kafka to graylog

Graylog Central (peer support)
1

Hi All,

I am sending logs from my CentOS-7 Servers to Kafka using filebeat 7.4 and from Kafka to graylog. I have created Syslog-Kafka input on graylog. All are working fine, the messages are received successfully but the source is displayed by graylog as unknown, the facility is unknown and other beats-related fields don’t exist.

Filebeat.yml

filebeat:
  inputs:
  - encoding: plain
    exclude_files: []
    fields:
      type: log
      enabled: true
      close_eof: true
      apache: true
      fields_under_root: true
    ignore_older: 0
    multiline:
      match: after
      negate: true
      pattern: ^\d{4}-\d{2}-\d{2}
    paths:
    - /var/log/kaarya/*.log
    scan_frequency: 10s
    symlinks: false
    tail_files: true
output:
  kafka:
    enabled: true
    codec.format:
        string: '%{[message]}'
    hosts: ["192.168.21.167:9092"]
    topic: 'GraylogLogs'
    version: '0.11.0.0'
    partition.round_robin:
            reachable_only: false

    required_acks: -1
    max_message_bytes: 1000000

Graylog version - 3.0

47%20AM
Screen Shot 2019-10-08 at 4.58.47 AM.png1426×561 69.4 KB

Quick reply will be appreciated.

2

please re-read your own text slowly again:

You send in Syslog messages picked up via filebeat into kafka and read that with a kafka syslog input on graylog …

How should Graylog Identify the messages as Syslog when they are ingested via beats?

You need to parse the messages yourself, I would advice to take the kafka raw input and create a processing pipeline to parse all fields.

3

@jan Thanks for the reply.
I am sending data directly to Kafka through filebeat and it reads that with a Kafka Syslog input on graylog.

I also tried with Kafka raw input but the result is same. Can you please tell me what I am missing here.

Thanks

4

Graylog does not have the option currently to reat BEATS via Kafka.

You need to work with Kafka RAW and parse that yourself with the processing pipeline. You should open a feature request for beats via kafka input.

5

Thanks, @jan. I will work on Kafka RAW.
Also, can we solved this problem if we route our messages with Logstash or rsyslog? I tried implementing this with the help of this Documentation: https://marketplace.graylog.org/addons/113fd1cb-f7d2-4176-b427-32831bd554ee but I failed.

Do we have any detailed documentation? or is there any different tools through which we can solve this?

6

depending on the use case it might not be suitable to use syslog or beats.

That highly depends what you ingest and what features you need from the shippers.

7

@jan I simply want logs to be print on graylog from kafka broker. For this I need to first send logs to Kafka. Can we use sidecar to send logs to Kafka?

8

Sidecar is just a program to configure collectors - not to send or collect logs.

You need to know what kind of logfiles you want to collect from what you tell it is not clear what your goal is.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.