Sending logs from filebeat to kafka and kafka to graylog

Hi All,

I am sending logs from my CentOS-7 Servers to Kafka using filebeat 7.4 and from Kafka to graylog. I have created Syslog-Kafka input on graylog. All are working fine, the messages are received successfully but the source is displayed by graylog as unknown, the facility is unknown and other beats-related fields don’t exist.

Filebeat.yml

filebeat:
  inputs:
  - encoding: plain
    exclude_files: []
    fields:
      type: log
      enabled: true
      close_eof: true
      apache: true
      fields_under_root: true
    ignore_older: 0
    multiline:
      match: after
      negate: true
      pattern: ^\d{4}-\d{2}-\d{2}
    paths:
    - /var/log/kaarya/*.log
    scan_frequency: 10s
    symlinks: false
    tail_files: true
output:
  kafka:
    enabled: true
    codec.format:
        string: '%{[message]}'
    hosts: ["192.168.21.167:9092"]
    topic: 'GraylogLogs'
    version: '0.11.0.0'
    partition.round_robin:
            reachable_only: false

    required_acks: -1
    max_message_bytes: 1000000

Graylog version - 3.0

Quick reply will be appreciated.

please re-read your own text slowly again:

You send in Syslog messages picked up via filebeat into kafka and read that with a kafka syslog input on graylog …

How should Graylog Identify the messages as Syslog when they are ingested via beats?

You need to parse the messages yourself, I would advice to take the kafka raw input and create a processing pipeline to parse all fields.

@jan Thanks for the reply.
I am sending data directly to Kafka through filebeat and it reads that with a Kafka Syslog input on graylog.

I also tried with Kafka raw input but the result is same. Can you please tell me what I am missing here.

Thanks

Graylog does not have the option currently to reat BEATS via Kafka.

You need to work with Kafka RAW and parse that yourself with the processing pipeline. You should open a feature request for beats via kafka input.

Thanks, @jan. I will work on Kafka RAW.
Also, can we solved this problem if we route our messages with Logstash or rsyslog? I tried implementing this with the help of this Documentation: https://marketplace.graylog.org/addons/113fd1cb-f7d2-4176-b427-32831bd554ee but I failed.

Do we have any detailed documentation? or is there any different tools through which we can solve this?

depending on the use case it might not be suitable to use syslog or beats.

That highly depends what you ingest and what features you need from the shippers.

@jan I simply want logs to be print on graylog from kafka broker. For this I need to first send logs to Kafka. Can we use sidecar to send logs to Kafka?

Sidecar is just a program to configure collectors - not to send or collect logs.

You need to know what kind of logfiles you want to collect from what you tell it is not clear what your goal is.