FileBeat and Sidecar: who trim the message?


(Gianluca Valentini) #1

Hi,
I’d like to share with you this strange (for me) filebeat behaviour.

I’m using filebeat to read a log file (MySql generic log).
this is an extract of the log file:

171009 14:04:26	    7 Connect	root@10.10.100.53 on 
		            7 Query	select @@version_comment limit 1
171009 14:06:26	    7 Connect	root@10.10.100.55 on 

Splitting the message using \t we can parse the message.

The strange situation is that on graylog the massage value that I can use and parse is trimmed

image

The empty spaces are missing! Why?

So, can somebody explain me what happen? Who trim the message?
Is this the correct behaviour?

Thanks
Gianluca


(Jochen) #2

What’s your Filebeat Configuration?


(Gianluca Valentini) #3

Hi @jochen
this is my filebeat.yml

filebeat:
  prospectors:
  - encoding: UTF-8
    fields:
      collector_node_id: graylog-collector-sidecar
      gl2_source_collector: 6439f440-9cd8-4793-948e-ff3937d73d45
      source_name: DBMS
      type: log
    ignore_older: 0
    paths:
    - /var/lib/mysql/mysql-mock.log
    scan_frequency: 10s
    tail_files: true
    type: log
output:
  logstash:
    hosts:
    - 10.10.100.211:5047
path:
  data: /var/cache/graylog/collector-sidecar/filebeat/data
  logs: /var/log/graylog/collector-sidecar
tags:
- mysqlmock

that comes from this Graylog collectors configurations:


Do you think that there could be a misconfiguration?

Thanks
Gianluca


(Jochen) #4

Graylog trims string values in messages automatically:


(Gianluca Valentini) #5

Ok thanks.
But I don’t understand why the messagge is trimmed.
How you can manage a message that comes from a positional log?

In my case, but there are others example like SAP log, the position is as important as the presence of white spaces in the row.

So how I could resolve my problem? Is possible to find a workaround?
Gianluca


(Jochen) #6

I think you probably want to merge the two log lines instead of having them as separate messages in Graylog.

You can use the multiline feature (see the Graylog Collector Sidecar configuration) for this.


(Gianluca Valentini) #7

Hi @jochen

I’m reading the MySql general query log (tha is a general record of what mysq is doing)
So each row is different from the others and could be not related with the before one.
I don’t know if merging the timestamp could be a solution even if it is intuitively correct.

Can you explain me how does the multiline features works? Can you give me an example please?

For example, the MySql file ends with the SQL command executed and sometimes it can be null

171011 10:09:11 11 Quit

Is the the possibility that the multiline adding something different then the ‘null’ sql command (that is the correct value)?

Thanks
Gianluca


(Jochen) #8

Please refer to https://www.elastic.co/guide/en/beats/filebeat/5.6/multiline-examples.html


(Gianluca Valentini) #9

Thanks a lot.

I will take a look to your suggested link


(system) #10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.