Hi,
I’d like to share with you this strange (for me) filebeat behaviour.
I’m using filebeat to read a log file (MySql generic log).
this is an extract of the log file:
171009 14:04:26 7 Connect root@10.10.100.53 on
7 Query select @@version_comment limit 1
171009 14:06:26 7 Connect root@10.10.100.55 on
Splitting the message using \t we can parse the message.
The strange situation is that on graylog the massage value that I can use and parse is trimmed
The empty spaces are missing! Why?
So, can somebody explain me what happen? Who trim the message?
Is this the correct behaviour?
Thanks
Gianluca
jochen
(Jochen)
October 10, 2017, 4:08pm
2
What’s your Filebeat Configuration?
Hi @jochen
this is my filebeat.yml
filebeat:
prospectors:
- encoding: UTF-8
fields:
collector_node_id: graylog-collector-sidecar
gl2_source_collector: 6439f440-9cd8-4793-948e-ff3937d73d45
source_name: DBMS
type: log
ignore_older: 0
paths:
- /var/lib/mysql/mysql-mock.log
scan_frequency: 10s
tail_files: true
type: log
output:
logstash:
hosts:
- 10.10.100.211:5047
path:
data: /var/cache/graylog/collector-sidecar/filebeat/data
logs: /var/log/graylog/collector-sidecar
tags:
- mysqlmock
that comes from this Graylog collectors configurations:
Do you think that there could be a misconfiguration?
Thanks
Gianluca
jochen
(Jochen)
October 11, 2017, 6:46am
4
Graylog trims string values in messages automatically:
if ((RESERVED_FIELDS.contains(trimmedKey) && !RESERVED_SETTABLE_FIELDS.contains(trimmedKey)) || !validKey(trimmedKey)) {
if (LOG.isDebugEnabled()) {
LOG.debug("Ignoring invalid or reserved key {} for message {}", trimmedKey, getId());
}
return;
}
if (FIELD_TIMESTAMP.equals(trimmedKey) && value != null && value instanceof Date) {
fields.put(FIELD_TIMESTAMP, new DateTime(value));
} else if (value instanceof String) {
final String str = ((String) value).trim();
if (isRequiredField || !str.isEmpty()) {
fields.put(trimmedKey, str);
}
} else if (value != null) {
fields.put(trimmedKey, value);
}
}
public static boolean validKey(final String key) {
Ok thanks.
But I don’t understand why the messagge is trimmed.
How you can manage a message that comes from a positional log?
In my case, but there are others example like SAP log, the position is as important as the presence of white spaces in the row.
So how I could resolve my problem? Is possible to find a workaround?
Gianluca
jochen
(Jochen)
October 11, 2017, 8:22am
6
I think you probably want to merge the two log lines instead of having them as separate messages in Graylog.
You can use the multiline feature (see the Graylog Collector Sidecar configuration) for this.
Hi @jochen
I’m reading the MySql general query log (tha is a general record of what mysq is doing)
So each row is different from the others and could be not related with the before one.
I don’t know if merging the timestamp could be a solution even if it is intuitively correct.
Can you explain me how does the multiline features works? Can you give me an example please?
For example, the MySql file ends with the SQL command executed and sometimes it can be null
171011 10:09:11 11 Quit
Is the the possibility that the multiline adding something different then the ‘null’ sql command (that is the correct value)?
Thanks
Gianluca
jochen
(Jochen)
October 11, 2017, 8:58am
8
Thanks a lot.
I will take a look to your suggested link
system
(system)
Closed
October 25, 2017, 9:00am
10
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.