How to merge log lines from multiple logs

[dmirc]

How to merge log lines from multiple logs (matching time, or some ID)

For example, I have a TT2 test log and in that log, there is a line “Test class started: Ts5225”
I want to merge this line matching ID Ts5225 in the NODE log, basically, this line should be starting the section where test Ts5225 is run.

[jan (Jan Doberstein)]

he @dmirc

such is not possible with a plain vanilla Graylog.

you need to make this move during collection of the log, but once committed to Graylog this is not possible.

[dmirc]

Hi @jan,

Thank you for your time and help.

I’ve done some of line merges, in this case by timestamp:

#cat filebeat.yml

filebeat.inputs:
- type: log
paths:
- /logs/**/*.log
multiline.pattern: ‘^[1]{4}-[0-9]{2}-[0-9]{2}’
multiline.negate: true
multiline.match: after
fields:
tags: [“applog”]
branch: ${branch}
buildnumber: ${buildnumber}
fields_under_root: true
queue.mem:
events: 65536

logging.level: debug
logging.selectors: ["*"]
output.logstash:
#   hosts: ["10.254.68.42:5044"]
   hosts: ["graylog:5044"]

But now im not sure how to proceed:

1. continue to regex in filebeat config
2. or is it possible to make streams(maybe 2 streams), pipeline, and then extract the line with: Test class started: Ts5225, and after just make some order in output. so that i have logs lines/blocks in correct flow.

What would you suggest?

---

1. 0-9 ↩︎