How to append a log file for rsyncs in graylog

Description of your problem

How can you append to a log file in gray log? I have rsync logs saved to a location on my storage. This rsync log is synced via filebeat to graylog.

If the rsync has started and its a large rsync then the log stops at “sending incremental file list”. Once this is complete then the rsync starts again but the log file in gray log does not get appended and a new log file gets created. However my saved log file does get appended on my storage.

The logs are saved to the messages field in graylog.

Description of steps you’ve taken to attempt to solve the issue

In filebeat config I have used the option multiline.max_lines: 5000 and this works fine for small rsyncs.

Operating system information

  • CentOS 7.9

Package versions

  • Graylog 4.1.2

Hello,

For Linux Use >> instead of > when directing output to a file:

your_command >> file_to_append_to

Example:

$ echo "hello" > file
$ echo "world" >> file
$ cat file 
hello
world

Not sure how to do this in Graylog.

Hi gsmith, i know how to use the append command in Linux. My question is how do i get a rsync log that is going into graylog keep appending to the messages section. Looks like if the rsync takes along time to gather the incremental file list before it actually starts the rsync graylog then thinks that log is finished and when the actually rsync starts it creates another log file. This way I have 1 rsync logging in 2 log files in graylog.

To be honest I don’t think this is a Graylog question but I’ll try to help. Can you show how your doing this rsync? Show any of your configurations or screen shots would be appreciated. Perhaps some examples would be nice.

I spoke to a graylog support engineer and he said this can’t be done. Either add a wait in the filebeat process or pass the final log as one once its all written then send to graylog.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.