Log formatting for logs coming from beat input into graylog

[tor]

Description of your problem

Currently my logs are coming into graylog via a beats input. But if i have 1 log file on my server and that has 20 lines in the log for example then this log files is coming into gray log as 20 different log entries

Operating system information

CentOS 7

Here is my log on the server.

---------------------------------------- TRANSFER START 2021-09-02|11:12:31.478 ----------------------------------------
 SCRIPT="/Volumes/resources/bin/datasync" 
 ACTION="out" 
 USER="username" 
 SOURCE_PATH="/Volumes/projects/test/" 
 DEST_PATH="root@dataio:/dataio/outbox/test/outbox" 
 SOURCE_IP="192.168.1.74" 
 DEST_IP="192.168.4.52" 
 Raw rsync output=
"sending incremental file list
test_07/
test_07/210901_to/
ntest_07/210901_to/999_010_v016_h264.mov
       3.07M 100%  152.62MB/s    0:00:00 (xfer#1, to-check=2/6)
test_07/210901_to_02/
ntest_07/210901_to_02/999_010_v017_h264.mov
       2.85M 100%   90.62MB/s    0:00:00 (xfer#2, to-check=1/6)
test_07/210901_to_02/999_050_parisSky_v006_h264.mov
       2.51M 100%   63.09MB/s    0:00:00 (xfer#3, to-check=0/6)

sent 8.44M bytes  received 81 bytes  5.63M bytes/sec
total size is 8.44M  speedup is 1.00

--------------------------- TRANSFER FINISHED 2021-09-02|11:12:33 - TOTAL TIME - 00h:00m:03s ---------------------------"

[gsmith]

Hello,

You need to configure you Beat/s during collection of the logs, but once committed to Graylog I don’t this this is not possible.

Filebeat supports merging multiple lines into a single event: you can find it here Managing Multiline Messages.
Hope that helps

[tor]

Thanks for this gsmith. I was wondering as I am using filebeat which is built in graylog. I have turned of the filebeat service because graylog sidecar only needs access to the binary of filebeat service.

Does that mean i can still use the filebeat.yml in the installed filebeat on the server? Or would these settings need to be added to the filebeat.cfg file in graylog GUI?

[gsmith]

@tor

Or would these settings need to be added to the filebeat.cfg file in graylog GUI?

You would do all your setting in the Web UI. No need to go into the Graylog server.

[tor]

Sorry what web user interface? The sidecar interface in graylog? That is the only one I have access too.

[gsmith]

Sorry,
Its Friday and I’m really tired. I meant this section.

@tor

That is the only one I have access too.

That’s the only one you need
Hope that helps

[tor]

Thanks for that gsmith. But I have 2 files a filebeat-conf file and a Log collector filbeat Linux. I can edit both. Which one of these would I add that multiple lines configs would it be the filebeat Linux conf file or my filebeat-conf file?

[gsmith]

Hello,

Pro tip:
Any time you want to reconfigure a file, or adjust settings to any part of your environment always make a copy from the original one.
That being said, I would high recommend the following.

Navigate to localhost:9000/system/sidecars/configuration

Choose your Filebeat Collector ( Linux, Windows)

Click on “More actions” and choose clone. Think ahead for name your convention.

I will choose FileBeat for Linux because its for my Linux server

Click done. Now go to your new Log Collector called tor_filebeat and Click on Edit

Now fill in the settings you want.

Bonus round, if things go bad you have the original copy of FileBeat for Linux.

NEXT

If everything is working at this point configure your FileBeat

@tor

I add that multiple lines configs

as shown below in this section on the Web UI.

Should be good. If you have unique server/s I would create those Beats separately and then create Beats your other similar servers/Node/PC/devices. It all depends on what you want to do.

Hope that help, That will be $200.00 for the screenshots

[tor]

Thanks for this gsmith appreciated. I was testing some configs because at present my collectors configuration and the Log Collector config look exactly the same.

I did a test by hashing out the path in the Log Collector config 1st and my logs were still coming into gray log. When I hashed the path in the collectors configuration. Then my logs stopped. So now I know that the collectors config is being used for log path.

Is there a separation what goes into which file. Your example has stated that the multi line variables go into the Log Collectors, is this correct? and I presume collectors configuration is use for the path where the logs are located?

[gsmith]

Hello,

@tor

Is there a separation what goes into which file.

Should only be one file (FileBeat) that your configuring. I also showed you where to configure that from the screen shot above.

@tor

Your example has stated that the multi line variables go into the Log Collectors, is this correct?

The link I posted above for multi-line variables goes in your FileBeat configuration file.

In the screenshot below is where you would do all your configurations for FileBeat. I marked it with a RED box.

I think your getting confused with Graylog Sidecar install and FileBeat package Install. Or I’m just not understanding what you mean. If this is correct, please post screenshot/s to help us better understand your questions.

Install the GL Sidecar on all your remote devices and control those GL Sidecars from a central hub which from the Web User Interface.

@tor

I presume collectors configuration is use for the path where the logs are located

I’m not sure what you mean by “where logs are locate”. Please enlighten me.

Here is a brief description taken from the Graylog Documents.

“Graylog Collector Sidecar is a lightweight configuration management system for different log collectors, also called Backends. The Graylog node(s) act as a centralized hub containing the configurations of log collectors. On supported message-producing devices/hosts, Sidecar can run as a service (Windows host) or daemon (Linux host).”

Graylog Sidecar is basically a wrapper for Log collectors (Nxlog, FileBeat. WinlogBeat, etc…) hence you would execute all your configuration on the Graylog’s Web UI.

When you configure your Log Collectors ( FileBeat) the generated configuration file from the web interface is located here.

/var/lib/graylog-sidecar/generated

There should be no need to go into this file manually.

Hope that helps

[tor]

The path field is the location in my collector config where it will look for the logs to send.

This shows my log collector filebeat file. This is the file you said to add the multiline entries.

This shows the name of my collector config file and also the name of my log collector file

This is the collectors file with the path of the log file. Should the multiline entry be in this file?

Shows my collector config file running.

Shows my sidecar running on my server " Collector"

So to round up there are 2 files 1. a collector config file, 2. Log collector file for file beat.

On Log collector it does not like the location of my logs so I have added it to the collector config file.

[gsmith]

Hello,

@tor

This is the collectors file with the path of the log file. Should the multiline entry be in this file?

Yes , and good job

Not much more I can tell you accept what I posted above.

Hope that helps you out. I know the struggle is real.

[tor]

Hi gsmith,

I got the multiline part working and my log file is coming in as 1 message.

However i have noticed that my logs seem to come in twice. I have the 2 same logs in graylog appear. Do you know if there is some setting somewhere where this can be changed?

[tor]

would that be because it uses 2 streams?

[tor]

I fixed it. It was because of 2 streams

[gsmith]

Good Job @tor , see you can do this

Also, when you need help I would highly suggest adding some more info to your questions. Its VERY hard to see or understand what your going through.