Log message deplicate

Hi All,

I’m using rsyslog as input to Graylog and the configuration as below:

$ModLoad imfile

$InputFilePollInterval 10
$InputFileName <logfile1>
$InputFileTag <tag1>
$InputFileStateFile <state1>
$InputFileSeverity info
$InputFileFacility local1

$InputFilePollInterval 1
$InputFileName <logfile2>
$InputFileTag <tag2>
$InputFileStateFile <state2>
$InputFileSeverity info
$InputFileFacility local2

local1.info @localhost:5140
local2.info @localhost:5140

And I found that the log messages shown on Graylog are duplicated. Sometimes only some of messages duplicate once/ twice(some duplicate 5 times) and sometimes messages are missing.

After checked the configurations on Graylog and rsyslog, I have no ideas for above case.

For the duplicate messages, id are different such as:

The logs are generated by python program around 100 messages per mintues.

There is only one index and I choose one stream to view the log.

Please help. Thanks

Sorry for missing info:
OS Information: RHEL 7.9
Package Version: Graylog 4.0.5+d95b909

Hi Henrylam,

I suspect this will be an interaction between Rsyslog.d and the log generated by the python program. Neither of these are Graylog components, and I’m no expert on Rsyslog.d, so perhaps someone else can illuminate on this.

Personally, I would look to swap use of Rsyslog.d to instead use Filebeats. It’s easier to set up and maintain than Rsyslog and more importantly, it keeps track of which messages Graylog has received - so if Graylog is stopped and then later resumed, no messages are dropped in the intervening period.

Hello @Henrylam

I have to completely agree with @tellistone suggestion about FileBeat. When I first started out I want to use a service that was native to Linux (Rsyslog) but later I found out that is was more difficult to configure Rsyslog as my environment grew or if we needed to improve on how logs were sent to Graylog.

That being said. Have you tried to only use one of these?

On one of my CentOS 7.9 Lab servers I’m just running this configuration.

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
$ModLoad imudp
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
kern.debug /var/log/firewall.log
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
*.*@  <--- UDP connection
*.*@@  <--- TCP connection

Here are some sites I have used in the past to help me out.


Hope that helps

1 Like

Thanks @gsmith and @tellistone.

I will look into rsyslog service and update in here if the problem addressed.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.