Extra entrys in graylog -


(Andy Hughes) #1

I am using Rsyslog to forward logs to the Graylog server, but I am getting extra lines that dont appear in the original log, appearing in graylog.

Any ideas on what may cause this please?

I have attached 2 screen shots showing the grep of the vs the graylog equiv, you will see in graylog the lines are duplicated about 50 times

here is my rsyslog config for the example file

$ModLoad imfile

$InputFileName  /usr/local/expensys/test-f/tomcat/logs/expensys-local-app.log
$InputFileTag   exlog-test-f-app
$InputFileStateFile     exlog-test-f-app
$InputFileSeverity      info
$InputFileFacility      local7
$InputRunFileMonitor
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
local7.*  @@192.168.xxx.xxx:1514

here is the line in my rsyslog master config to add the server

*.* @@192.168.201.222:1514

What logs would be helpful?


(Andy Hughes) #2

Also forgot to mention I have a exclude.conf file to stop local7 also going to boot log

:programname, contains, "exlog-"  ~

My full rsyslog.conf is (the forwarder is at the end)

# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler


# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log
#local7.*                                               /dev/null

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
*.* @@192.168.201.222:1514


# ### end of the forwarding rule ###
#*.* @@172.30.4.106:1514;RSYSLOG_SyslogProtocol23Format

(Jochen) #3

I guess you’re referring to the “Sign-On tokens” part (where the mouse cursor on the screenshot is).

These aren’t extra lines but simply a line wrap because the browser window is too small to show the complete message in a single line.


(Andy Hughes) #4

in the original message on the server I am getting 1 entry every 10 minuest.

in graylog, there are 10 identical lines appear with the same time stamp for 1 single line that appears in the original log of same timestamp


(Andy Hughes) #5

so for each SSO Sign on token entry on server log, I am getting 10 of that line appearing in Graylog on that timestamp


(Jochen) #6

Which version of rsyslog are you using?
Has the state file been successfully created and updated?
Are there other custom configuration snippets in /etc/rsyslog.d/?

It could also be a side effect of having $InputFilePollInterval and $InputFilePersistStateInterval 1 with the same value (1 second), try setting $InputFilePersistStateInterval to a higher value (5 seconds sounds like a good compromise).


(Andy Hughes) #7

Thanks, I have no understanding of this line, can you advise what it does?

$InputFilePersistStateInterval


(Andy Hughes) #8

there is a conf file for each log file I am inputting all following the same formats at the one posted,
there is a exclude line as outlined above as it was outputting the same logs to the boot log
and then the oiutput to TCP entry line that I put in the rsyslog.conf file

I havent added anything else, do you think i have missed something?

I’ll try the change you reccomend now for that log file, will be a pain if thats the cause , as I have 90 conf files :slight_smile:


(Andy Hughes) #9

I made the change you suggested for that log, this had no impact, and a further 10 line entrys just appeared


(Jochen) #10

See http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html#input-parameters for a description of the configuration settings.


(Andy Hughes) #11

Thanks for this, have take a look and altered according to suggested change, no joy Im afraid, is there any way to view the raw input arring ( the spool) that you know of?


(Jochen) #12

You could use Wireshark or tcpdump to check what rsyslog is sending to Graylog.

Is this line part of multiple configuration files in /etc/rsyslog.d?

This might cause sending messages multiple times. Make sure to read http://www.rsyslog.com/doc/v8-stable/configuration/global/options/rsconf1_includeconfig.html as well.


(Andy Hughes) #13

I am unsure, as have always done it this way

The line you mention exists at the end of each file feed in and output config so each log I want sent in addition to standard message log, I add a conf file that matches this

$ModLoad imfile

$InputFileName  /usr/local/expensys/test-f/tomcat/logs/expensys-local-app.log
$InputFileTag   exlog-test-f-app
$InputFileStateFile     exlog-test-f-app
$InputFileSeverity      info
$InputFileFacility      local7
$InputRunFileMonitor
$InputFilePollInterval 10
$InputFilePersistStateInterval 0
local7.*  @@192.168.201.222:1514

For example on this server I have the following files, each one unique for the log file to be sent

-rw-r--r-- 1 root root 324 Apr 18 10:09 apache-access-test4.conf
-rw-r--r-- 1 root root 369 May 16 10:52 application-automation-test.conf
-rw-r--r-- 1 root root 351 May 16 10:06 application-ops-alpha.conf
-rw-r--r-- 1 root root 351 May 16 10:08 application-ops-bravo.conf
-rw-r--r-- 1 root root 363 May 15 10:44 application-support-alpha.conf
-rw-r--r-- 1 root root 363 Apr 18 14:10 application-support-bravo.conf
-rw-r--r-- 1 root root 369 Apr 18 14:11 application-support-charlie.conf
-rw-r--r-- 1 root root 343 May 23 11:14 application-test-f.conf
-rw-r--r-- 1 root root 369 May 15 10:45 application-transform-alpha.conf
-rw-r--r-- 1 root root 369 May 15 10:46 application-transform-bravo.conf
-rw-r--r-- 1 root root 294 May  5 07:58 bckp-test4.conf
-rw-r--r-- 1 root root  37 May  5 07:32 exlog-exclude.conf
-rw-r--r-- 1 root root 305 Apr 18 14:11 postfix-test4.conf
-rw-r--r-- 1 root root 372 Apr 18 14:12 tomcat-automation-test.conf
-rw-r--r-- 1 root root 377 May 16 10:55 tomcat-build-test-alpha.conf
-rw-r--r-- 1 root root 354 May 16 10:46 tomcat-ops-alpha.conf
-rw-r--r-- 1 root root 354 May 16 10:47 tomcat-ops-bravo.conf
-rw-r--r-- 1 root root 366 May 16 10:50 tomcat-support-alpha.conf
-rw-r--r-- 1 root root 368 Apr 18 14:13 tomcat-support-bravo.conf
-rw-r--r-- 1 root root 369 Apr 18 14:14 tomcat-support-charlie.conf
-rw-r--r-- 1 root root 345 May 16 11:19 tomcat-test-f.conf
-rw-r--r-- 1 root root 372 May 16 11:20 tomcat-transform-alpha.conf
-rw-r--r-- 1 root root 372 May 16 11:21 tomcat-transform-bravo.conf

Doesnt this tell it to use the local7 facility and output?

I also have this line at the end of the rsyslog.conf file

*.* @@192.168.201.222:1514

am i missing a trick?


(Jochen) #14

Each of these statements opens a “channel” for messages in the facility “local7” to be sent to Graylog.

I’m pretty sure that you have the same number of these statements in your configuration files as you have message duplicates in Graylog.

You can also test this by using the rsyslog stdout output module: http://www.rsyslog.com/doc/v8-stable/configuration/modules/omstdout.html


(Andy Hughes) #15

From my understanding , this send the main logs that are included in local 7 by standard, like the messages and securitylogs, and the Confs are also a include?


(Jochen) #16

Each of these statements are sending messages with facility “local7” to 192.168.201.222:1514 and you have this statement in all of your custom configuration snippets.


(Andy Hughes) #17

Ok I removed it, all we lost were the standard logs, its still duplicating the other lines,

I have already stopped all the additional stipulated ones from the rsyslog.conf directory being duplicated by using the exclude file that stops all tagged logs “exlog” from going to boot.log (local7)

exlog-exclude.conf contains the line
:programname, contains, “exlog-” ~ , this stops these files going again under the . directive


(Jochen) #18

So what’s your complete, merged rsyslog configuration now?


(Andy Hughes) #19

??? I havemt merge it, thats not plausible, they have to be seperate so I can control them by scripted site installs , do you mean you want to see all of the config files, or just the one that I am using for this example ?


(Andy Hughes) #20
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

##########################################
#testf listed here for include purposes


$ModLoad imfile

$InputFileName  /usr/local/expensys/test-f/tomcat/logs/expensys-local-app.log
$InputFileTag   exlog-test-f-app
$InputFileStateFile     exlog-test-f-app
$InputFileSeverity      info
$InputFileFacility      local7
$InputRunFileMonitor
$InputFilePollInterval 10
$InputFilePersistStateInterval 0
local7.*  @@192.168.201.222:1514


###########################################







#### RULES ####

$RepeatedMsgReduction on

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler
q


# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log
#local7.*                                               /dev/null

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
#*.* @@192.168.201.222:1514


# ### end of the forwarding rule ###
#*.* @@172.30.4.106:1514;RSYSLOG_SyslogProtocol23Format