guys, most of my servers are coming into graylog through syslog messages as a resolved hostname however I just added a new server to monitor incoming syslogs but it’s coming in as an IP address and not a resolved hostname. I’m running Graylog 3.0 on centos7. I’ve tried adding the hostname and IP address to the /etc/hosts file and have looked at my search domain in my /etc/resolv.conf file and everything looks good. I see that this was an issue in graylog 2.4 and found an article mentioning it would be fixed in newer releases. I’m wondering if that hasn’t been fixed and I need to add something called a pipeline? or modify my input to have the “full message” stored? thanks in advance for any help!
first of all, did you checked the source?
So how does the servers write their hostname into the original log message? As IP or as hostname?
How did you have configured your syslog input exactly?
I copied the following line (the same line i’ve put into all of the rest of my servers) see below:
#Graylog server settings
. @@graylog:5014;RSYSLOG_SyslogProtocol23Format
I just installed authbind on the server so I could use port 514 to allow my printers to send syslog messages in. Do you think that would that be causing any issues? these are the ports I have configured with authbind (!12201 514 !514)
I have and input setup in graylog for 5014 and 514
Did you checked the sources? How do they write into their local log?
I believe through rsyslog.conf. The server is running Oracle Linux 6.10 right now. I don’t believe were pushing logs from any other application. is there a way to check where the logs are coming from? I can attach the whole rsyslog.conf if that helps?
inside the rsyslog.conf file:
#Graylog server settings
. @@graylog:5014;RSYSLOG_SyslogProtocol23Format
here’s the section in the rsyslog.conf file that I believe is pushing logs to /var/log/messages
Don’t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
here’s some additional information:
I ran the following command: to see what files were in use:
lsof -p $(pgrep -f syslog) -a +D /var/log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 1923 root 1w REG 249,0 6295945 793792 /var/log/secure
rsyslogd 1923 root 2w REG 249,0 2735194 793791 /var/log/messages
rsyslogd 1923 root 4w REG 249,0 2365 793790 /var/log/maillog
rsyslogd 1923 root 8w REG 249,0 244734 793785 /var/log/cron
once again HOW DOES THE LOGFILE LOOK ON THE SERVER?
Does it contain the hostname, the IP or localhost?
What is the content of /var/log/messages is my questions since reply #1
1 Like
yes it’s writes the to the log as the localhost name (servername) there is NO ip address in the /var/log/messages.
fix your syslog configuration to solve your problem
my syslog is exactly the same as the other servers that are coming in with hostnames. below is the output of the rsylog.conf file in /etc Please let me know if you see anything wrong:
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
#Graylog server settings
*.* @@graylog:5014;RSYSLOG_SyslogProtocol23Format
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###find what is different between all servers …
I guess it has something todo with the DNS resolution of the host.
Yeah, I checked that too. I added my server that’s coming in as an IP address into the /etc/hosts file on graylog and also added graylog to the /etc/hosts file on the server to bypass dns. that still doesn’t work. is anyone else having this issue in graylog 3.0?
I compared the rsyslog.conf on one server that is functioning correctly and the other server that is not and they’re rsyslog.conf files are a perfect match.
and what is about the /etc/hosts and hostnames of the servers, Sherlock?
How do they write in their local logfiles?
Selinux was the problem, I put that into permissive mode, restarted the rsyslog service and logs started coming through. Jan, you should put that in your documentation that selinux causes issues with rsyslogs. I found this recommendation from jtkarvo on the Graylog post “Rsyslog not sending logs”.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.