Crashing and Burning...rsyslog assistance

adam · April 13, 2017, 1:36pm

Good Morning everyone,

I’m working on setting up Graylog to receive logs from another Bro IDS running Ubuntu 16.04.2. Both of the servers are on the same network. For reference i’m following this Content pack plugin. https://marketplace.graylog.org/addons/5e6cf3c6-7bdc-4a2c-bdca-441407e4a016. I believe this is configured correctly, but providing this to paint a picture of my end goal.

In the Graylog web interface I setup my input like this:
Bind address: 0.0.0.0
Port: 15514
All Other Options as Default:
Here is my netstat -plntu output from my Graylog Server showing that 15514 is listening.

tcp        0      0 10.128.0.13:9300        0.0.0.0:*               LISTEN      861/java        
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      885/sshd        
tcp        0      0 127.0.0.1:7001          0.0.0.0:*               LISTEN      862/etcd        
tcp        0      0 0.0.0.0:42341           0.0.0.0:*               LISTEN      861/java        
tcp        0      0 0.0.0.0:3333            0.0.0.0:*               LISTEN      861/java        
tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN      859/mongod      
tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      862/etcd        
tcp        0      0 10.128.0.13:9200        0.0.0.0:*               LISTEN      861/java        
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      857/nginx.conf  
tcp        0      0 0.0.0.0:34929           0.0.0.0:*               LISTEN      861/java        
tcp6       0      0 :::22                   :::*                    LISTEN      885/sshd        
tcp6       0      0 :::15514                :::*                    LISTEN      858/java        
tcp6       0      0 :::4001                 :::*                    LISTEN      862/etcd        
tcp6       0      0 :::9350                 :::*                    LISTEN      858/java        
tcp6       0      0 :::9000                 :::*                    LISTEN      858/java        
tcp6       0      0 :::2379                 :::*                    LISTEN      862/etcd        
udp        0      0 0.0.0.0:38073           0.0.0.0:*                           653/dhclient    
udp        0      0 0.0.0.0:68              0.0.0.0:*                           653/dhclient    
udp        0      0 10.128.0.13:123         0.0.0.0:*                           1150/ntpd       
udp        0      0 127.0.0.1:123           0.0.0.0:*                           1150/ntpd       
udp        0      0 0.0.0.0:123             0.0.0.0:*                           1150/ntpd

On my Remote Ubuntu Server where I want the logs to be sent from I have the following rsyslog configuration.
Here is my /etc/rsyslog.conf

#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
module(load="imklog")   # provides kernel logging support
#module(load="immark")  # provides --MARK-- message capability

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

# Enable non-kernel facility klog messages
#$KLogPermitNonKernelFacility on

###########################
#### GLOBAL DIRECTIVES ####
###########################
#Set max message size for Large syslog messages
$MaxMessageSize 64k
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#

#
# Emergencies are sent to everybody logged in.
#
#*.emerg                                :omusrmsg:*

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
#       news.=crit;news.=err;news.=notice;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       /dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
# you must invoke `xconsole' with the `-file' option:
# 
#    $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
#      busy site..
#
#daemon.*;mail.*;\
#       news.err;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       |/dev/console

Then here is my 50-default.conf file in /etc/rsyslog.d/

:app-name, regex, "bro_*" {
    *.* @@10.128.0.13:15514;RSYSLOG_SyslogProtocol23Format
     stop
}

The 10.128.0.13 is my Graylog server and listening on the 15514 port as shown above.

Here are some outputs from the troubleshooting:
From the Remote Ubuntu Server

/usr/sbin/rsyslogd -f /etc/rsyslog.d/50-default.conf -N1
rsyslogd: version 8.16.0, config validation run (level 1), master config /etc/rsyslog.d/50-default.conf
rsyslogd: End of config validation run. Bye.

From the Graylog Server

/usr/sbin/rsyslogd -f /etc/rsyslog.d/50-default.conf -N1
rsyslogd: version 7.4.4, config validation run (level 1), master config /etc/rsyslog.d/50-default.conf
rsyslogd: End of config validation run. Bye.

Any assistance would be greatly appreciated. Sorry about the formatting, couldn’t figure out font adjustment.

jochen · April 13, 2017, 1:50pm

Does it work if you unconditionally send messages to Graylog, i. e. if you don’t try to match against app-name in your rsyslog configuration?

adam · April 13, 2017, 1:57pm

Just changed my 50-defaults.conf to reflect this:

  *.* @@10.128.0.13:15514;RSYSLOG_SyslogProtocol23Format

then restarted rsyslog services but no change. No logs are being sent from remote server to the Graylog server.

jochen · April 13, 2017, 1:58pm

Please use triple backticks to retain the formatting of your text snippets:

```
text
```

jochen · April 13, 2017, 1:59pm

Make sure that there are no packet filters/firewalls blocking access to port 15514/tcp on your Graylog node and that the machine can actually send packets to 10.128.0.13:15514.

You can also use netcat or a similar tool to send a debug message.

adam · April 13, 2017, 2:07pm

Good idea about NC. I can successful send a netcat message from the remote ubuntu server to the graylog server via 15514. In my POC environment I do not have the firewall configured on either server.

ubuntu@graylog:/var/opt/graylog/data/maps$ sudo ufw status
Status: inactive

Ubuntu Server

root@BRO:/etc/rsyslog.d# nc 10.128.0.13 15514
hi

Graylog Server

ubuntu@graylog:/var/opt/graylog/data/maps$ nc -l 15514
hi

adam · April 13, 2017, 2:29pm

Interesting update. I stopped the input in Graylog so I could use the 15514 port. Upon starting the input, messages started to be delivered using the unconditional filter. I’m going to go back and test the app-name filter now. Stand by.

adam · April 14, 2017, 4:26pm

Sorry about my delay. I could receive logs from my remote ubuntu server after stopping and starting the input in Graylog. When I add back in the :app_name, command it does not send logs.

As another route I went to my 50-defaults.conf file and added this:

Bro DNS Logs                    -/var/log/bro/current/dns.log

and added another stream rule in Graylog saying "

A message must match at least one of the following rules
Field: DNS.log
Type: Match Exactly
Value: /var/log/bro/current/dns.log

And even with this change, I have no logs feeding into my Graylog server.

jochen · April 14, 2017, 9:42pm

Are you sure that your rsyslog configuration with matching the :app-name is correct at all? Obviously it works without the conditional…

alias454 · April 17, 2017, 12:39pm

What happens if you use this template for rsyslog instead of RSYSLOG_SyslogProtocol23Format

template (name=“GRAYLOGRFC5424” type=“string”
string=“<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME:::lowercase% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n”
)

Alternatively, replace the :app-name property with :syslogtag in the regex rule.

adam · April 17, 2017, 2:45pm

Jochen:
I’m assuming that the configuration is correct. In Graylog i’m matching against bro_* with a match reg expression. Now this brings up a question. My logs are in my var/log/bro/currrent directory with a name of dns.log, conn.log. This is a little outside of my understanding of how the regex will cpature based on the bro_* filter. No where in the log or log name does it have bro_* etc. Could this by it is not picking up the logs and moving them?

Alias454, thanks for jumping in. I added

template (name="GRAYLOGRFC5424" type="string"
string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME:::lowercase% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
)

Right above my previous command of

:app-name, regex, "bro_*" {
    *.* @10.128.0.13:15514;GRAYLOGRFC5424
    stop
}

I tried both TCP and UDP connections adjusting the inputs each time with no change.

What would cause the :syslogtag to pick up logs? Sorry for my lack of understanding on this.

alias454 · April 18, 2017, 12:47am

Honestly, rsyslog can be brittle at times. You can check this for info related to filters
http://www.rsyslog.com/doc/v7-stable/configuration/filters.html

One thing to try is write debug messages to disk or turn on the “Store full message” feature in Graylog. That way we can see exactly what is being sent.

Make sure all your protocols match ie if sending via tcp, set the input for tcp if using udp etc etc. Also, how are you sending logs from BRO? Are you using syslog-ng and Security Onion or are you using a stand-alone install of BRO?

adam · April 18, 2017, 1:30pm

I will double check and make sure that i’m making @ to a UDP input and @@ to a TCP input. I will also look into the turning those settings on and see what I find.

I have a ubuntu 16.04.02 build running rsyslog v-8.16.0 and stand-alone install of bro. My Graylog server is running rsyslog v-7.4.4.

As I learning about Graylog, how does Graylog know to pickup bro_*? My logs are not labeled like (bro_dns.log, bro_conn.log, etc). I’m wondering if this is part of my problem. I don’t see where it would be renamed during the process either.

jochen · April 18, 2017, 3:43pm

Let’s just establish some facts from this topic:

Graylog Syslog inputs are working if stuff is actually being sent to them.
rsyslog correctly forwards logs to Graylog if the configuration is correct.
The rsyslog template RSYSLOG_SyslogProtocol23Format is working and shouldn’t be replaced by a custom (error-prone) template.

I don’t know how Bro is using the syslog facilities rsyslog provides, but to me it seems like you’re thinking a bit too complicated…

alias454 · April 18, 2017, 11:19pm

It seems I haven’t checked this page in awhile https://github.com/Graylog2/graylog-guide-syslog-linux/commit/ec34286b3697069ba4bc84f5fd535fbda8712e77

Topic		Replies	Views
[solved] How i get rsyslog messages in graylog? misunderstanding Graylog Central (peer support)	5	9075	February 1, 2019
Graylog 5.0.8 not recieving rsyslog messages Graylog Central (peer support)	4	335	July 14, 2023
Graylog inputs do not seem to be working Graylog Central (peer support)	5	2952	September 7, 2018
Rsyslog and logs from Ubuntu to graylog Graylog Central (peer support)	2	11821	March 26, 2022
Adding Graylog to rSyslog Server Graylog Central (peer support)	3	822	February 9, 2018

Crashing and Burning...rsyslog assistance

Related topics