Setup of Graylog and NXLog

Graylog Central (peer support)
1

I have set up nxlog for windows 2016 server to send logs to graylog via udp with GELF.
I can clearly see with tcpdump that I am getting messages on port 31338 udp, Graylog is set up to listen to the exact same port for GELF udp. But I get no traffic or messages whatsoever.
I’ve checked the server logs for errors and there aren’t any.
I am using Graylog 3.2.6 on CentOS 7 and NXlog 3.1.2319 on Windows Server 2016

Here is the nxlog config :

Panic Soft
#NoFreeOnExit TRUE

define ROOT C:\Program Files\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf\nxlog.d
define LOGDIR %ROOT%\data

include %CONFDIR%\*.conf
define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data

Module xm_syslog Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 Module xm_exec Module xm_fileop 
# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
    Every   1 hour
    Exec    if (file_exists('%LOGFILE%') and \
               (file_size('%LOGFILE%') >= 5M)) \
                file_cycle('%LOGFILE%', 8);
</Schedule>

# Rotate our log file every week on Sunday at midnight
<Schedule>
    When    @weekly
    Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
</Schedule>
Module xm_gelf Module im_msvistalog Module om_udp Host 10.3.1.202 Port 31338 OutputType GELF Path win=>graylog

Here is the iptables setup :
[root@graylog sysconfig]# cat iptables | grep 31338
-A INPUT -p udp -m state --state NEW -m udp --dport 31338 -j ACCEPT

Here is the tcpdump :

[root@graylog sysconfig]# tcpdump -i ens160 udp port 31338 -vvv -X

70 packets captured
70 packets received by filter

[root@graylog sysconfig]# netstat -tulnp | grep 31338
udp 0 0 0.0.0.0:31338 0.0.0.0:* 1476/java
udp 0 0 0.0.0.0:31338 0.0.0.0:* 1476/java
udp 0 0 0.0.0.0:31338 0.0.0.0:* 1476/java
udp 0 0 0.0.0.0:31338 0.0.0.0:* 1476/java

I have no idea why the logs don’t get ingested.

Any input is welcome!

1 Like
2

I’ll try…
1.) Does the log indicate errors? tail -n50 -f /var/log/graylog-server/server.log
2.) Are the timestamps correct?
3.) What is the order in “Message Processors Configuration” (system → configurations)
4.) Do you have active rules and pipelines or extractors on GELF-input ?

1 Like
3

Hey, alexmercer
Try to check your NXLog config.
Open CMD on your Windows you try to connect.
Type in
“C:\Program Files\nxlog\nxlog.exe” -v

nxlog.exe [OPTIONS]
[-h] print help
[-f] run in foreground, otherwise try to start the nxlog service
[-c conffile] specify an alternate config file
[-i] install service available to service manager
[-u] uninstall service
[-s] stop running nxlog service
[-v] verify configuration file syntax

when you want to apply changes you have to restart the Service

1 Like
4

C:\Program Files\nxlog>nxlog -v
2023-01-11 16:05:22 INFO configuration OK

The config of the nxlog seems to be ok. And also I can clearly see tcpdump getting some messages on port 31338 udp, so something is being sent.

I found this line as a WARN in the server.log :

2023-01-11T16:08:07.319+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=Win10BMW, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=583f07d3-e23c-4297-b4d0-97e883276a4d} (channel [id: 0x87db04ca, L:/0.0.0.0:31338]) should be 262144 but is 425984.

The timestamps are correct.
Changed the buffer size, but still nothing is happening.

The order in the Message Processors Configuration is the following :

|1|AWS Instance Name Lookup|active|
|2|GeoIP Resolver|active|
|3|Pipeline Processor|active|
|4|Message Filter Chain|active|

No pipelines or extractors are configured, I just wanted to start ingesting some logs and then build up from there.

Thank you for the support!

5

stop the input and start an raw/udp input on that port. do you see something?

6

as a test, change it (or deactivate e.g. AWS/GeoIP)
|4|Message Filter Chain|active|
|3|Pipeline Processor|active|
|2|GeoIP Resolver|active
|1|AWS Instance Name Lookup|active|

7

Started a RAW udp plaintext input and i get a WARN again :

2023-01-11T16:56:52.552+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Win10BMW-raw, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=583f07d3-e23c-4297-b4d0-97e883276a4d} (channel [id: 0x3fd859c2, L:/0.0.0.0:31338]) should be 262144 but is 425984.

No messages come through and no network activity is detected in the input.

I have also rearranged the Message Processors and still I don’t see any activity on the input.

8

Does another logging work?
Does the index increase? curl -X GET http://127.0.0.1:9200/_cat/indices?pretty=true

For iptables, it’s better you match of already existing sessions: iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT before it’s as new session.

9

I don’t have experience with the buffer sizes of the inputs, but did you try to put in the values of the error message?

Another Idea: how full are your buffers? If you click the number of in/out logs on the top right and then on your node you will see three buffers: input, processing, and output. how full are they?

10

Hey @alexmercer

That can be a OS issue.

Example:

Check the current UDP/IP receive buffer default and limit by typing the following commands:

$ sysctl net.core.rmem_max
net.core.rmem_max = 212992
$ sysctl net.core.rmem_default
net.core.rmem_default = 212992

If the values are less than 26214400 bytes (25MB) you should add the following lines to the /etc/sysctl.conf file:

net.core.rmem_max=26214400
net.core.rmem_default=26214400

Changes to /etc/sysctl.conf do not take effect until reboot. To update the values , type the following commands as root:

$ sudo sysctl -w net.core.rmem_max=26214400
net.core.rmem_max = 26214400
$ sudo sysctl -w net.core.rmem_default=26214400
net.core.rmem_default = 26214400

Execute

Root #  sysctl -p

Restart Graylog Server

root # systemctl restart  graylog-server

After looking over your NXlog configuration I see route is missing.
Nxlog.conf Should look something like this:

define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define LOGFILE C:\Program Files (x86)\nxlog\data\nxlog.log

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data

### This is for Nxlog syslog file. 

LogFile %ROOT%\data\nxlog.log
#LogLevel DEBUG <---Only enable Debug mode to resolve issue the  comment it back out. this will create a large file over time.


### This is for rotating Nxlog log file.
<Extension _fileop>
    Module xm_fileop
    # Check the log file size every hour and rotate if larger than 5 MB
    <Schedule>
        Every 1 hour
        <Exec>
            if (file_exists('%LOGFILE%') and file_size('%LOGFILE%') >= 5M)
                file_cycle('%LOGFILE%', 8);
        </Exec>
    </Schedule>
    # Rotate log file every week on Sunday at midnight
    <Schedule>
        When    @weekly
        Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
    </Schedule>
</Extension>

### Extentions for file types.

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Extension _json>
    Module      xm_json
</Extension>

<Extension gelf>
    Module      xm_gelf
 </Extension>
 

<Input input>
    Module      im_msvistalog    
</Input>

<Output out>
    Module      om_udp
    Host        10.3.1.202 
    Port        31338  
    OutputType  GELF 
    Exec  $Message = $raw_event;	
</Output>

<Route >
    Path        input => out
</Route>
11

Thank you for the replies!

Indices are getting bigger because of all the other inputs and I have noticed that other inputs are also alerting about the buffer size :

2023-01-12T10:18:12.306+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=BS HQ pfSense Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=583f07d3-e23c-4297-b4d0-97e883276a4d} (channel [id: 0xb439df97, L:/0.0.0.0:31330]) should be 262144 but is 524288.

The buffers also indicate that they are free - there is not a lot of activity in the other inputs : I average about 50msgs/s and it all gets cleared quickly.

This is so weird, whenever windows is involved something doesn’t work …

I have also changed the net.core.rmem_max and default values, but nothing changed in the behaviour of the system.

12

Hey @alexmercer

Thats just a warning, Graylog just yelling about the OS system config.
You can look at you NXlog logs on the Window server 2016 maybe that might help.
So is the issue your not see the logs on the GUI? Perhaps check your timezone/Dat & time on both GL server and Windows 2016 server. just an idea.

Here ar ecouple links i forgot to add earlier for anyone having this issue.

13

Time and date match exactly, both on Graylog and the Windows machine. I know NXlog is sedning something, because I can see it in the tcpdump command when listening on the same port the input is set up. I don’t get why whatever is sent is not parsed by Graylog, could it be the wrong format and it’s being disregarded as gibberish ? I am shooting in the dark at this point.

14

Hello,

Not sure, but the nxlog configuration you posted in the beganing did not look correct. I was wondering if you could post the recent nxlog configuration and the INPUT used. If so, please use the Markdown for log and configuration files. It makes it easy to detect issues.

TCPDUMP shows that “remote device” is sending logs/message to Graylog server on port 31338 but the message perhaps are not being ingested by Elasticsearch/Opensearch. Just taking a guess couple things I know of that prevents messages showing on the GUI

1.ES/OS,
2.timestamp,
3.service blocking. ( firewall, SELinux, etc…)
4.file types (i.e, log file sent to the wrong type of INPUT)

Check elasticsearch log files.

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.