Graylog dosen't close tcp sessions

[grayman (Lars Andersson)]

I got a problem with our graylog servers. We got two instances, both behave roughly the same.

On the server I can list that a server has say 80+ tcp connections to a client.
On the client I can list that the client has 3 tcp connections to the server.
Clients are windows-server, mixed versions, logging by means of nxlog.

On the server side the amount of sessions steadily increases (aprox evenly redistributed over the clients).
Problem showed around a week ago when one graylog server ran out of network sockets. After 5 days it’s using 14.000 sockets.

netstat -vapnt|grep EST|wc -l
13899
uptime
07:47:26 up 5 days,

OS on server: Latest and greatest Ubuntu 20.04 with all patches.
Graylog: Latest and greatest: graylog-server 4.2.5-1

Nothing special in the logs.

Have anyone else experienced this?

Plenty of workarounds so no immediate problem ( for example rebooting weekly should keeps things dandy ).

We will try to put an identical client on the same subnet as a graylog server to see what happens.
Also some tcpdumping of course.

Just wanted to see if we’re alone with our problem. Found one earlier post regarding similar problem but no answer. XKCD nailed it as usual with this: xkcd: Wisdom of the Ancients

[tmacgbay]

@grayman

xkcd: Wisdom of the Ancients

All. The. Time.

This doesn’t seem normal - my first guess is a NXlog configuration issue… @gsmith is better at NXlog than I am (I use Beats) maybe he has an idea? (I am pretty sure he likes XKCD too…)

[gsmith]

Hello,

@grayman

this: xkcd: Wisdom of the Ancients

Of course

@grayman by chance can you show your nxlog config and/or the input config your using.
I haven’t personally see this but I want to test this out in the lab to see if I get the same results.
By chance do you have IPTABLES enabled or firewalld?

Here is my output with netstat -vapnt|grep EST|wc -l

[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
115
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
113
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
112
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
112
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]# netstat -vapnt|grep EST|wc -l
114
[root@graylog graylog_user]#

EDIT: I Also I found this on GitHub is this what your running into? This is more regarding a TCP keepalive.

TCP connections not closed