Well, I have an issue on a new install of Graylog 3.1.(Ubuntu Server 18.04 LTS) I am inputting a single source (Syslog TCP) and it seems to be constantly resetting the connection. After 20-30 minutes of it doing that, the input goes completely non responsive and then just floods the server.log with the following:
2019-12-02T12:22:55.744-06:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5dd6e9e6513b5f1b3607ccf9] (channel [id: 0xc631e548, L:/10.0.0.3:5555 ! R:/18.104.22.168:10874]) (cause io.netty.util.internal.OutOfDirectMemoryError: f
ailed to allocate 16777216 byte(s) of direct memory (used: 1010827264, max: 1020067840))
Restarting the process fixes the issue for the next 20-30 minutes until it happens again. Whatever is out of memory is not a physical out of memory as while in this state I show 3.39G used with 13G free.
It looks like all the dropped connections is the problem but I have a second rsyslog server taking the same stream from the same device (tcp) without a single issue.
Just not sure where to start looking, I am new to Graylog and am still learning. Help!