WARN o.g.g.n.t.TCPNIOTransport [grizzly-nio-kernel(1) SelectorRunner] GRIZZLY0004: Can not set TCP_NODELAY to true

benstiller · October 30, 2021, 2:53pm

Description of your problem

Hi all,

My server.log file is filled with these messages about “GRIZZLY0004: Can not set TCP_NODELAY to true”. What’s wrong and how can I correct?

Description of steps you’ve taken to attempt to solve the issue

Environmental information

Operating system information

FreeBSD 12.2-RELEASE-p10, running inside a FreeBSD iocage jail

Package versions

Graylog 4.1.5
MongoDB 4.0.27
Elasticsearch 7.15.1
openjdk11-11.0.12+7.1_1

2021-10-30 10:49:32,633 WARN o.g.g.n.t.TCPNIOTransport [grizzly-nio-kernel(1) SelectorRunner] GRIZZLY0004: Can not set TCP_NODELAY to true

java.net.SocketException: Connection reset by peer

at sun.nio.ch.Net.setIntOption0(Native Method) ~[?:1.8.0_302]

at sun.nio.ch.Net.setSocketOption(Net.java:353) ~[?:1.8.0_302]

at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:188) ~[?:1.8.0_302]

at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:287) ~[?:1.8.0_302]

at sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:322) ~[?:1.8.0_302]

at org.glassfish.grizzly.nio.transport.TCPNIOTransport$DefaultChannelConfigurator.postConfigure(TCPNIOTransport.java:792) [graylog.jar:?]

at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.configureAcceptedChannel(TCPNIOServerConnection.java:161) [graylog.jar:?]

at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.onAccept(TCPNIOServerConnection.java:221) [graylog.jar:?]

at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:485) [graylog.jar:?]

at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) [graylog.jar:?]

at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:79) [graylog.jar:?]

at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) [graylog.jar:?]

at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391) [graylog.jar:?]

at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360) [graylog.jar:?]

at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324) [graylog.jar:?]

at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255) [graylog.jar:?]

at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [graylog.jar:?]

at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [graylog.jar:?]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]

gsmith · November 1, 2021, 10:37pm

Hello && Welcome

First thing I noticed was this.

Have you seen this?

What documentation did you use to install Graylog?
Second, I’m not sure but are you using a proxy server?

I also found these.

https://download.oracle.com/javaee-archive/grizzly.java.net/users/2014/06/6330.html
https://bugster.forgerock.org/jira/browse/OPENDJ-3199

benstiller · November 2, 2021, 5:31pm

Hi @gsmith,

Thanks for the note about the Elasticsearch version. I hadn’t seen that.

This is an older instance, and I upgraded from ES6 to 7 months ago. I didn’t go back and read the new install doc page, but I see that’s where the screenshot is from. Interestingly the upgrade page in the docs, doesn’t mention the ES version.

To your questions:

I installed this a couple years ago using some instructions about installing Graylog in a FreeBSD jail. I believe it was this link: Setting up Graylog in a FreeNAS jail – sign13
Progressive updates to Elasticsearch and all other installed packages are just from doing ‘pkg upgrade’ occasionally. The ES6 to 7 upgrade was manual from a few months ago.
I am running graylog behind an haproxy reverse-proxy on opnsense (FreeBSD). Additionally it’s using SSL-passthrough, if that helps.

I’ve been getting these messages for several months. Prior to when I upgraded from ES 6 to 7, although I don’t recall exactly those dates, so I don’t suspect that is the cause of this specific error (but maybe others to come – we’ll see!).

Thank you for those links. It seems back in 2014, there was some investigation of the JDK as the source of the problem, but no conclusions. See RE: GRIZZLY0004 and GRIZZLY0005 continuous errors

Any advice appreciated on troubleshooting with haproxy in front of graylog, if that could be the cause

Kind regards, Ben

gsmith · November 2, 2021, 9:44pm

Hello,

My knowledge is limited on haproxy but I can look into it. Maybe find something and post it here.
EDIT: showing the full log file would help greatly. What you have showed could be a couple different issues. When posting configuration files or code please use the markdown. This will help spot small details to resolve your issue. The HowTo can be found here.

Service logs, configuration, and environment variables

benstiller · November 6, 2021, 11:59pm

Hi @gsmith ,

Here’s some additional server.log entries, repeating the same error. If there’s a way to export the values from my graylog.conf I can add that also. I couldn’t find an easy way to do that however.

Thank you!

server.log

2021-11-06 19:55:38,948 WARN o.g.g.n.t.TCPNIOTransport [grizzly-nio-kernel(1) SelectorRunner] GRIZZLY0004: Can not set TCP_NODELAY to true
java.net.SocketException: Connection reset by peer
	at sun.nio.ch.Net.setIntOption0(Native Method) ~[?:1.8.0_312]
	at sun.nio.ch.Net.setSocketOption(Net.java:353) ~[?:1.8.0_312]
	at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:188) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:287) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:322) ~[?:1.8.0_312]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport$DefaultChannelConfigurator.postConfigure(TCPNIOTransport.java:792) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.configureAcceptedChannel(TCPNIOServerConnection.java:161) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.onAccept(TCPNIOServerConnection.java:221) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:485) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:79) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2021-11-06 19:55:50,960 WARN o.g.g.n.t.TCPNIOTransport [grizzly-nio-kernel(1) SelectorRunner] GRIZZLY0004: Can not set TCP_NODELAY to true
java.net.SocketException: Connection reset by peer
	at sun.nio.ch.Net.setIntOption0(Native Method) ~[?:1.8.0_312]
	at sun.nio.ch.Net.setSocketOption(Net.java:353) ~[?:1.8.0_312]
	at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:188) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:287) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:322) ~[?:1.8.0_312]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport$DefaultChannelConfigurator.postConfigure(TCPNIOTransport.java:792) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.configureAcceptedChannel(TCPNIOServerConnection.java:161) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.onAccept(TCPNIOServerConnection.java:221) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:485) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:79) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2021-11-06 19:56:00,970 WARN o.g.g.n.t.TCPNIOTransport [grizzly-nio-kernel(1) SelectorRunner] GRIZZLY0004: Can not set TCP_NODELAY to true
java.net.SocketException: Connection reset by peer
	at sun.nio.ch.Net.setIntOption0(Native Method) ~[?:1.8.0_312]
	at sun.nio.ch.Net.setSocketOption(Net.java:353) ~[?:1.8.0_312]
	at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:188) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:287) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:322) ~[?:1.8.0_312]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport$DefaultChannelConfigurator.postConfigure(TCPNIOTransport.java:792) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.configureAcceptedChannel(TCPNIOServerConnection.java:161) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.onAccept(TCPNIOServerConnection.java:221) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:485) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:79) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2021-11-06 19:56:08,978 WARN o.g.g.n.t.TCPNIOTransport [grizzly-nio-kernel(1) SelectorRunner] GRIZZLY0004: Can not set TCP_NODELAY to true
java.net.SocketException: Connection reset by peer
	at sun.nio.ch.Net.setIntOption0(Native Method) ~[?:1.8.0_312]
	at sun.nio.ch.Net.setSocketOption(Net.java:353) ~[?:1.8.0_312]
	at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:188) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:287) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:322) ~[?:1.8.0_312]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport$DefaultChannelConfigurator.postConfigure(TCPNIOTransport.java:792) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.configureAcceptedChannel(TCPNIOServerConnection.java:161) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.onAccept(TCPNIOServerConnection.java:221) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:485) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:79) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2021-11-06 19:56:43,021 WARN o.g.g.n.t.TCPNIOTransport [grizzly-nio-kernel(1) SelectorRunner] GRIZZLY0004: Can not set TCP_NODELAY to true
java.net.SocketException: Connection reset by peer
	at sun.nio.ch.Net.setIntOption0(Native Method) ~[?:1.8.0_312]
	at sun.nio.ch.Net.setSocketOption(Net.java:353) ~[?:1.8.0_312]
	at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:188) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:287) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:322) ~[?:1.8.0_312]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport$DefaultChannelConfigurator.postConfigure(TCPNIOTransport.java:792) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.configureAcceptedChannel(TCPNIOServerConnection.java:161) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.onAccept(TCPNIOServerConnection.java:221) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:485) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:79) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2021-11-06 19:56:51,029 WARN o.g.g.n.t.TCPNIOTransport [grizzly-nio-kernel(1) SelectorRunner] GRIZZLY0004: Can not set TCP_NODELAY to true
java.net.SocketException: Connection reset by peer
	at sun.nio.ch.Net.setIntOption0(Native Method) ~[?:1.8.0_312]
	at sun.nio.ch.Net.setSocketOption(Net.java:353) ~[?:1.8.0_312]
	at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:188) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:287) ~[?:1.8.0_312]
	at sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:322) ~[?:1.8.0_312]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport$DefaultChannelConfigurator.postConfigure(TCPNIOTransport.java:792) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.configureAcceptedChannel(TCPNIOServerConnection.java:161) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.onAccept(TCPNIOServerConnection.java:221) [graylog.jar:?]
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:485) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:79) [graylog.jar:?]
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324) [graylog.jar:?]
	at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [graylog.jar:?]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]

tmacgbay · November 7, 2021, 2:54pm

grep -v "^#\|^$" /etc/graylog/server/server.conf

Make sure to use </> and obfuscate where needed when posting.

I googled and found at this link here granted it is not Graylog and refers to windows but it explains the OJDK issue pretty well and offers what might be a relevant solution to keeping it from filling logs. This could be either a java or FreeBSD or combination error but it looks like by my search it it just a log annoyance and can be shut off by modifying the log4j2.xml file?

Caveat that is google talking, not personal experience.

benstiller · November 7, 2021, 5:26pm

Thanks @tmacgbay

I added this line to the log4j2.xml file

<AsyncLogger level="ERROR" name="org.glassfish.grizzly.nio.transport.TCPNIOTransport"/>

And the errors are no longer filling the log. Yay!

I realize doesn’t actually correct the underlying issue, but as long as it’s just an annoyance message as indicated, this will do for now.

For others with this issue, note that the line above must be inside the

<Loggers>

</Loggers>

section of the log4j2.xml file.

This issue can be closed.

system · November 21, 2021, 5:26pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
[GelfTcpTransport] Channel disconnected! Graylog Central (peer support)	2	430	December 30, 2020
Graylog's logs show ERROR information: ... cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer) Graylog Central (peer support)	2	4104	November 11, 2019
Graylog unable to startup Graylog Central (peer support)	7	3329	January 7, 2018
Error After 3.0 Upgrade Graylog Central (peer support)	2	884	March 21, 2019
Startup issue - address already in use Graylog Central (peer support)	2	2583	November 30, 2018