Web Interface issuing TCP_RESETS. Cannot connect

j0m0z · March 7, 2017, 11:23pm

I am running the latest statble 2.2 version of graylog on ubuntu 16.04

I have a new install that seems to have come up and is running but connecting to the web interface I get a tcp reset. Its driving me crazy. server.conf:

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = <snip>
root_password_sha2 = <snip>
root_timezone = America/Los_Angeles
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://10.1.255.201:9000/api/
web_listen_uri = http://10.1.255.201:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

jan · March 8, 2017, 6:42am

hej @j0m0z

did you check your Graylog server.log? Did you check if some firewall is blocking the port? Did you check if you can reach by network the IP?

j0m0z · March 8, 2017, 6:01pm

Hi Jan. Thanks for responding!

There are no firewall rules blocking access as it’s an ANY ANY rule to the server from my workstation. Also, I have disabled UFW on the server. I forgot to say that this is a repo install (ver. 2.2.2) on Ubuntu 16.04 server. The graylog server config is pretty much out of the box with just these four settings changed:

password_secret = cut
root_password_sha2 = cut
rest_listen_uri = http://10.1.255.201:9000/api/
web_listen_uri = http://10.1.255.201:9000/

It appears that the resets are coming from the web server. Here is a packet capture of the communication:

126 2.685632275 my_ip 10.1.255.201 TCP 74 60508 → 9000 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=795129447 TSecr=0 WS=128

Frame 126: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) on interface 0
Ethernet II, Src: Dell_87:17:4f (90:b1:1c:87:17:4f), Dst: CiscoInc_ff:fd:90 (00:08:e3:ff:fd:90)
Internet Protocol Version 4, Src: my_ip, Dst: 10.1.255.201
Transmission Control Protocol, Src Port: 60508 (60508), Dst Port: 9000 (9000), Seq: 0, Len: 0

127 2.686250685 10.1.255.201 my_ip TCP 74 9000 → 60508 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=4294956963 TSecr=795129447 WS=128

Frame 127: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) on interface 0
Ethernet II, Src: CiscoInc_ff:fd:90 (00:08:e3:ff:fd:90), Dst: Dell_87:17:4f (90:b1:1c:87:17:4f)
Internet Protocol Version 4, Src: 10.1.255.201, Dst: my_ip
Transmission Control Protocol, Src Port: 9000 (9000), Dst Port: 60508 (60508), Seq: 0, Ack: 1, Len: 0

128 2.686259189 my_ip 10.1.255.201 TCP 66 60508 → 9000 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=795129447 TSecr=4294956963

Frame 128: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) on interface 0
Ethernet II, Src: Dell_87:17:4f (90:b1:1c:87:17:4f), Dst: CiscoInc_ff:fd:90 (00:08:e3:ff:fd:90)
Internet Protocol Version 4, Src: my_ip, Dst: 10.1.255.201
Transmission Control Protocol, Src Port: 60508 (60508), Dst Port: 9000 (9000), Seq: 1, Ack: 1, Len: 0

129 2.686299606 my_ip 10.1.255.201 HTTP 391 GET / HTTP/1.1

Frame 129: 391 bytes on wire (3128 bits), 391 bytes captured (3128 bits) on interface 0
Ethernet II, Src: Dell_87:17:4f (90:b1:1c:87:17:4f), Dst: CiscoInc_ff:fd:90 (00:08:e3:ff:fd:90)
Internet Protocol Version 4, Src: my_ip, Dst: 10.1.255.201
Transmission Control Protocol, Src Port: 60508 (60508), Dst Port: 9000 (9000), Seq: 1, Ack: 1, Len: 325
Hypertext Transfer Protocol

130 2.686726521 10.1.255.201 my_ip TCP 60 9000 → 60508 [RST, ACK] Seq=1 Ack=326 Win=29312 Len=0

Frame 130: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface 0
Ethernet II, Src: CiscoInc_ff:fd:90 (00:08:e3:ff:fd:90), Dst: Dell_87:17:4f (90:b1:1c:87:17:4f)
Internet Protocol Version 4, Src: 10.1.255.201, Dst: my_ip
Transmission Control Protocol, Src Port: 9000 (9000), Dst Port: 60508 (60508), Seq: 1, Ack: 326, Len: 0

As you can see the handshake completes and then the GET request is made with an immediate RST response. I did check the server log for any errors and the server seemed to come up normally. Issuing a netstat -a, I can see that all the listeners are listening on the right ports. I turned on DEBUG for the graylog server and got the following error when trying to connect:

2017-03-08T09:11:38.898-08:00 DEBUG [TCPNIOTransport] TCPNIOConnection (TCPNIOConnection{localSocketAddress={/10.1.255.201:9000}, peerSocketAddress={/my_ip:60562}}) (allocated) read exception
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_121]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_121]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_121]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.8.0_121]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_121]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.readSimpleByteBuffer(TCPNIOUtils.java:345) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.allocateAndReadBuffer(TCPNIOUtils.java:238) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.read(TCPNIOTransport.java:583) [graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:75) [graylog.jar:?]
at org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter.java:173) [graylog.jar:?]
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) [graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) [graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) [graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) [graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) [graylog.jar:?]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) [graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:526) [graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) [graylog.jar:?]
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) [graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) [graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) [graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) [graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) [graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) [graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) [graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2017-03-08T09:11:38.898-08:00 DEBUG [DefaultFilterChain] GRIZZLY0013: Exception during FilterChain execution
java.io.EOFException: null
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.read(TCPNIOTransport.java:597) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:75) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter.java:173) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) [graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) [graylog.jar:?]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) [graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:526) [graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) [graylog.jar:?]
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) [graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) [graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) [graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) [graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) [graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) [graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) [graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

Connection reset by peer? seems either I am confused (most likely) or Java is confused.

jochen · March 8, 2017, 10:24pm

What happens if you try to access the Graylog REST API or the web interface from the same machine they’re running on?

$ curl -i 'http://10.1.255.201:9000/'
$ curl -i -H 'Accept: application/json' 'http://10.1.255.201:9000/api?pretty=true'

j0m0z · March 9, 2017, 10:43am

So I’m feeling pretty dumb. I have an application layer firewall that has a setting (filter) application-default. Even with an ANY ANY policy from my IP, http over port 9000 was being blocked, but blocked in the way a WAF would do it. It was allowing the connection, but when the GET request was made it determined that HTTP was not the default application for port 9000 and sent spoofed resets in both directions. All is working now. Thank you both Jan and jochen for your time and reply’s.

Topic		Replies	Views
New setup for graylog the web interface is not working Graylog Central (peer support)	2	3128	June 21, 2017
Unable to connect to Graylog Web Interface Graylog Central (peer support)	3	10937	November 16, 2018
Graylog server WEB interface not starting Graylog Central (peer support)	5	8787	March 20, 2017
Graylog web interface problem! Graylog Central (peer support)	12	4599	February 19, 2019
Web interface not working all of sudden Graylog Central (peer support)	10	1419	August 31, 2021

Web Interface issuing TCP_RESETS. Cannot connect

Related Topics