Graylog 3.0.1 - io.netty.handler.timeout.ReadTimeoutException

jesse23428 · May 3, 2019, 12:27am

Hello,

I’m getting lots of these errors after a graylog upgrade.
2019-05-02T16:55:18.605-07:00 ERROR [AbstractTcpTransport] Error in Input [GELF HTTP/5cc0dc1ddb8cf7506033f8af] (channel [id: 0xc5381725, L:/192.168.114.221:12201 ! R:/:18011]) (cause io.netty.handler.timeout.ReadTimeoutException)

I upgraded to 3.0.1-2 after I saw this issue raised. Gelf Http input ReadTimeoutException

Version of graylog installed
graylog-server/stable,stable,now 3.0.1-2 all [installed]

I can say that the incidence of the error is an order of magnitude less after the upgrade than before but the symptom still appears.

Any thoughts? Let me know if you need more information…

jan · May 5, 2019, 11:26am

could be an issue with your sender - but without context that is hard to tell. Are the connections keeped open? Does the sender close the connection or just drops?

jesse23428 · May 8, 2019, 1:08am

Thanks for response.

The sender in this case is Serilog.Sinks.Graylog which delegates to System.Net.Http.HttpClient

Client Source Code:

github.com

whir1/serilog-sinks-graylog/blob/master/src/Serilog.Sinks.Graylog.Core/Transport/Http/HttpTransportClient.cs

using System;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using Serilog.Debugging;

namespace Serilog.Sinks.Graylog.Core.Transport.Http
{
    public class HttpTransportClient : ITransportClient<string>
    {
        private readonly string _graylogUrl;
        private readonly HttpClient _httpClient;

        public HttpTransportClient(string graylogUrl)
        {
            _graylogUrl = graylogUrl;
            _httpClient = new HttpClient();
            _httpClient.DefaultRequestHeaders.ExpectContinue = false;
            _httpClient.DefaultRequestHeaders.CacheControl = new CacheControlHeaderValue { NoCache = true };
        }

This file has been truncated. show original

Through inspecting open TCP connections on the graylog server it looks like the connection stays established for a while and then disappear. This disappearance is correlated with the timing of the netty.handler.timeout.ReadTimeoutException

tcp FIN-WAIT-2 0 0 [::ffff:192.168.114.221]:12201 [::ffff:SENDER_IP]:12332
tcp ESTAB 0 0 [::ffff:192.168.114.221]:12201 [::ffff:SENDER_IP]:52538

Hazarding a guess… at a high level… perhaps an exception is being logged when maybe it shouldn’t be. I haven’t noticed any dropped messages because of this behavior though so why show an exception? Confidence about this guess is pretty low though

Maybe a reasonable next step is try to create a simpler reproducible case and go from there… i might pull down the repo and try to do that…

jan · May 8, 2019, 7:02am

that would be really helpful as we currently would not be able to investigate on that in time.

system · May 22, 2019, 7:02am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Gelf Http input ReadTimeoutException Graylog Central (peer support)	7	1296	March 28, 2019
GELF HTTP input issue after upgrading to 3.0 Graylog Central (peer support)	6	3746	March 27, 2019
Graylog's logs show ERROR information: ... cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer) Graylog Central (peer support)	2	4116	November 11, 2019
Getting error when sending data from AWS Lambda Graylog Central (peer support)	6	789	June 30, 2021
Error After 3.0 Upgrade Graylog Central (peer support)	2	884	March 21, 2019

Graylog 3.0.1 - io.netty.handler.timeout.ReadTimeoutException

Related topics