Error in Input causing logs to drop

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

I’m losing syslog records to a recurring error on all 3 of my graylog hosts, when running with SSL I can only see the following about every couple seconds:


ERROR [AbstractTcpTransport] Error in Input [] (channel [id:, L: ! R:]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)

2. Describe your environment:

  • OS Information:
    Rocky Linux release 8.10 (Green Obsidian) across all 11 hosts
    Epyc 7543P, 64GB ram
    os on 2x 480GB SDD R1
    opensearch db on 4x 960GB SSD in R10 on opensearch nodes

  • Package Version:

Graylog 6.1.7+6ec0bac on graylog-01 (Eclipse Adoptium 17.0.14 on Linux 4.18.0-553.36.1.el8_10.x86_64)

{
  "name" : "opensrch-01",
  "cluster_name" : "graylog-search",
  "cluster_uuid" : "",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.18.0",
    "build_type" : "rpm",
    "build_hash" : "99a9a81da366173b0c2b963b26ea92e15ef34547",
    "build_date" : "2024-10-31T19:11:45.959566657Z",
    "build_snapshot" : false,
    "lucene_version" : "9.12.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

HAProxy version 1.8.27-493ce0b, released 2020/11/06

(step 2 follows as I blew the character limit )

3. What steps have you already taken to try and solve the problem?

I have deleted and recreated the inputs, I have created them as syslog tcp, as well as raw tcp plaintext. I have attempted them with and without SSL.

No configuration resolves the issue.

I have applied various tuning as recommended in the documentation to no avail

I have researched the errors and they all seem to point to packages used by varied product (salesforce, graylog and a couple others) all seeing the same issue. I have also seen the same issue on this forum, without a solution as they get left unanswered (some with either a magical “I fixed it” or “this isn’t really an issue”)

4. How can the community help?

At this point I’m stuck- I’m not actually seeing the errors anywhere except the logs and lack of records, I’m not seeing weird issues with other processes, nor anything I can find with opensearch to graylog links (which I would think would be exhibiting the exact same issues with tcp resets dropping active connections). I think this is an issue with one of the classes dropping an invalid error, but I have no way of narrowing it down.

Any help would be greatly appreciated, and I apologize for the huge first post. I truly am stuck.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

error as seen without TLS

2025-02-06T20:57:13.521-08:00 WARN  [DefaultFilterChain] GRIZZLY0013: Exception during FilterChain execution
java.lang.IllegalStateException: Unknown protocol cl02-fe-053 ipachecker 2379404 - - Checking IPA sync status with ipamon.
        at org.glassfish.grizzly.http.Protocol.valueOf(Protocol.java:80) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpHeader.getProtocol(HttpHeader.java:765) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.prepareResponse(HttpServerFilter.java:751) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.encodeHttpPacket(HttpServerFilter.java:723) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.commitAndCloseAsError(HttpServerFilter.java:1053) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.sendBadRequestResponse(HttpServerFilter.java:1045) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.onHttpHeaderError(HttpServerFilter.java:690) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpCodecFilter.java:524) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.handleRead(HttpServerFilter.java:269) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118) [graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96) [graylog.jar:?]
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51) [graylog.jar:?]
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510) [graylog.jar:?]
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82) [graylog.jar:?]
        at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:69) [graylog.jar:?]
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:381) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:353) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:319) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:248) [graylog.jar:?]
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) [graylog.jar:?]
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) [graylog.jar:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2025-02-06T20:57:13.868-08:00 WARN  [DefaultFilterChain] GRIZZLY0013: Exception during FilterChain execution
java.lang.IllegalStateException: Unknown protocol ldap-04 ns-slapd - - - GSSAPI server step 3
        at org.glassfish.grizzly.http.Protocol.valueOf(Protocol.java:80) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpHeader.getProtocol(HttpHeader.java:765) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.prepareResponse(HttpServerFilter.java:751) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.encodeHttpPacket(HttpServerFilter.java:723) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.commitAndCloseAsError(HttpServerFilter.java:1053) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.sendBadRequestResponse(HttpServerFilter.java:1045) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.onHttpHeaderError(HttpServerFilter.java:690) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpCodecFilter.java:524) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.handleRead(HttpServerFilter.java:269) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118) [graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96) [graylog.jar:?]
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51) [graylog.jar:?]
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510) [graylog.jar:?]
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82) [graylog.jar:?]
        at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:69) [graylog.jar:?]
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:381) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:353) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:319) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:248) [graylog.jar:?]
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) [graylog.jar:?]
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) [graylog.jar:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2025-02-06T20:57:13.869-08:00 WARN  [DefaultFilterChain] GRIZZLY0013: Exception during FilterChain execution
java.lang.IllegalStateException: Unknown protocol ldap-02 rsyslogd - - - action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2359 ]
        at org.glassfish.grizzly.http.Protocol.valueOf(Protocol.java:80) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpHeader.getProtocol(HttpHeader.java:765) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.prepareResponse(HttpServerFilter.java:751) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.encodeHttpPacket(HttpServerFilter.java:723) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.commitAndCloseAsError(HttpServerFilter.java:1053) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.sendBadRequestResponse(HttpServerFilter.java:1045) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.onHttpHeaderError(HttpServerFilter.java:690) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpCodecFilter.java:524) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.HttpServerFilter.handleRead(HttpServerFilter.java:269) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118) [graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96) [graylog.jar:?]
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51) [graylog.jar:?]
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510) [graylog.jar:?]
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82) [graylog.jar:?]
        at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:69) [graylog.jar:?]
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:381) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:353) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:319) [graylog.jar:?]
        at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:248) [graylog.jar:?]
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) [graylog.jar:?]
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) [graylog.jar:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]

graylog config:

# /etc/graylog/server.conf
is_leader = true

node_id_file = /etc/graylog/server/node-id

password_secret = <screened>
root_username = <screened>
root_password_sha2 = <screened> 

root_email = 

root_timezone = UTC

bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin

####################
# HTTP/REST settings
####################

http_bind_address = graylog-01:9000
## The following default to http_bind_address:
#http_publish_uri = http://graylog-01:9000/
#http_external_uri = http://graylog:9000/
#rest_listen_uri = http://graylog-01:9000/api/
#web_listen_uri = http://graylog-01:9000/
#
http_enable_cors = True
http_enable_gzip = True
http_enable_tls = True

http_tls_cert_file = /usr/share/graylog-server/tls/ssl.cer

http_tls_key_file = /usr/share/graylog-server/tls/ssl.key

###http_tls_key_password = 
#
http_max_header_size = 8192
http_thread_pool_size = 16

#####################
# OpenSearch settings
#####################

# must use elasticsearch_hosts here, not opensearch_hosts, at least with
# graylog-5.1
elasticsearch_hosts = http://opensrch-01:9200,http://opensrch-02:9200,http://opensrch-03:9200,http://opensrch-04:9200,http://opensrch-05:9200,http://opensrch-06:9200

opensearch_connect_timeout = 10s
opensearch_socket_timeout = 60s
#opensearch_idle_timeout = 
opensearch_max_total_connections = 20
opensearch_max_total_connections_per_route = 2
opensearch_max_retries = 2
opensearch_discovery_enabled = False
opensearch_discovery_filter = 
opensearch_discovery_frequency = 30s
opensearch_compression_enabled = False
rotation_strategy = count
opensearch_max_docs_per_index = 20000000
opensearch_max_size_per_index = 1073741824
opensearch_max_time_per_index = 1d
opensearch_disable_version_check = True
no_retention = False
opensearch_max_number_of_indices = 20
retention_strategy = delete
opensearch_index_prefix = graylog
opensearch_template_name = graylog-internal
allow_leading_wildcard_searches = False
allow_highlighting = False
opensearch_analyzer = standard
opensearch_request_timeout = 1m
opensearch_index_optimization_timeout = 1h
opensearch_index_optimization_jobs = 20
index_ranges_cleanup_interval = 1h
index_field_type_periodical_interval = 1h
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
outputbuffer_processor_keep_alive_time = 5000
outputbuffer_processor_threads_core_pool_size = 3
outputbuffer_processor_threads_max_pool_size = 30
udp_recvbuffer_sizes = 1048576
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking

# Message journal settings

message_journal_enabled = True
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 12h
message_journal_max_size = 5gb
message_journal_flush_age = 1m
message_journal_flush_interval = 1000000
message_journal_segment_age = 1h
message_journal_segment_size = 100mb

async_eventbus_processors = 2
lb_recognition_period_seconds = 3
lb_throttle_threshold_percentage = 95

stream_processing_timeout = 2000
stream_processing_max_faults = 3
alert_check_interval = 60
output_module_timeout = 10000
stale_master_timeout = 2000
shutdown_timeout = 30000

##################
# MongoDB Settings
##################
mongodb_uri = mongodb://<screened>:<screened>@graylog-01:27017,graylog-02:27017,graylog-03:27017/graylog?replicaSet=rs0
mongodb_max_connections = 100
mongodb_threads_allowed_to_block_multiplier = 5
# /etc/graylog/server.conf
is_leader = true

node_id_file = /etc/graylog/server/node-id

password_secret = <screened>
root_username = <screened>
root_password_sha2 = <screened> 

root_email = 

root_timezone = UTC

bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin

####################
# HTTP/REST settings
####################

http_bind_address = graylog-01:9000
## The following default to http_bind_address:
#http_publish_uri = http://graylog-01:9000/
#http_external_uri = http://graylog:9000/
#rest_listen_uri = http://graylog-01:9000/api/
#web_listen_uri = http://graylog-01:9000/
#
http_enable_cors = True
http_enable_gzip = True
http_enable_tls = True

http_tls_cert_file = /usr/share/graylog-server/tls/ssl.cer

http_tls_key_file = /usr/share/graylog-server/tls/ssl.key

###http_tls_key_password = 
#
http_max_header_size = 8192
http_thread_pool_size = 16

#####################
# OpenSearch settings
#####################

# must use elasticsearch_hosts here, not opensearch_hosts, at least with
# graylog-5.1
elasticsearch_hosts = http://opensrch-01:9200,http://opensrch-02:9200,http://opensrch-03:9200,http://opensrch-04:9200,http://opensrch-05:9200,http://opensrch-06:9200

opensearch_connect_timeout = 10s
opensearch_socket_timeout = 60s
#opensearch_idle_timeout = 
opensearch_max_total_connections = 20
opensearch_max_total_connections_per_route = 2
opensearch_max_retries = 2
opensearch_discovery_enabled = False
opensearch_discovery_filter = 
opensearch_discovery_frequency = 30s
opensearch_compression_enabled = False
rotation_strategy = count
opensearch_max_docs_per_index = 20000000
opensearch_max_size_per_index = 1073741824
opensearch_max_time_per_index = 1d
opensearch_disable_version_check = True
no_retention = False
opensearch_max_number_of_indices = 20
retention_strategy = delete
opensearch_index_prefix = graylog
opensearch_template_name = graylog-internal
allow_leading_wildcard_searches = False
allow_highlighting = False
opensearch_analyzer = standard
opensearch_request_timeout = 1m
opensearch_index_optimization_timeout = 1h
opensearch_index_optimization_jobs = 20
index_ranges_cleanup_interval = 1h
index_field_type_periodical_interval = 1h
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
outputbuffer_processor_keep_alive_time = 5000
outputbuffer_processor_threads_core_pool_size = 3
outputbuffer_processor_threads_max_pool_size = 30
udp_recvbuffer_sizes = 1048576
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking

# Message journal settings

message_journal_enabled = True
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 12h
message_journal_max_size = 5gb
message_journal_flush_age = 1m
message_journal_flush_interval = 1000000
message_journal_segment_age = 1h
message_journal_segment_size = 100mb

async_eventbus_processors = 2
lb_recognition_period_seconds = 3
lb_throttle_threshold_percentage = 95

stream_processing_timeout = 2000
stream_processing_max_faults = 3
alert_check_interval = 60
output_module_timeout = 10000
stale_master_timeout = 2000
shutdown_timeout = 30000

##################
# MongoDB Settings
##################
mongodb_uri = mongodb://<screened>:<screened>@graylog-01:27017,graylog-02:27017,graylog-03:27017/graylog?replicaSet=rs0
mongodb_max_connections = 100
mongodb_threads_allowed_to_block_multiplier = 5
capturing only traffic from the local haproxy logs to specific graylog host:

tail -f /var/log/haproxy-traffic.2.log | grep "232\:3514" | tee ~/haproxy-01.log
output:

Feb 25 09:28:21 haproxy-01 haproxy[241398]: c.c.c.59:57068 h.h.h.235:56826 g.g.g.232:3514 [25/Feb/2025:09:25:52.369] hasyslogfe hasyslogbackend/graylog-01 1/21/148648 0 cD 29/24/23/7/0 0/0
Feb 25 09:28:56 haproxy-01 haproxy[241398]: c.c.c.15:59338 h.h.h.235:45438 g.g.g.232:3514 [25/Feb/2025:09:27:36.054] hasyslogfe hasyslogbackend/graylog-01 1/21/80003 0 cD 31/23/22/7/0 0/0
Feb 25 09:29:01 haproxy-01 haproxy[241398]: c.c.c.21:43230 h.h.h.235:45478 g.g.g.232:3514 [25/Feb/2025:09:25:43.484] hasyslogfe hasyslogbackend/graylog-01 1/21/198069 0 cD 30/23/22/7/0 0/0
Feb 25 09:29:26 haproxy-01 haproxy[241398]: c.c.c.12:47558 h.h.h.235:56670 g.g.g.232:3514 [25/Feb/2025:09:27:49.687] hasyslogfe hasyslogbackend/graylog-01 1/21/96655 0 cD 30/20/19/7/0 0/0
Feb 25 09:29:31 haproxy-01 haproxy[241398]: c.c.c.36:54730 h.h.h.235:54880 g.g.g.232:3514 [25/Feb/2025:09:27:28.034] hasyslogfe hasyslogbackend/graylog-01 1/21/123356 0 cD 28/18/17/6/0 0/0
Feb 25 09:29:49 haproxy-01 haproxy[241398]: c.c.c.26:39112 h.h.h.235:50848 g.g.g.232:3514 [25/Feb/2025:09:28:38.858] hasyslogfe hasyslogbackend/graylog-01 1/21/71090 0 cD 29/17/16/5/0 0/0
Feb 25 09:29:52 haproxy-01 haproxy[241398]: c.c.c.3:49218 h.h.h.235:55704 g.g.g.232:3514 [25/Feb/2025:09:26:38.548] hasyslogfe hasyslogbackend/graylog-01 1/21/194250 0 cD 29/18/17/5/0 0/0
Feb 25 09:29:58 haproxy-01 haproxy[241398]: c.c.c.68:35480 h.h.h.235:41598 g.g.g.232:3514 [25/Feb/2025:09:28:58.402] hasyslogfe hasyslogbackend/graylog-01 1/21/60340 0 cD 28/17/16/5/0 0/0
Feb 25 09:30:02 haproxy-01 haproxy[241398]: c.c.c.14:37944 h.h.h.235:51514 g.g.g.232:3514 [25/Feb/2025:09:28:01.863] hasyslogfe hasyslogbackend/graylog-01 1/21/120286 0 sD 30/17/16/4/0 0/0
Feb 25 09:30:02 haproxy-01 haproxy[241398]: c.c.c.21:42262 h.h.h.235:41626 g.g.g.232:3514 [25/Feb/2025:09:29:01.790] hasyslogfe hasyslogbackend/graylog-01 1/21/60501 0 cD 29/16/15/3/0 0/0
Feb 25 09:31:02 haproxy-01 haproxy[241398]: c.c.c.82:41442 h.h.h.235:56206 g.g.g.232:3514 [25/Feb/2025:09:26:02.157] hasyslogfe hasyslogbackend/graylog-01 1/21/299879 0 cD 44/26/25/8/0 0/0
Feb 25 09:31:06 haproxy-01 haproxy[241398]: c.c.c.3:53872 h.h.h.235:52292 g.g.g.232:3514 [25/Feb/2025:09:30:05.776] hasyslogfe hasyslogbackend/graylog-01 1/20/60274 0 cD 42/25/24/7/0 0/0
Feb 25 09:31:08 haproxy-01 haproxy[241398]: c.c.c.12:37218 h.h.h.235:36526 g.g.g.232:3514 [25/Feb/2025:09:29:52.389] hasyslogfe hasyslogbackend/graylog-01 1/21/75906 0 cD 41/24/23/6/0 0/0
Feb 25 09:31:54 haproxy-01 haproxy[241398]: c.c.c.37:49584 h.h.h.235:52466 g.g.g.232:3514 [25/Feb/2025:09:30:54.199] hasyslogfe hasyslogbackend/graylog-01 1/20/60491 0 cD 34/22/21/7/0 0/0
Feb 25 09:32:02 haproxy-01 haproxy[241398]: c.c.c.68:39182 h.h.h.235:44348 g.g.g.232:3514 [25/Feb/2025:09:30:14.443] hasyslogfe hasyslogbackend/graylog-01 1/21/107969 0 cD 31/20/19/6/0 0/0
Feb 25 09:32:04 haproxy-01 haproxy[241398]: c.c.c.59:58698 h.h.h.235:48388 g.g.g.232:3514 [25/Feb/2025:09:30:04.015] hasyslogfe hasyslogbackend/graylog-01 1/21/120000 0 cD 30/19/18/5/0 0/0
Feb 25 09:32:13 haproxy-01 haproxy[241398]: c.c.c.22:36812 h.h.h.235:48380 g.g.g.232:3514 [25/Feb/2025:09:30:03.786] hasyslogfe hasyslogbackend/graylog-01 1/21/129752 0 cD 30/18/17/5/0 0/0
Feb 25 09:32:19 haproxy-01 haproxy[241398]: c.c.c.93:37412 h.h.h.235:48334 g.g.g.232:3514 [25/Feb/2025:09:29:55.842] hasyslogfe hasyslogbackend/graylog-01 1/21/143614 0 cD 27/15/14/4/0 0/0
#----------------------------------------------------------------------
# Global settings
#----------------------------------------------------------------------
global
    log h.h.h.235:514 local2

    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    maxconn 3000
    user haproxy
    group haproxy
    daemon

    stats socket /var/lib/haproxy/stats level admin

    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
    ssl-server-verify none
    ssl-dh-param-file /etc/haproxy/dhparams.pem

#----------------------------------------------------------------------
# Common defaults
#----------------------------------------------------------------------
defaults
    mode    http
    log     global
    option  httplog
    option  tcpka
    option  dontlognull
    option  http-keep-alive
    option  forwardfor  except 127.0.0.0/8
    retries 3
    timeout http-request 30s
    timeout queue 1m
    timeout connect 30s
    timeout client 1m
    timeout server 1m
    timeout http-keep-alive 30s
    timeout check 30s
    maxconn 3000
    log-format "%ci:%cp %bi:%bp %si:%sp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"

#----------------------------------------------------------------------
# Main frontend which proxys web traffic to backends
#----------------------------------------------------------------------

frontend stats
    mode http
    no log
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if TRUE

frontend graylog
    bind *:80
    bind *:443 ssl crt /etc/haproxy/ssl.full
    mode http
#    http-request add-header X-Forwarded-Host %[req.hdr(host)]
#    http-request add-header X-Forwarded-Server %[req.hdr(host)]
#    http-request add-header X-Forwarded-Port %[dst_port]
    http-request redirect scheme https unless { ssl_fc }
#    acl is_graylog hdr_dom(host) -i -m str graylog
    default_backend habackend

#----------------------------------------------------------------------
# Roundrobin balancing http traffic between the various backends
#----------------------------------------------------------------------
backend habackend
    balance roundrobin
    mode http
    stick-table type ip size 1m expire 1d
    stick on src
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    http-request set-header X-Graylog-Server-URL https://graylog/
    cookie SERVERID insert indirect
    server graylog-01 graylog-01:9000 check ssl verify none
    server graylog-02 graylog-02:9000 check ssl verify none
    server graylog-03 graylog-03:9000 check ssl verify none

#----------------------------------------------------------------------
# graylog listener
#----------------------------------------------------------------------
listen syslogs
    bind :3514
    mode tcp
#    option tcplog
    balance roundrobin
    option httpchk GET /system/lbstatus
    server graylog-01 graylog-01:9000 check ssl verify none
    server graylog-02 graylog-02:9000 check ssl verify none
    server graylog-03 graylog-03:9000 check ssl verify none

#----------------------------------------------------------------------
# Syslog frontend for passing logs to backends
#----------------------------------------------------------------------
frontend hasyslogfe
    bind :3514
    mode tcp
#    option tcplog
    default_backend hasyslogbackend

#----------------------------------------------------------------------
# Roundrobin balancing rsyslog traffic between the various backends
#----------------------------------------------------------------------
backend hasyslogbackend
    balance leastconn
    mode tcp
#    stick-table type ip size 1m expire 1d
#    stick on src
    server graylog-01 graylog-01:3514 check ssl verify none
    server graylog-02 graylog-02:3514 check ssl verify none
    server graylog-03 graylog-03:3514 check ssl verify none

Changed timeouts in haproxy to 10m, changed loadbalancing method from round robin to least connection - no change.

Disabled ssl for the input both in haproxy and graylog - errors gone.

Changed timeouts back to 30s and loadbalancing to roundrobin, errors return.

Changed loadbalancing back to round robin and timeouts to 10m, errors remain.

Changing loadbalancing to least connection, and timeouts to 30s, errors remain.

For only 80 hosts, I shouldn’t be seeing this- but it appears my solution for now is to disable ssl for the inputs, keep timeouts above 30s (I haven’t tested anything other than 10m and 30s however) and use least connection for the load balancing.

error referenced in the topic:

2025-02-26T02:24:29.366-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0xfc5097c7, L:/g.g.g.234:3514 ! R:/h.h.h.234:43168]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:30.322-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0xc5ac9138, L:/g.g.g.234:3514 ! R:/172.29.255.235:53410]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:31.389-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x8a012fad, L:/g.g.g.234:3514 ! R:/h.h.h.234:43194]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:32.344-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0xe5701a64, L:/g.g.g.234:3514 ! R:/172.29.255.235:53414]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:33.411-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x003d71c4, L:/g.g.g.234:3514 ! R:/h.h.h.234:43204]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:34.367-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x57eaf586, L:/g.g.g.234:3514 ! R:/172.29.255.235:35534]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:35.433-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0xb4f95300, L:/g.g.g.234:3514 ! R:/h.h.h.234:43218]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:36.388-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x3e8d5483, L:/g.g.g.234:3514 ! R:/172.29.255.235:35546]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:37.455-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0xe44b3cf3, L:/g.g.g.234:3514 ! R:/h.h.h.234:43230]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:38.411-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x6f124bef, L:/g.g.g.234:3514 ! R:/172.29.255.235:35548]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:39.477-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0xc897b097, L:/g.g.g.234:3514 ! R:/h.h.h.234:46268]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:40.433-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x8f87bc1f, L:/g.g.g.234:3514 ! R:/172.29.255.235:35564]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:41.499-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x7bbe9ede, L:/g.g.g.234:3514 ! R:/h.h.h.234:46284]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:42.455-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x195832f5, L:/g.g.g.234:3514 ! R:/172.29.255.235:35580]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:43.521-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x250ad965, L:/g.g.g.234:3514 ! R:/h.h.h.234:46300]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:44.477-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x437344c5, L:/g.g.g.234:3514 ! R:/172.29.255.235:52958]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:45.543-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x88bcd367, L:/g.g.g.234:3514 ! R:/h.h.h.234:46304]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:46.499-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x873d58bf, L:/g.g.g.234:3514 ! R:/172.29.255.235:52980]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)
2025-02-26T02:24:47.565-08:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/rsyslog/67bcd225a7be9966a0349d6b] (channel [id: 0x47f41228, L:/g.g.g.234:3514 ! R:/h.h.h.234:46316]) (cause io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer)

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.