Filebeat Failed to publish events

Graylog Central (peer support)
1

Hello,

Filebeat gives the error

    2018-03-23T10:28:11+01:00 ERR  Failed to publish events caused by: read tcp 10.152.150.4:39210->10.154.229.210:5044: i/o timeout
    2018-03-23T10:28:11+01:00 ERR  Failed to publish events caused by: client is not connected

But the target 10.154.229.210 is reachable, I can do a telnet and send via netcat:

t6706@itpccl012347LX:~$ echo '<14>itpccl012347LX test nachricht for Graylog' | netcat -v -w 1 10.154.229.210 5044
Connection to 10.154.229.210 5044 port [tcp/*] succeeded!

The target is a Netscaler Loadbalancer and the IP ist not DNS reverse lookable.

If I use as target the direct IP of a node in the Graylog cluster (which is reverse DNS resolvable) everything works fine and filebeat can send the messages.

What could be the problem here? Is it possbile to make filebeat just send the message, and to not try a reverse DNS lookup?

Here the filebeat.yml configuration:

t6706@itpccl012347LX:~$ cat /etc/graylog/collector-sidecar/generated/filebeat.yml 
filebeat:
  prospectors:
  - fields:
      collector_node_id: itpccl012347LX
      gl2_source_collector: 65cb9ca6-4f43-498a-a1ff-fb58307e7782
      type: log
    ignore_older: 0
    paths:
    - /home/t6706/syslog_info_0199.log
    scan_frequency: 10s
    tail_files: true
    type: log
  - encoding: plain
    exclude_files: []
    fields:
      collector_node_id: itpccl012347LX
      gl2_source_collector: 65cb9ca6-4f43-498a-a1ff-fb58307e7782
      type: log
    ignore_older: 0
    paths:
    - /var/log/syslog
    scan_frequency: 10s
    tail_files: true
    type: log
output:
  logstash:
    hosts:
    - 10.154.229.210:5044
path:
  data: /var/cache/graylog/collector-sidecar/filebeat/data
  logs: /var/log/graylog/collector-sidecar
tags:
- importe
bulk_max_size: 1024

Filebeat and Graylog are the latest versions.

Are there similar problems out there?

Thanks in advance!

Regards,

Dietmar

2

What type of input have you created in Graylog and what is its complete configuration?

3

Hello,

here the Input configuration:

Beats_Input

bind_address: 0.0.0.0
override_source: <empty>
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password: ********
4

Does Filebeat always emit the same error message?
Have you tried running Filebeat in debug mode to get more details about the errors?
Have you captured the network communication with Wireshark/tcpdump and can verify that Filebeat is indeed sending packets over the network?

5

Hallo Jochen,

yes, the error message ist the same. But, if I try to send the output directly to a Graylog Cluster node (not via the loadbalancer) it works fine.
The Input is ok, because other systems can send logs via this Input without problems (direct).

I have not yet tried the debug mode of filebeat. Is there a way to configure this debug level in Graylog oder Collector_Sidecar?

I have not yet captured any packets, since I don’t think this is the problem. I will try if the debug mode of filebeat gives reason to do so.

Regards,

Dietmar Schurr

6

No, that’s not possible.

See Debugging | Filebeat Reference [5.6] | Elastic for details about running Filebeat in debug mode.

7

Hello Jochen,
now I started filebeat in debug mode, the error messages are the same:
2018-03-23T14:03:09+01:00 DBG [logstash] close connection
2018-03-23T14:03:09+01:00 DBG [logstash] close connection
2018-03-23T14:03:09+01:00 ERR Failed to publish events caused by: write tcp 10.152.150.4:40676->10.154.229.210:5044: use of closed network connection

If I start with
output:
logstash:
hosts:
- monlog1.int.it.gkvi:5044
which means filebeat will send directly to a Graylog node, circumventing the Loadbalancer, then it works, there are no errors.
Only difference (beside the fact that it is a loadbalancer): The IP of the LB is not reverse resolvable (via DNS), but the Graylog node ist DNS resolvable.
Could this be the problem?

Thanks in advance for the help.

Regards,
Dietmar

8

I would be very surprised if this was a problem at all.

I’d recommend using Wireshark or tcpdump to capture the network communication between Filebeat and the configured receiver to find out how they are different when sending to the load balancer and Graylog directly.

9

Only difference (beside the fact that it is a loadbalancer): The IP of the LB is not reverse resolvable (via DNS), but the Graylog node ist DNS resolvable.
Could this be the problem?

I’m not sure if the beats protocol/framework allows/is able to be run behind a LB - But I couldn’t find a notice about that in the documentation or the code. In the past I always use the loadbalancing feature of the beats in favor than using a LB.

10

Hallo Jan,

thanks for the answer. I will use the loadbalancer feature.
Is the assumption correct, that the load balancing feature in filebeat is not only for performance reasons, but can do also HA (no messages will be lost if a GL node is down) load balancing?

Thanks in advance,

Dietmar Schurr

11

Filebeat (and any other Beat) will locally buffer messages if it fails to receive an acknowledgement from the server.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.