I am experimenting with Sidecars to send Windows logs via beats. I have it working fine on one Windows server that is on the same subnet as my Graylog server, but a Windows server on a different subnet is not able to send in logs.
There is a firewall between the two subnets. I have ports 5044 and 9000 open between the two subnets. Running a packet trace on my firewall, as well as Wireshark/tcpdump on the Windows host and Graylog, shows me that packets over port 9000 are making it round trip between the Windows host in one subnet and the Graylog host in the other subnet, so at first glance it looks like my firewall rules are allowing traffic properly between the two subnets.
However, 1) I get no logs loaded via the sidecar, 2) the sidecar status page in Graylog shows a node IP address of 22.214.171.124 (which is obviously not correct), and 3) while the graylog-collector-winlogbeat service is running on the Windows server that is sending logs successfully, the server that is not sending logs does not have that service running.
Any ideas what could cause this?