Hi there
I currently have an A10 networks device sending Syslog messages (RFC5424) via UDP to a 3-node Graylog cluster on a Global UDP input (UDP 1514). This seems to work fine, but I am looking to make everything inherently more ‘reliable’ and would like to move the comms over to TCP instead.
Graylog refuses to process messages received from this device via TCP though. Syslog TCP Inputs do not ingest any messages at all, and Raw TCP Inputs only successfully ingest messages from this device upon shutting down the Input (as if it’s flushing a buffer or journal when doing so) - I have tried this on TCP 1514 and TCP 5555 with the same results.
I have run packet captures and can confirm that the actual content of the Syslog message is the same regardless of TCP/UDP transport methods - as one would expect.
I do not seem to be able to get any information from Graylog’s debug/trace functionality about the ignored/discarded data, and also have not observed any of the common solutions applying (future-dated timestamps, journal full, disk full, etc). The input statistics in Graylog’s web interface do recognise the network traffic hitting the Input in terms of Network input measurement. I’m not quite sure how to investigate further.
I have tested having the A10 device send TCP syslogs to a simple rsyslog collector, which works flawlessly with minimal configuration. Additionally, I have also successfully sent and ingested TCP Syslog messages to the Graylog cluster from a test machine that isn’t the A10 device.
All Graylog nodes are running on Ubuntu 18.04.3 LTS with UFW disabled and AppArmor disabled (for ease of debugging), Graylog 3.1.4+1149fe1 with MongoDB 4.0.15 and ElasticSearch 6.8.6. Was previously having this same problem on Graylog 3.0.2 with MongoDB 4.0.x and ES 6.8.6 too.
