Couldn’t find any solution to this so here I am asking you guys. Basically, I have a UDP input for all my network devices and it can only process out about 3.5K msg/s. After that, the process buffer starts filling up so it was probably an extractor or some other parsing process causing this. Turns out that if I disable the force rDNS option, the input can process up to 14K msg/s
Question is, how can I improve this? It looks like I have to use rDNS since some syslog messages get erroneous source names if I don’t.
Just to follow up on this. I installed and configured Unbound and added both A and PTR records for my sources. I confirmed that Graylog is using these entries to parse the source of the messages by the meands of using rDNS.
But still, it looks like it’s rated at ~3K msg/s.
Is there anything else that can be done to improve this?
Would using pipelines or extractors to replace the source with a specific string, based on the gl2_remote_ip field be faster? (a quick example on how the extractor would look would be nice)
Let’s say I’d like to parse all messages coming into a certain input (Syslog UDP) and set the ‘source’ field to some string that depends on the source IP (would be mimicking what DNS does).
For instance, this is the pseudo code as I don’t know the syntax of Pipeline rules yet:
if message field ‘gl2_source_ip’ == 10.0.0.1
then set message field ‘source’ to ‘host1’
if message field ‘gl2_source_ip’ == 10.0.0.2
then set message field ‘source’ to ‘host2’