Graylog Process Buffer Issue (extractors)

Graylog Central (peer support)
1

Hello,

I have an issue where my process buffer fills up to 100% and messages just keep getting hunt up. I did a ton of troubleshooting, and isolated it now to an extractor issue. I’ve gone so far as to figure out exactly which one which is a sonicwall one where I’m grabbing the source IP from a “message”

I’m confused though because I have 2 extractors (regular expressions) one to grab the source IP and one to grab the destination IP. They are basically written the same, but the destination IP works fine with 0 issues. The source IP one causes my buffer to fill up. If i delete it, my buffers get cleared out perfectly, so I know its an issue.

I just don’t get why since they are the same!?

(?:dst|dstip|[dD]estination\saddress)=“?(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})”?

This one is the issue though…
(?:src|srcip|[sS]ource\saddress)=“?(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})”?

2

It could be helpfull to put a line in here that reflects what is to be processed.

It could be that if both src and dst are in the same line the src extractor matches partly on the IP addres.

3

Well, I changed them all to 1,2 instead 1,3 and that solved the buffer issue, but I dont think the expression is still working correctly. For example:

Here is an example a sonicwall message. Sometimes it pulls out the sourceip just fine, but sometimes it doesn’t. It seems to be completely random.

id=firewall sn=18B169D90480 time=“2024-08-27 07:49:38” fw=50.200.150.186 pri=6 c=262144 m=98 msg=“Connection Opened” app=2 n=79981025 src=10.10.0.11:61438:X0 dst=75.75.75.75:53:X1 dstMac=f4:b5:2f:05:c7:75 proto=udp/dns sent=80 dpi=0 rule=“16 (LAN->WAN)” fw_action=“NA”

sometimes the rule will be X.X.X.X:X0 and sometimes it may have a port there too like X.X.X.X:5043:X0

4

It could be that this is a java issue, can this help you out on this issue?

8.16. Matching IPv4 Addresses - Regular Expressions Cookbook, 2nd Edition [Book])%5Cb%20…

5

I dunno, I did the most simpliest of expressions using: src\s*=\s*([\d.]+)

and I still got issues after about 10min my buffer started to slowly fill up. Why after 10min or so, I have no clue.

If I delete the expression completely, and wait 10 or more, buffer is fine… no issues. 30minutes later, still fine. So I know its because of that expression I made.

Question is, why!! lol

6

Well, it could be because of their own java and so regex implementation, you could try putting an issue on graylog githup. I have seen some more talks on the community pointing to java style regex.

Is it a pipeline regex, or on the input right away?

7

Its on the input. I dont recall having issues in the past and I’ve been using graylog for years. I use the whois threatintel plugin as well paired with this.

I’ve been upgrading graylog a bunch though over the summer. I’m on 5.2.10 now and I installed that maybe 2-3 weeks ago. Maybe the issue started then…

8

ok so i found something intresting… if i recreate the reg ex with a different “store as field” name…no buffer issues.

However, if I edit it and change it back to “sourceip” for my store as field option, I get the buffer issues right away.

Coincidentally, the whois plugin always uses this field, so I wonder if the issue is with the plugin and the lookups it does on the IP address

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.