Post your pipeline rules here with comments or how-to documentation.`-- VARIABLE $$Ports_Seen_Threshold$$ String
– Ignoring log entries with src_port 53 (DNS) due to Unity Media Connect Box reset issues
WITH IP_Ports AS (
SELECT distinct device_serial_id AS Device_ID, src_ip AS Source_IP, dst_port, in_interface AS Interface, CAST(timestamp AS DATE) AS Scan_Date
FROM xgfw_data
WHERE log_type = 'Firewall'
AND log_component = 'Appliance Access'
AND src_port <> 53
ORDER BY dst_port
),
IP_Port_Count AS (
SELECT Device_ID, Source_IP, COUNT(dst_port) AS Port_Count, array_join(array_agg(dst_port), ',') AS Port_List, Interface, Scan_Date
FROM IP_Ports
GROUP BY Source_IP, Interface, Scan_Date, Device_ID
)
SELECT
'PORTSCAN DETECTED' AS Detection, Scan_Date, Source_IP, Port_Count, Port_List, Interface, Device_ID
FROM IP_Port_Count
WHERE Port_Count >= $$Ports_Seen_Threshold$$
ORDER BY Scan_Date DESC`
Hey, guys I recently detected an ongoing scan deep in the logs we collect.
Found this code but I can’t see some functions that would help translate the above code correctly. Any ideas as to how I can go about it.
All of those parts of the message (Device_ID, src_ip etc.) can be broken out using extractors or the pipeline once you have the data broken out to fields, they can be acted upon based on Alerts to send you a message via e-mail (or… Slack, Teams, Pagerduty, http…). Does that make sense?
(NOTE: I marked the code as such with the </> forum tool to make it more readable and moved the question the “Graylog Central” area where questions are asked/answered.)
What @tmacgbay suggested is the way to go, this would depend what you want to use. Pipeline would be you best option using regex expression then create the fields for each. I would like to suggest that only create the fields that are need because it will take up more space.
Here are a couple examples I have used since the logs look similar to one of my devices.
rule "Source Port"
when
has_field("message")
then
let batman = regex("srcport=(\\d\\))",to_string($message.message));
set_field("robin", batman["1"]);
end
Actually testing out my pipeline above I found it was incorrect.
This should work better.
rule "Extract fields"
when
has_field("message")
then
let siteName = regex("(srcport=\\s*(\\S+))", to_string($message.message));
set_field("siteName", siteName["1"]);
debug (siteName);
end
rule "Extract multiple fields"
when
has_field("message")
then
let srcport = regex("(srcport=\\s*(\\S+))", to_string($message.message));
let protocol = regex("(proto=\\s*(\\S+))", to_string($message.message));
let sourceInterface = regex("(srcintf=\\s*(\\S+))", to_string($message.message));
set_field("srcport", srcport["1"]);
set_field("protocol", protocol["1"]);
set_field("sourceInterface", sourceInterface["1"]);
end
@tmacgbay It was tuff because my Extractor REGEX did not work in the pipeline
I’m not sure they pay you enough for posts like this @gsmith!
I thought about using the key_value() function with the data but only specific fields are desired.
(On a side note it helps a LOT if you post the text of the message rather than a picture, it gives others something to play with for the answer…)
With regex it would be more efficient to have a single search and use the grouping to pull out the things you want… something like this:
rule "Extract multiple fields"
when
has_field("message")
then
let BatSignals = regex("srcport=(\\S+).*proto=(\\S+).*srcintf=(\\S+)", to_string($message.message));
set_field("srcport", BatSignals["1"]);
set_field("protocol", BatSignals["2"]);
set_field("sourceInterface", BatSignals["3"]);
end