1. Describe your incident:
I have 4 rules throughout 2 stages (Stage 0 and Stage 1). The rules in Stage 0 are for GeoIP to show the geolocation of a network event. These rules are working fine. In my next stage I have 2 rules setup using MISP to look at Domains and also a syscheck for files added to the system. The rules are not working. I am new to GrayLog and especially pipelines. I followed some tutorials online but still no avail. I provided a screenshot of my MISP lookup table and my two rules for MISP.
rule "OSSEC SYSCHECK ADDED - MISP THREAT INTEL"
when
$message.rule_group1 == "ossec" AND $message.rule_group2 == "syscheck" AND $message.syscheck_event == "added"
then
let ldata = lookup(
lookup_table: "misp_query",
key: to_string($message.syscheck_sha256_after)
);
set_fields(
fields: ldata,
prefix: "misp_"
);
end
rule "DOMAIN CHECK - MISP THREAT INTEL"
when
has_field("data_dns_question_registered_domain")
then
let ldata = lookup(
lookup_table: "misp_query",
key: to_string($message.data_dns_question_registered_domain)
);
set_fields(
fields: ldata,
prefix: "misp_"
);
end
3. What steps have you already taken to try and solve the problem?
Tried updating the rules
Tested the Data Adapter and it works perfectly
Tried Debug in rule source but did not help
Tried creating a Second Pipeline
Tried putting everything on one stage
Checked the lookup table was named correctly
Checked my rule groups
checked field names
Set continue processing on next stage when none or more rules on this stage match (on Stage 0)
4. How can the community help?
Just an explanation if there is something I am missing or something wrong?
Correct me if im wrong, the first set of rule in Stage 1 are working but stage 2 with the other rules are not. When you create the second pipe did you test each rule separate to see if they work at all?
Have you tired to use “Simulate processing” for your rules?
Not sure if you know the flow of rule and/or stages here is a brief overview.Stage #1 if rule/s match go to second stage rule do not match stop. Since you have 4 rules does either of them match, example:
rule #1 looking for a match in ${message.message} for user_name create field someone =user_name
rule #2 has_field Someone, then do what ever.
Or maybe after looking back over you rule something like this
rule " OSSEC SYSCHECK ADDED - MISP THREAT INTEL"
when
has_field("rule_group1") AND contains(to_string($message.rule_group1),"ossec") &&
has_field("rule_group2") AND contains(to_string($message.rule_group2),"syscheck")
then
let ldata = lookup(
lookup_table: "misp_query",
key: to_string($message.syscheck_sha256_after)
);
set_fields(
fields: ldata,
prefix: "misp_"
);
end
When I have issues with pipelines I like to simplify until it works and then build from there.
For example start with your then clause and in when only to set a single field that says the when matched
Then start building from there one at a time, writing variable values to plain fields as string etc so that you can see everything happening step by step, until you find your error.
If you have access to decorators in your version, pipeline decoraters are AMAZING and make it so fast to test your updates quickly. Decorators
Hello,
Thank you for your response. I just had a realization though. It would only add the MISP fields in the log if the sha256 from the file downloaded matches a sha256 in the MISP database, is that correct? Same with the domain? It would only add the MISP fields if it gets a hit from the database? So maybe the rules are working, there’s just no hits? or should it be giving me MISP information for everything?
I do believe so, specially if you try to match sha256.
Thats is possible, maybe it is working but nothing bad is happening. Just an idea, if this is a dev setup perhaps mimic some thing that should trigger the rule and make those fields.
Can you post the config on your data adapter that powers the lookup table (redact what you need to) there can be options set to return default data on no match depending on the kind of data adaptor.
Ah okay yes you will only get a response when there is a match. You could try hardcodeing the variable to a sha value know gets a response to check all the code, and then switch the variable back to the message value after.
Thank you so much for your help. I just wanted to make sure it was setup correctly but not really in an environment to just download live malware. Thank you for clarifying!
