Combining values for graph

Hello,

Am attempting to parse modsecurity logs. Something that works and I currently have two fields:

RuleId
transaction_request_uri

I want to have a field with RuleId - transaction_request_uri. I managed to do this easily with a decorator however that will not allow me to make the pie graph I want.

What is the recommended method of achieving this. And are there any examples?

the recommend way would be to use the processing pipelines.

if you already have that with a decorator - now the question is how - that should be doable for you with the processing pipelines.

Thank you for the advice

I have been attempting to generate a pipeline that works but sadly no such luck.

rule "modsecurity-concat-alert_v2"
when
  has_field("document_type") and to_string($message.document_type) == "modsecurity"
then
  let alert_path_tmp = concat(to_string($message.alert), to_string($message.transaction_request_uri));
  set_field("alert_path", alert_path_tmp);
end 

Changing the alert_path_tmp to “blabla” works. Could it be that extractors are run after pipeline? And therefor the fields are not available yet?

EDIT:
I have done some additional debugging. The value’s are not available it seems

  let debug_message = concat("Dropped message from ", to_string($message.alert));
  debug(debug_message);

2019-07-09T17:32:06.129+02:00 INFO [Function] PIPELINE DEBUG: Dropped message from

As far as i can tell this means that the value is not yet available. Because the extractors run after. Am going to look into parsing the json in the pipeline just to get these two value’s

Sadly my skill level isn’t high enough to get this to work.

This is an example of the message

{"transaction":{"client_ip":"172.104.62.117","time_stamp":"Tue Jul  9 18:04:12 2019","server_id":"1c99b919232e4014434ab71b8e9a2d0c917e2a08","client_port":44966,"host_ip":"87.233.198.123","host_port":80,"unique_id":"15626882520.290244","request":{"method":"GET","http_version":1.0,"uri":"/","headers":{}},"response":{"http_code":200,"headers":{"HTTP/1.0":"200 OK","Date":"Tue, 09 Jul 2019 16:04:12 GMT","Upgrade":"h2,h2c","Server":"Apache/2.4.29 (Ubuntu)","Strict-Transport-Security":"max-age=63072000; includeSubDomains","Connection":"Upgrade","Link":"<https://www.edworks.info/index.php?rest_route=/>; rel=\"https://api.w.org/\"","Vary":"Accept-Encoding","Content-Type":"text/html; charset=UTF-8"}},"producer":{"modsecurity":"ModSecurity v3.0.3 (Linux)","connector":"HAProxy Dynamic Update version 1.6 2017/01/25 compiled with HAProxy version 1.8.0-2.0.0-195.864 '2019/06/19'\n","secrules_engine":"Enabled","components":["OWASP_CRS/3.0.2\""]},"messages":[{"message":"Request Missing a Host Header","details":{"match":"Matched \"Operator `Eq' with parameter `0' against variable `REQUEST_HEADERS:Host' (Value: `0' )","reference":"","ruleId":"920280","file":"/etc/hapee-1.8/modsec.rules.d/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf","lineNumber":"579","data":"","severity":"4","ver":"OWASP_CRS/3.0.0","rev":"2","tags":["application-multi","language-multi","platform-multi","attack-protocol","OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_HOST","WASCTC/WASC-21","OWASP_TOP_10/A7","PCI/6.5.10"],"maturity":"9","accuracy":"9"}}]}}

I thought that this should work

rule "duckhunt-modsecurity-concat-alert_v2"
when
  has_field("trueserver_document_type") and to_string($message.trueserver_document_type) == "duckhunt-modsecurity"
then
  let json_result = parse_json(to_string($message.message));
  let json_fields = select_jsonpath(json_result, 
    {
       client_ip: "$.client_ip"
    }
  );

  set_field("client_ip2", json_fields.client_ip);
end 

But it doesn’t and it is only one of the fields i want, ruleid which is in a list which makes getting to the first ruleid probably even harder.

when you look into the docs: http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#select-jsonpath

this should be something like:

rule "duckhunt-modsecurity-concat-alert_v2"
when
  has_field("trueserver_document_type") and to_string($message.trueserver_document_type) == "duckhunt-modsecurity"
then
  let json_result = parse_json(to_string($message.message));
  let json_fields = select_jsonpath(json_result, .client_ip);

  set_field("client_ip2", json_fields);
end 

you select a specific part of the json tree and not design a new one.

ps: the above is not tested.

1 Like

Thank you for the push in the right direction. I got it working!!!

This is my finalized pipeline for those looking for a simulare solution.

rule "duckhunt-modsecurity-concat-alert_v2"
when
  has_field("trueserver_document_type") and to_string($message.trueserver_document_type) == "duckhunt-modsecurity"
then
  let json_result = parse_json(to_string($message.message));
  let json_fields = select_jsonpath(json_result, 
    {
       ruleId: "$['transaction']['messages'][0]['details']['ruleId']",
       method: "$['transaction']['request']['method']",
       uri: "$['transaction']['request']['uri']"
       
    }
  );

  let tmp =  concat(to_string(json_fields.ruleId), " ");
  let tmp2 = concat(tmp, to_string(json_fields.method));
  let tmp3 = concat(tmp2, " ");
  let tmp4 = concat(tmp3, to_string(json_fields.uri));
 
  set_field("alert", tmp4);

end

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.