HTTP notification AbuseIPDB

[jonathanb (Jonathan Bastin)]

Hi,

I am trying to set up reporting to AbuseIPDB, which is proving to be much harder than expected. I have the data in Graylog to create a stream and send the data. I need to create a HTTP post:

curl https://api.abuseipdb.com/api/v2/report \
  --data-urlencode "ip=127.0.0.1" \
  -d categories=18,22 \
  --data-urlencode "comment=SSH login attempts with user root." \
  -H "Key: YOUR_OWN_API_KEY" \
  -H "Accept: application/json"

Is there any way to do that within Graylog 5.x?

[dscryber (David Sciuto)]

Hey @jonathanb thanks for asking.I think you can set up reporting to AbuseIPDB in Graylog 5.x using a Graylog pipeline rule.

Generally, here’s how:

1. Create a new pipeline rule in Graylog and specify the stream you want to use for reporting.
2. In the pipeline rule, add a rule that matches the specific log events you want to report to AbuseIPDB.
3. Use the Graylog built-in HTTP Output to create a HTTP POST request to the AbuseIPDB API. You can configure the URL, headers, and parameters as shown in your curl example.
4. Save the pipeline rule and test it to make sure it’s working as expected.

Note: You will need to replace “YOUR_OWN_API_KEY” with your actual API key in the HTTP header.

Hope this helps.

[jonathanb (Jonathan Bastin)]

dscryber,

Thank you for the response. I presume the pipeline is to create the variables? And then those are passed to the HTTP output and placed into the URL?

I have been trying to read the documentation to understand.

Thank you

[dscryber (David Sciuto)]

Yes. Create variables from input data, process them, and then pass them to an HTTP output module where they are formatted and placed into a URL as query parameters or part of the endpoint. The URL is then used to send an HTTP request to a remote server to retrieve or update data.

Here in the community we have some pipeline experts, like @tmacgbay . You may want to reach out to him.

[gsmith]

hello @jonathanb

Just chimmming in, I’m @tmacgbay apprentice in pipelines ( AKA Robin).

@jonathanb

I presume the pipeline is to create the variables? And then those are passed to the HTTP output and placed into the URL?

Found this in the forum.

Manage email JsonArray

Try to use this 2 pipeline rules. I assume, that you have email addresses extracted in field recipients First one (step 1) clean unnecesery chars ["] from recipient field, split values with comma, extract first email, extract domain, compare with lookup table, setup field internal (true/false) rule "email jsonarray 1.1" when has_field("recipients") AND contains(to_string($message.recipients), "@") then // Remove unneccesary strings []" from array field let fix_strings = regex_repl…

[jonathanb (Jonathan Bastin)]

@gsmith thank you for that info.

I am trying to read through the example to see what fits.

Trying to write it out then as a start:

rule "AbuseIPDB"
when
  has_field("Snort_source_IP") &&
  is_null(lookup_value("AbuseIPDB_reported", to_string($message.Snort_source_IP)))
then
let ip = is_ip($message.Snort_source_IP);
let discription = is_string($message.Snort_Description);
let url = to_url(["https://api.abuseipdb.com/api/v2/report?key=diuyef73ubysdvciujqw9h&"ip"&categories=18,22&"discription]);
let setlookup = lookup_set_value("AbuseIPDB_reported",ip,"true");
end

So with the above I am now confused how to call that URL.

[tmacgbay]

It looks like you are interested in querying api.abuseIPDB.com with data from a message field and having the Graylog message contain the results in a separate new field?

The way other queries are done like this is via System/Lookup Tables. There you can start with a Data Adapter… they have several built in for things like DNS lookup, GrayNoise, ThreatFox… some of which are only available for the licensed version. It looks to me like the closest you can get is by creating a “HTTP JSONPath” adapter, then a Cache and Lookup Table associated with it. I have created lookup tables but I haven’t worked with this adapter. It has some instruction built into the configuration page that looks like it will give you enough to get there… more than the online docs.

If you do get it working, it would be super helpful to future readers/searchers if you documented your steps and posted them either here or even better in the rules and templates exchange.

[jonathanb (Jonathan Bastin)]

@tmacgbay
I will be setting this up as a content package or providing instructions once working.

The lookup is not to spam reporting every time the IP comes up. So I am writing to a lookup to see if previously reported by my system.

So the removing the lookup the code I need to get 100%:

rule "AbuseIPDB"
when
  has_field("Snort_source_IP") 
then
let ip = is_ip($message.Snort_source_IP);
let discription = is_string($message.Snort_Description);
let url = to_url(["https://api.abuseipdb.com/api/v2/report?key=diuyef73ubysdvciujqw9h&"ip"&categories=18,22&"discription]);
end

How every the URL doesn’t work either as you need to set headers.

[tmacgbay]

Maybe my coffee hasn’t kicked in…

You are trying to send (you said POST) data external to Graylog with a URL that describes the path to send it. If you create a lookup table with the “HTTP JSONPath” Adapter, you can do just that. It will likely return something you are interested in for confirmation or exists/NULL or FQDN or whatever…

on a side note you may need to adjust some functions:

let ip = is_ip($message.Snort_source_IP);

Might be to_ip() or even to_string() since the function is_ip() returns a boolean as to whether the Snort_source_IP is an IP or not.

Same thing with is_string() vs to_string()

[jonathanb (Jonathan Bastin)]

@tmacgbay

Right I now have this:

rule "AbuseIPDB"
when
  has_field("Snort_source_IP") &&
  is_null(lookup_value("AbuseIPDB_reported", to_string($message.Snort_source_IP)))
then
//Get the Offending IP
let ip = to_string($message.Snort_source_IP);
//Get the Snort Description
let discription = to_string($message.Snort_Description);

//Create the IP string
let concatip = concat("ip=",to_string($message.Snort_source_IP));
//Encode the IP
let encodedip = base64url_encode(concatip);

//Create the Comment string
let concatdiscription = concat("comment=",to_string($message.Snort_Description));
let encodeddiscription = base64url_encode(concatdiscription);

//Set Categories
let cat = to_string("categories=18,22");

//Create full request
let reporturl_1 = concat(encodedip,cat);
let reporturl = concat(reporturl_1,encodeddiscription);

//Lookup to report ip
let report = lookup_value("AbuseIPDB_Report", reporturl);

//Below line has an error can not index value
//set_field("abuseConfidenceScore", report["abuseConfidenceScore"]);

let setlookup = lookup_set_value("AbuseIPDB_reported",ip,"true");
end

I have the return error to fix, however, I also need to put quotes (") round the IP address in:
let concatip = concat(“ip=”,to_string($message.Snort_source_IP));

[tmacgbay]

@jonathanb

let concatip = concat(“ip=”,to_string($message.Snort_source_IP));

For quotes, you can escape them … I think this would work to put quotes around the IP…

let concatip   = concat("ip=\"",to_string($message.Snort_source_IP));
let concat_fin = concat(concatip,"\""); 
// do something with concat_fin here

Not sure what the return data looks like in “report”, you can use the debug function to see it show up in the Graylog log file…

 // use $ tail -f /var/log/graylog-server/server.log to watch for the results of the below debug message
 //
 debug(concat("============ results in report: ",to_string(report)));

[jonathanb (Jonathan Bastin)]

@tmacgbay

Right, we are close. I didn’t need “” I checked the API manual.

2023-02-02T18:58:46.754Z INFO  [Function] PIPELINE DEBUG: ============ URL report: ip%3D172.16.0.53&categories=18,22%26comment%3DET+SCAN+Suspicious+inbound+to+MSSQL+port+1433
2023-02-02T18:58:46.890Z WARN  [HTTPJSONPathDataAdapter] Couldn't read single JSONPath from response - returning empty result (Expected to find an object with property ['data'] in path $ but found 'java.lang.String'. This is not a json object according to the JsonProvider: 'com.jayway.jsonpath.spi.json.JsonSmartJsonProvider'.)
2023-02-02T18:58:46.890Z INFO  [Function] PIPELINE DEBUG: ============ results in report:

Code is

rule "AbuseIPDB"
when
  has_field("Snort_source_IP") &&
  is_null(lookup_value("AbuseIPDB_reported", to_string($message.Snort_source_IP)))
then
//Get the Offending IP
let ip = to_string($message.Snort_source_IP);
//Get the Snort Description
let description = to_string($message.Snort_Description);

//Create the IP string
let concatip = concat("ip=",to_string($message.Snort_source_IP));
//Encode the IP
let encodedip = urlencode(concatip);

//Create the Comment string
let concatdescription = concat("&comment=",description);
let encodeddescription = urlencode(concatdescription);

//Set Categories
let cat = to_string("&categories=18,22");

//Create full request
let reporturl_1 = concat(encodedip,cat);
let reporturl = concat(reporturl_1,encodeddescription);

//Lookup to report ip
debug(concat("============ URL report: ",to_string(reporturl)));
let report = lookup_value("AbuseIPDB_Report", reporturl);
debug(concat("============ results in report: ",to_string(report)));

//Below line has an error can not index value
//set_field("abuseConfidenceScore", report["abuseConfidenceScore"]);

let setlookup = lookup_set_value("AbuseIPDB_reported",ip,"true");
end

The Lookup table is:
URL: https://api.abuseipdb.com/api/v2/report?${key}
Single value JSONPath: $.data.abuseConfidenceScore
HTTP User-Agent: Graylog
HTTP Headers:
NAME VALUE
Accept application/json
Key fvdoijhsrvoinfvbopinfbsdlkn

So somewhere the HTTP is failing. As far as I can tell the URL string is correct from the rule but something is still not right.

Doing debug in curl:

curl https://api.abuseipdb.com/api/v2/report? -d ip%3D42.112.28.53&categories=18,22%26comment%3DET+SCAN+Suspicious+inbound+to+MSSQL+port+1433 -H "Key:sfdkbvjhfkbjhefewqfwesfewfewfwe" -H "Accept: application/json"

I get:

<html>
<head>
<meta charset="UTF-8" />
<meta http-equiv="refresh" content="0;url='https://api.abuseipdb.com/login'" />
<title>Redirecting to https://api.abuseipdb.com/login</title>
</head>
<body>
Redirecting to <a href="https://api.abuseipdb.com/login">https://api.abuseipdb.com/login</a>.
</body>
</html>curl https://apcurl https://api.abuseipdb.com/api/v2/report -d "ip%3D42.112.28.53&categories=18,22%26comment%3DET+SCAN+Suspicious+inbound+to+MSSQL+port+1433" -H "Key:3d7b40120f4187598987198177eff300663d093f8d492c2634a44af391ea6f0ed650bab946b9f7f7" -H "Accept: application/json"
{"errors":[{"detail":"The ip field is required.","status":422,"source":{"parameter":"ip"}}]}[1]+  Done                    curl https://api.abuseipdb.com/api/v2/report? -d ip%3D42.112.28.53

[tmacgbay]

@jonathanb

[HTTPJSONPathDataAdapter] Couldn't read single JSONPath from response - returning empty result

May need to adjust what you are passing to be in json format with parse_json()

Of Note:
This is all new territory here for me so don’t take what I am saying to be correct… I am guessing as much as or more than you. IT may be that this Data Adapter requires data to be passed as json but that won’t be the right format for what you want to do…

[jonathanb (Jonathan Bastin)]

@tmacgbay
Me too I don’t know exactly what curl is doing with

--data-urlencode

However, looking at Python it is not submitting the IP in the URL which is the issue. It is posting it as a form

url = 'https://api.abuseipdb.com/api/v2/report'

# String holding parameters to pass in json format
params = {
    'ip':'180.126.219.126',
    'categories':'18,20',
    'comment':'SSH login attempts with user root.'
}

headers = {
    'Accept': 'application/json',
    'Key': 'YOUR_OWN_API_KEY'
}

response = requests.request(method='POST', url=url, headers=headers, params=params)

So am I at a roadblock with what Graylog can do or can I pass into the HTTP lookup this way?

Otherwise, this needs a custom-written plugin.

[tmacgbay]

I may have been wrong about parse_json()…

I think you are close, hard to tell where it is off.

@jonathanb

let reporturl = concat(reporturl_1,encodeddescription);

Take the debug() results from reporturl and stick it in to the ${key} portion of https://api.abuseipdb.com/api/v2/report?${key}… throw all that in a browser and see what you come back with… modify that URL line until it comes back with what you want in the browser, then adjust the pipeline rule concat()s to match your modifications.

[jonathanb (Jonathan Bastin)]

@tmacgbay

I have done a load of tests with curl. The request URL is
https://api.abuseipdb.com/api/v2/report

It then wants a post with data:
ip=IPADDRESS
categories=NUMBERS,NUMBERS
comment=YOURCOMMENT

From what I understand of the lookup it can only pass data in the URL. AbuseIPDB wants the data in the data post (assuming that is the right terminology). As such it would need to be the HTTP Output but I don’t see it as an option.

If you request the URL with all the data the webserver dumps the request.

[jonathanb (Jonathan Bastin)]

@dscryber

Use the Graylog built-in HTTP Output to create a HTTP POST request to the AbuseIPDB API. You can configure the URL, headers, and parameters as shown in your curl example.

I am not able to see the HTTP Output under creating a new output. I am on Graylog 5.0.2+59d96f8 open source. Should it be there?

[David_W (David Warner)]

From my experience the Graylog Open version can only do GET not POST HTTP requests if the API requires POST I think you are out of luck without creating a custom plugin.

[jonathanb (Jonathan Bastin)]

@tmacgbay Thank you for all your input on this one. I have learnt a lot about pipeline rules.

Shamefully, I think writing a full plugin is out of my reach and I am going to have to can the idea. I hope that for someone that has the HTTP output there is enough information here to help.

If anyone wants to collaborate on a plugin please contact me as this is a great project for network admins.