Is there anyway I can get the top 5 IP address (field ‘x-forwarded-for’ from the IIS logs) with the field ‘cs-method’ containing ‘GET’ via a http request or curl command ?
I need this to implement rate limiting for our web servers.
Regards,
Harry W.
Hey @harryw,
you can use the terms endpoint of the search API.
curl -u user:pass "https://graylog.example.com/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true" -H "Accept: application/json" -H "Content-Type: application/json"
api/search/universal/relative/terms -> API-Endpoint
query=cs-method%3Aget -> The query "cs-method:GET"
range=7200 -> Your timerange in seconds
field=x-forwarded-for& -> field that should be aggregated
size=5 -> size of the aggregation/count of entries
pretty=true -> Makes everything pretty 
, can be left out when called by a script.
The terms get sorted by the score of the query, so it sorts “most occurrences first” in the values to be returned, but the order in the response is not sorted.
###Example response:
{
"time" : 11,
"terms" : {
"192.168.2.15" : 55,
"192.168.2.11" : 99,
"192.168.2.13" : 77,
"192.168.2.14" : 69,
"192.168.2.12" : 88
},
"missing" : 0,
"other" : 0,
"total" : 12,
"built_query" : "{\"from\":0,\"query\":{\"bool\":{\"must\":{\"query_string\":{\"query\":\"cs-method:get\",\"allow_leading_wildcard\":false}},\"filter\":{\"bool\":{\"must\":{\"range\":{\"timestamp\":{\"from\":\"2017-03-17 02:54:33.989\",\"to\":\"2017-03-17 06:54:33.989\",\"include_lower\":true,\"include_upper\":true}}}}}}},\"aggregations\":{\"gl2_filter\":{\"filter\":{\"bool\":{\"must\":{\"range\":{\"timestamp\":{\"from\":\"2017-03-17 02:54:33.989\",\"to\":\"2017-03-17 06:54:33.989\",\"include_lower\":true,\"include_upper\":true}}}}},\"aggregations\":{\"gl2_terms\":{\"terms\":{\"field\":\"x-forwarded-for\",\"size\":5}},\"missing\":{\"missing\":{\"field\":\"x-forwarded-for\"}}}}}}"
}
Greetings - Phil
I tried the following;
curl -u admin:xxxxxx “https://rawqclm121v.rwwaq.com.au/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true” -H “Accept: application/json” -H "Content-Type: application/json"
But I’m getting back
curl: (56) Received HTTP code 503 from proxy after CONNECT
Any idea’s
Hey,
add the flag -v to the call, then we can see where the error occurs.
So try
curl -u admin:xxxxxx -v "https://rawqclm121v.rwwaq.com.au/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true" -H "Accept: application/json" -H "Content-Type: application/json"
Greetings - Phil
Ok, I think it’s my proxy settings inb my OVA machine …
CONNECT rawqclm121v.rwwaq.com.au:12900 HTTP/1.1
Host: rawqclm121v.rwwaq.com.au:12900
User-Agent: curl/7.35.0
Proxy-Connection: Keep-Alive
Accept: application/json
Content-Type: application/json
< HTTP/1.1 504 Gateway Timeout
< Proxy-Agent: IWSS
< Content-Type: text/html; charset=UTF-8
< Cache-Control: no-cache
< Connection: close
< Date: Fri, 17 Mar 2017 08:15:31 GMT
<
I just have to find out how to exclude the ‘*.rwwaq.com.au’ domain from going out through my proxies but I’m a novice with Linux so please bear with me.
Ok I’ve got a little further …
$ curl --noproxy ‘*’ -v -u admin:xxxxxx “https://rawqclm121v.rwwaq.com.au:12900/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true” -H “Accept: application/json” -H “Content-Type: application/json”
A little hint: --noproxy DomainToExclude,DomainToExclude2
Try
curl -u admin:xxxxxx --noproxy *.rwwaq.com.au "https://rawqclm121v.rwwaq.com.au/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true" -H "Accept: application/json" -H "Content-Type: application/json"
No problem, we all have started small 
Greetings - Phil
EDIT: Damn, you beat me by 2 seconds 
Are you using a self-signed certificate? Can you connect to Graylog without any SSL warning in your browser?
Good question.
I’m only using the certificate which came with the OVA.
I haven’t generated any certificates.
If I enter the following URL in my Chrome browser window …
http://rawqclm121v.rwwaq.com.au:12900/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true"%20-H%20"Accept:%20application/json"%20-H%20"Content-Type:%20application/json"
I initially get prompted for userid/password and then the following information is displayed …
{“time”:104,“terms”:{},“missing”:0,“other”:0,“total”:0,“built_query”:"{“from”:0,“query”:{“bool”:{“must”:{“query_string”:{“query”:“cs-method:get”,“allow_leading_wildcard”:true}},“filter”:{“bool”:{“must”:{“range”:{“timestamp”:{“from”:“2017-03-17 06:34:28.592”,“to”:“2017-03-17 08:34:28.592”,“include_lower”:true,“include_upper”:true}}}}}}},“aggregations”:{“gl2_filter”:{“filter”:{“bool”:{“must”:{“range”:{“timestamp”:{“from”:“2017-03-17 06:34:28.592”,“to”:“2017-03-17 08:34:28.592”,“include_lower”:true,“include_upper”:true}}}}},“aggregations”:{“gl2_terms”:{“terms”:{“field”:“x-forwarded-for”,“size”:5}},“missing”:{“missing”:{“field”:“x-forwarded-for”}}}}}}"}
As you can see no IP addresses are returned in x-forwarded-for field.
So I tried the query in the Graylog web interface and the display returned the field values correctly.
Can you send a screenshot of your query on the Graylog Web Interface? Maybe I mistyped a field in the query.
Another way to get the cURL command is by using the Firefox Dev-Console.
terms?query=[...].-H "Authorization:"Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" part and exchange it with the -u option for correct authentication.Greetings - Phil
Hi Phil.
I may have to rethink my idea of using this list to rate limit the IP addresses for Graylog/IIS.
It’s like whack-a-mole, as soon as you filter/limit one IP address another one pops up in it’s place.
Thank you for the work in helping me.
Regards,
Harry W.