Is there anyway I can get the top 5 IP address (field ‘x-forwarded-for’ from the IIS logs) with the field ‘cs-method’ containing ‘GET’ via a http request or curl command ?
I need this to implement rate limiting for our web servers.
api/search/universal/relative/terms -> API-Endpoint query=cs-method%3Aget -> The query "cs-method:GET" range=7200 -> Your timerange in seconds field=x-forwarded-for& -> field that should be aggregated size=5 -> size of the aggregation/count of entries pretty=true -> Makes everything pretty , can be left out when called by a script.
The terms get sorted by the score of the query, so it sorts “most occurrences first” in the values to be returned, but the order in the response is not sorted.
Connection #0 to host proxy.rwwa.com.au left intact
curl: (56) Received HTTP code 504 from proxy after CONNECT
I just have to find out how to exclude the ‘*.rwwaq.com.au’ domain from going out through my proxies but I’m a novice with Linux so please bear with me.
If I enter the following URL in my Chrome browser window … http://rawqclm121v.rwwaq.com.au:12900/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true"%20-H%20"Accept:%20application/json"%20-H%20"Content-Type:%20application/json"
I initially get prompted for userid/password and then the following information is displayed …
{“time”:104,“terms”:{},“missing”:0,“other”:0,“total”:0,“built_query”:"{“from”:0,“query”:{“bool”:{“must”:{“query_string”:{“query”:“cs-method:get”,“allow_leading_wildcard”:true}},“filter”:{“bool”:{“must”:{“range”:{“timestamp”:{“from”:“2017-03-17 06:34:28.592”,“to”:“2017-03-17 08:34:28.592”,“include_lower”:true,“include_upper”:true}}}}}}},“aggregations”:{“gl2_filter”:{“filter”:{“bool”:{“must”:{“range”:{“timestamp”:{“from”:“2017-03-17 06:34:28.592”,“to”:“2017-03-17 08:34:28.592”,“include_lower”:true,“include_upper”:true}}}}},“aggregations”:{“gl2_terms”:{“terms”:{“field”:“x-forwarded-for”,“size”:5}},“missing”:{“missing”:{“field”:“x-forwarded-for”}}}}}}"}
As you can see no IP addresses are returned in x-forwarded-for field.
So I tried the query in the Graylog web interface and the display returned the field values correctly.
Can you send a screenshot of your query on the Graylog Web Interface? Maybe I mistyped a field in the query.
Another way to get the cURL command is by using the Firefox Dev-Console.
Open the Graylog Web Interface.
Press Ctrl + Shift + Q to open the Network Console
Perform your query “cs-method:get” with the wanted time range
select the Quick Values widget for the “x-forwarded-for”
Look into the Network Console (scroll a little to make it stop going down) and search for the line with the value file starting with terms?query=[...].
Right click and select Copy as cURL.
Done. You probably need to remove the -H "Authorization:"Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" part and exchange it with the -u option for correct authentication.
I may have to rethink my idea of using this list to rate limit the IP addresses for Graylog/IIS.
It’s like whack-a-mole, as soon as you filter/limit one IP address another one pops up in it’s place.
Thank you for the work in helping me.