List top 5 IP address via a CURL or HTTP request


#1

Is there anyway I can get the top 5 IP address (field ‘x-forwarded-for’ from the IIS logs) with the field ‘cs-method’ containing ‘GET’ via a http request or curl command ?
I need this to implement rate limiting for our web servers.

Regards,

Harry W.


(Philipp Ruland) #2

Hey @harryw,

you can use the terms endpoint of the search API.

curl -u user:pass "https://graylog.example.com/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true" -H "Accept: application/json" -H "Content-Type: application/json" 

api/search/universal/relative/terms -> API-Endpoint
query=cs-method%3Aget -> The query "cs-method:GET"
range=7200 -> Your timerange in seconds
field=x-forwarded-for& -> field that should be aggregated
size=5 -> size of the aggregation/count of entries
pretty=true -> Makes everything pretty :rainbow:, can be left out when called by a script.

The terms get sorted by the score of the query, so it sorts “most occurrences first” in the values to be returned, but the order in the response is not sorted.

###Example response:

{
  "time" : 11,
  "terms" : {
    "192.168.2.15" : 55,
    "192.168.2.11" : 99,
    "192.168.2.13" : 77,
    "192.168.2.14" : 69,
    "192.168.2.12" : 88
  },
  "missing" : 0,
  "other" : 0,
  "total" : 12,
  "built_query" : "{\"from\":0,\"query\":{\"bool\":{\"must\":{\"query_string\":{\"query\":\"cs-method:get\",\"allow_leading_wildcard\":false}},\"filter\":{\"bool\":{\"must\":{\"range\":{\"timestamp\":{\"from\":\"2017-03-17 02:54:33.989\",\"to\":\"2017-03-17 06:54:33.989\",\"include_lower\":true,\"include_upper\":true}}}}}}},\"aggregations\":{\"gl2_filter\":{\"filter\":{\"bool\":{\"must\":{\"range\":{\"timestamp\":{\"from\":\"2017-03-17 02:54:33.989\",\"to\":\"2017-03-17 06:54:33.989\",\"include_lower\":true,\"include_upper\":true}}}}},\"aggregations\":{\"gl2_terms\":{\"terms\":{\"field\":\"x-forwarded-for\",\"size\":5}},\"missing\":{\"missing\":{\"field\":\"x-forwarded-for\"}}}}}}"
}

Greetings - Phil


#3

I tried the following;
curl -u admin:xxxxxx “https://rawqclm121v.rwwaq.com.au/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true” -H “Accept: application/json” -H "Content-Type: application/json"
But I’m getting back
curl: (56) Received HTTP code 503 from proxy after CONNECT

Any idea’s


(Philipp Ruland) #4

Hey,

add the flag -v to the call, then we can see where the error occurs.

So try

curl -u admin:xxxxxx -v "https://rawqclm121v.rwwaq.com.au/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true" -H "Accept: application/json" -H "Content-Type: application/json"

Greetings - Phil


#5

Ok, I think it’s my proxy settings inb my OVA machine …

CONNECT rawqclm121v.rwwaq.com.au:12900 HTTP/1.1
Host: rawqclm121v.rwwaq.com.au:12900
User-Agent: curl/7.35.0
Proxy-Connection: Keep-Alive
Accept: application/json
Content-Type: application/json

< HTTP/1.1 504 Gateway Timeout
< Proxy-Agent: IWSS
< Content-Type: text/html; charset=UTF-8
< Cache-Control: no-cache
< Connection: close
< Date: Fri, 17 Mar 2017 08:15:31 GMT
<

  • Received HTTP code 504 from proxy after CONNECT
  • Connection #0 to host proxy.rwwa.com.au left intact
    curl: (56) Received HTTP code 504 from proxy after CONNECT

I just have to find out how to exclude the ‘*.rwwaq.com.au’ domain from going out through my proxies but I’m a novice with Linux so please bear with me.


#6

Ok I’ve got a little further …
$ curl --noproxy ‘*’ -v -u admin:xxxxxx “https://rawqclm121v.rwwaq.com.au:12900/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true” -H “Accept: application/json” -H “Content-Type: application/json”

  • Hostname was NOT found in DNS cache
  • Trying 192.168.127.151…
  • Connected to rawqclm121v.rwwaq.com.au (192.168.127.151) port 12900 (#0)
  • successfully set certificate verify locations:
  • CAfile: none
    CApath: /etc/ssl/certs
  • SSLv3, TLS handshake, Client hello (1):
  • Unknown SSL protocol error in connection to rawqclm121v.rwwaq.com.au:12900
  • Closing connection 0
    curl: (35) Unknown SSL protocol error in connection to rawqclm121v.rwwaq.com.au:12900

(Philipp Ruland) #7

A little hint: --noproxy DomainToExclude,DomainToExclude2

Try

curl -u admin:xxxxxx --noproxy *.rwwaq.com.au "https://rawqclm121v.rwwaq.com.au/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true" -H "Accept: application/json" -H "Content-Type: application/json"

No problem, we all have started small :slight_smile:

Greetings - Phil

EDIT: Damn, you beat me by 2 seconds :smiley: :joy:


(Philipp Ruland) #8

Are you using a self-signed certificate? Can you connect to Graylog without any SSL warning in your browser?


#9

Good question.
I’m only using the certificate which came with the OVA.
I haven’t generated any certificates.


#10

If I enter the following URL in my Chrome browser window …
http://rawqclm121v.rwwaq.com.au:12900/api/search/universal/relative/terms?query=cs-method%3Aget&range=7200&field=x-forwarded-for&size=5&pretty=true"%20-H%20"Accept:%20application/json"%20-H%20"Content-Type:%20application/json"
I initially get prompted for userid/password and then the following information is displayed …
{“time”:104,“terms”:{},“missing”:0,“other”:0,“total”:0,“built_query”:"{“from”:0,“query”:{“bool”:{“must”:{“query_string”:{“query”:“cs-method:get”,“allow_leading_wildcard”:true}},“filter”:{“bool”:{“must”:{“range”:{“timestamp”:{“from”:“2017-03-17 06:34:28.592”,“to”:“2017-03-17 08:34:28.592”,“include_lower”:true,“include_upper”:true}}}}}}},“aggregations”:{“gl2_filter”:{“filter”:{“bool”:{“must”:{“range”:{“timestamp”:{“from”:“2017-03-17 06:34:28.592”,“to”:“2017-03-17 08:34:28.592”,“include_lower”:true,“include_upper”:true}}}}},“aggregations”:{“gl2_terms”:{“terms”:{“field”:“x-forwarded-for”,“size”:5}},“missing”:{“missing”:{“field”:“x-forwarded-for”}}}}}}"}

As you can see no IP addresses are returned in x-forwarded-for field.
So I tried the query in the Graylog web interface and the display returned the field values correctly.


(Philipp Ruland) #11

Can you send a screenshot of your query on the Graylog Web Interface? Maybe I mistyped a field in the query.

Another way to get the cURL command is by using the Firefox Dev-Console.

  1. Open the Graylog Web Interface.
  2. Press Ctrl + Shift + Q to open the Network Console
  3. Perform your query “cs-method:get” with the wanted time range
  4. select the Quick Values widget for the “x-forwarded-for”
  5. Look into the Network Console (scroll a little to make it stop going down) and search for the line with the value file starting with terms?query=[...].
  6. Right click and select Copy as cURL.
  7. Done. You probably need to remove the -H "Authorization:"Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" part and exchange it with the -u option for correct authentication.

Greetings - Phil


#12

Hi Phil.

I may have to rethink my idea of using this list to rate limit the IP addresses for Graylog/IIS.
It’s like whack-a-mole, as soon as you filter/limit one IP address another one pops up in it’s place.
Thank you for the work in helping me.

Regards,

Harry W.