Widget Querying (aggregation) with Python

Templates and Rules Exchange Miscellaneous
1

This thread outlines how to query aggregated Graylog data with Python [the spoon-feeding I was hoping to find :slight_smile: ]. There may be better methods, but my script completes in a few seconds and gets me everything I need. Dashboard queries may also be an option, but I do not know how to accomplish those or if they are possible.

First I’d like to point out the authentication obstacle I faced: Create a new user as a service account (System > Users and Teams). Create a token for account and grant the account admin rights (none of my queries worked until admin rights were granted).

Create and save new Graylog search with all the parameters in which you are interested (add whatever message count and/or aggregation tables (widgets) you need). My code assumes a message count widget and an aggregation widget that outputs ip and ip count.

As @gsmith had suggested:
Go to API browser (System > Nodes > API Browser). Find (
Search/Saved: > Show/Hide > Try it out!)
Ctrl-F whatever you named your search in the results. From there you can find the “search_id” and “state” identifiers you’ll need for your query. You’ll need some additional identifiers as noted in the code.

For clarity, the functional, stand-alone code will be posted in its own message.

Acknowledgments:

I like thank @gsmith’s suggestion which eventually lead me to a thread from @Le-DOC which pointed me in the right direction:

Original post: Querying aggregated Graylog data with Python

–version 4.3.5

2 Likes
2 
import requests
import json

################################## CHANGE THESE VALUES ###################################################
myIP_Port = 'XXX.XXX.XXX.XXX:XXXX'
graylog_token = '11k51vt3efqq5eq7fvho448jelhn49u54s5cveb4aknlvtdjliei'
#FIND THE FOLLOWING FROM /api/api-browser/#!/Views/views_get_1 -> Ctrl-F your saved search's name
search_id = '62e1f883f74e2d1cce726e85'
state = 'a1647eb6-a064-4fe6-b459-1e4267d3f659'
#WIDGET MAPPING ARE ALSO FOUND IN VIEWS, BUT MAY BE FOUND EASIER BY POKING THROUGH JSON RESPONSE
widget_mapping_message_count = '59f4fa97-c3fe-40a0-868e-f78d67b4870f'
widget_mapping_aggregation = '47ba6e34-cb53-4c64-8a0e-318c038d1dd0'
##########################################################################################################

apiUrl = 'http://' + myIP_Port + '/api/views/search/' + search_id + '/execute'

headers = {
    'Accept': 'application/json',
    'Content-Type': 'application/json',
    'X-Requested-By': 'XMLHttpRequest',
}

response = requests.post(apiUrl, headers=headers, auth=(graylog_token, 'token'))
jeepsNorSons = response.json()

#GRAB MESSAGE COUNT
messageCount = jeepsNorSons['results'][state]['search_types'][widget_mapping_message_count]['total']

#GRAB TABLE
aggregation = jeepsNorSons['results'][state]['search_types'][widget_mapping_aggregation]['rows']

#THIS LOOP IS FOR DEMONSTRATION - MAY VARY FOR YOUR TABLE(S)
for i in aggregation:
    if len(i['key']):
        #print(i) #UNCOMMENT THIS TO EXPLORE
        ip = i['key'][0]
        count = i['values'][0]['value']
        print(ip + " - " + str(count))
2 Likes
3

Thank you for posting, Much appreciated :+1: