Extract JSON from Graylog HTTP API

I’m trying to monitor the status of our Graylog sidecars via Graylog’s email notifications, and the only way I could find to do that was by creating a JSON path from HTTP API input pointing to /api/sidecars/all

This returns a bunch of JSON and i’d like to somehow alert on this if either there is a sidecar where active=false, or a sidecar with a collector that is stopped.

I’ve tried just getting the sidecars array from that URI by using .sidecars, .sidecars.*, etc. as the JSON path, but anything other than or a top-level attribute like .active fails

This is the result field of the JSON path from HTTP API input:
{query=, total=8, only_active=false, sort=null, order=null, sidecars=[{“active”:true,“node_id”:“bf09332b-1eb6-4101-ab5a-24ad45176aa9”,“node_name”:“SRV-1”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.239”,“metrics”:{“disks_75”:[“C:\ (86%)”],“cpu_idle”:97.97,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“1 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:22.588Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“dc275dee-e958-4188-a5fb-049b74c06df6”,“node_name”:“SRV-2”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.79”,“metrics”:{“disks_75”:,“cpu_idle”:99.84,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“1 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:26.299Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“96ee608a-6d77-41fc-a9d9-4e809a880f6a”,“node_name”:“SRV-3”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.234”,“metrics”:{“disks_75”:[“C:\ (90%)”],“cpu_idle”:68.41,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“1 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:24.131Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“930423fe-28a6-4a23-bd0d-cba9a5d69b02”,“node_name”:“SRV-4”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.44”,“metrics”:{“disks_75”:[“D:\ (100%)”],“cpu_idle”:98.25,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“2 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b9”,“status”:0,“message”:“Running”,“verbose_message”:""},{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b9”,“configuration_id”:“5e66cbadb525ed04cced0e13”},{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:25.192Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“5b581057-9a9c-4cae-b87d-5f7906f30c0c”,“node_name”:“SRV-5”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.42”,“metrics”:{“disks_75”:[“D:\ (100%)”],“cpu_idle”:96.25,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“2 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b9”,“status”:0,“message”:“Running”,“verbose_message”:""},{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b9”,“configuration_id”:“5e66cbadb525ed04cced0e13”},{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:22.589Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“9b21e619-9934-41d6-a5d3-d936faf8ae11”,“node_name”:“SRV-6”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.149”,“metrics”:{“disks_75”:[“C:\ (91%)”],“cpu_idle”:80.81,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“1 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:24.131Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“98de60bb-23e1-4f27-ae26-f57a503155d9”,“node_name”:“SRV-7”,“node_details”:{“operating_system”:“Windows”,“ip”:“169.254.155.212”,“metrics”:{“disks_75”:[“C:\ (78%)”],“cpu_idle”:90.6,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“0 running / 0 stopped / 0 failing”,“collectors”:}},“assignments”:,“last_seen”:“2021-04-26T23:00:26.510Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“5ad36022-74ed-4a5d-b02d-5eff9f428d3b”,“node_name”:“SRV-8”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.40”,“metrics”:{“disks_75”:,“cpu_idle”:98.96,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“1 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:22.588Z”,“sidecar_version”:“1.0.2”,“collectors”:null}], filters=null, pagination={total=8, count=8, page=1, per_page=8}}

When I try to create a JSON extractor for this I get “nothing would be extracted”, and when I use a regex extractor “[({.*})]” to clean it up first, the JSON extractor only extracts the first element in the sidecars array.

What is the easiest way (be it pipeline rule or otherwise) to iterate over this data and generate an alertable field if any sidecar is either inactive or has collectors that are stopped.

The simplest thing would be to write a separate alert or (god forbid) a separate input for each server, but I want to do this in such a way that the alert will trigger if any current or future sidecars fail; I really don’t want to write a separate rule for every sidecar.

If I were you I would use a NMS system like Zabbix to get status of sidecars via API and send e-mail alert if not available.

1 Like

That would require me to both set up Zabbix, and it doesn’t solve the problem of alerting for future sidecars as well.

I don’t want to have to remember to set up a new alert for each new sidecar, I want to parse the results of /api/sidecars/all so I can alert on all sidecars.

Zabbix can automatically create items for monitoring by parsing json output of graylog Rest API using http agent item and LLD. You don’t need to create them manually.

https://www.zabbix.com/documentation/5.0/manual/discovery/low_level_discovery

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.