I’m trying to monitor the status of our Graylog sidecars via Graylog’s email notifications, and the only way I could find to do that was by creating a JSON path from HTTP API input pointing to /api/sidecars/all
This returns a bunch of JSON and i’d like to somehow alert on this if either there is a sidecar where active=false, or a sidecar with a collector that is stopped.
I’ve tried just getting the sidecars array from that URI by using .sidecars, .sidecars.*, etc. as the JSON path, but anything other than or a top-level attribute like .active fails
This is the result field of the JSON path from HTTP API input:
{query=, total=8, only_active=false, sort=null, order=null, sidecars=[{“active”:true,“node_id”:“bf09332b-1eb6-4101-ab5a-24ad45176aa9”,“node_name”:“SRV-1”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.239”,“metrics”:{“disks_75”:[“C:\ (86%)”],“cpu_idle”:97.97,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“1 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:22.588Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“dc275dee-e958-4188-a5fb-049b74c06df6”,“node_name”:“SRV-2”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.79”,“metrics”:{“disks_75”:,“cpu_idle”:99.84,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“1 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:26.299Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“96ee608a-6d77-41fc-a9d9-4e809a880f6a”,“node_name”:“SRV-3”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.234”,“metrics”:{“disks_75”:[“C:\ (90%)”],“cpu_idle”:68.41,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“1 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:24.131Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“930423fe-28a6-4a23-bd0d-cba9a5d69b02”,“node_name”:“SRV-4”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.44”,“metrics”:{“disks_75”:[“D:\ (100%)”],“cpu_idle”:98.25,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“2 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b9”,“status”:0,“message”:“Running”,“verbose_message”:""},{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b9”,“configuration_id”:“5e66cbadb525ed04cced0e13”},{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:25.192Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“5b581057-9a9c-4cae-b87d-5f7906f30c0c”,“node_name”:“SRV-5”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.42”,“metrics”:{“disks_75”:[“D:\ (100%)”],“cpu_idle”:96.25,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“2 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b9”,“status”:0,“message”:“Running”,“verbose_message”:""},{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b9”,“configuration_id”:“5e66cbadb525ed04cced0e13”},{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:22.589Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“9b21e619-9934-41d6-a5d3-d936faf8ae11”,“node_name”:“SRV-6”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.149”,“metrics”:{“disks_75”:[“C:\ (91%)”],“cpu_idle”:80.81,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“1 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:24.131Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“98de60bb-23e1-4f27-ae26-f57a503155d9”,“node_name”:“SRV-7”,“node_details”:{“operating_system”:“Windows”,“ip”:“169.254.155.212”,“metrics”:{“disks_75”:[“C:\ (78%)”],“cpu_idle”:90.6,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“0 running / 0 stopped / 0 failing”,“collectors”:}},“assignments”:,“last_seen”:“2021-04-26T23:00:26.510Z”,“sidecar_version”:“1.0.2”,“collectors”:null},{“active”:true,“node_id”:“5ad36022-74ed-4a5d-b02d-5eff9f428d3b”,“node_name”:“SRV-8”,“node_details”:{“operating_system”:“Windows”,“ip”:“192.168.16.40”,“metrics”:{“disks_75”:,“cpu_idle”:98.96,“load_1”:0.0},“log_file_list”:null,“status”:{“status”:0,“message”:“1 running / 0 stopped / 0 failing”,“collectors”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“status”:0,“message”:“Running”,“verbose_message”:""}]}},“assignments”:[{“collector_id”:“5e5d8d29b525ed3a53a115b6”,“configuration_id”:“5e5d98e8b525ed0857ba6582”}],“last_seen”:“2021-04-26T23:00:22.588Z”,“sidecar_version”:“1.0.2”,“collectors”:null}], filters=null, pagination={total=8, count=8, page=1, per_page=8}}
When I try to create a JSON extractor for this I get “nothing would be extracted”, and when I use a regex extractor “[({.*})]” to clean it up first, the JSON extractor only extracts the first element in the sidecars array.
What is the easiest way (be it pipeline rule or otherwise) to iterate over this data and generate an alertable field if any sidecar is either inactive or has collectors that are stopped.
The simplest thing would be to write a separate alert or (god forbid) a separate input for each server, but I want to do this in such a way that the alert will trigger if any current or future sidecars fail; I really don’t want to write a separate rule for every sidecar.