Search in Alerts & Events behaving unexpectedly

rivad · May 12, 2025, 7:42pm

I’m setting up GitHub - sowoi/graylog-alerts-to-icinga: Monitor graylog alerts with icinga2 to get Graylog events/alterts into Icinga and got into a fight with the search at “Alerts & Event” page and the query at /api/api-browser/global/index.html#!/Events/search_post_2.

I have alerts with descriptions (or titles?) like this:

HE PDF 4 heights errors
HE PDF 4 heights time-server error
HE hearchivp write errors

Some of my experiments:

If I enter description:HE PDF 4 heights errors into the search I get all 3.
If I enter description:HE I get nothing.
If I enter description:HE time-server it get 2. and NO_LEADER: There was no leader Graylog server node detected in the cluster.
If I enter description:whatever HE -"time-server" I get 1. & 3.
If I enter description:"HE PDF 4 heights errors" I get nothing.
If I enter description:"time-server" I get nothing.
If I enter description:graylog I get nothing.
If I enter description:whatever graylog I get OUTDATED_VERSION: You are running an outdated Graylog version and NO_LEADER: There was no leader Graylog server node detected in the cluster

So, I can’t search for an exact match and the first word directly after the colon gets ignored - how strange!

Can someone please explain this search/query behavior to me?

May I also suggest, that the tooltip and the documentation get improved?

The tooltip specifies the following and from this is expected description:HE and description:"time-server" and description:graylog to work:

Search Syntax Help

Available search fields

|Field|Description|
| — | — |
|id|Id of the event, which is a unique reference.|
|title|Title of the event.|
|description|Short description of the event.|
|key|The key of the event|

Examples

Find all events with a description containing security:
description:security

Find a event with the id 5f4dfb9c69be46153b9a9a7b:
id:5f4dfb9c69be46153b9a9a7b

The documentation under Alerts has only this

Search Panel
At the top of the page, you’ll find a search bar designed to help you search through events by using keywords. To further drill down on search results, you can use filters to find specific events or alerts.

My Environment:

OS Information: Debian 12
Package Version: 6.2.1

Many thanks to anyone who can shed some light onto this matter.

patrickmann · May 13, 2025, 7:59am

Hi - I think there is some confusion about the search box on the UI and the search API.
The UI provides search based on a limited set of fields and only filters down the data that’s already loaded in the browser. The API queries the backend directly.
Are you using the API? Then please share the full JSON request data.

I will follow-up on the UI search behavior. I agree it is puzzling why a substring returns nothing, when the full search string returns entries.

rivad · May 13, 2025, 8:56am

I tried to use the UI search box to quickly test the queries to use in the Icinga check that uses the API. I also use the Graylog REST API browser to figure out, what’s going on.

Doesn’t looks like they are separate to me as, for example:

{"sort_direction": "desc", "timerange": { "type": "relative", "from" : "864000"},  "query": "description:he \"time-server\"",  "sort_by": "timestamp"}

Results in:

{
  "events": [
    {
      "event": {
        "id": "01JTXCXRGSFM2M3E25E3HMHC5C",
        "event_definition_type": "aggregation-v1",
        "event_definition_id": "66d5d3117cc5852aad4b50a1",
        "origin_context": "urn:graylog:message:es:lfops-default_741:0f7ab020-2db3-11f0-985e-005056a1bf10",
        "timestamp": "2025-05-10T15:25:52.052Z",
        "timestamp_processing": "2025-05-10T15:27:22.649Z",
        "timerange_start": null,
        "timerange_end": null,
        "streams": [
          "000000000000000000000002"
        ],
        "source_streams": [
          "64804345385b9b5903fc2f82"
        ],
        "message": "HE PDF 4 heights time-server error",
        "source": "localhost",
        "key_tuple": [],
        "key": null,
        "priority": 2,
        "scores": {},
        "associated_assets": [],
        "alert": true,
        "fields": {},
        "group_by_fields": {},
        "replay_info": {
          "timerange_start": "2025-05-10T15:22:16.200Z",
          "timerange_end": "2025-05-10T15:27:16.200Z",
          "query": "source:icthepdf* AND message:\"response from time-stamp server\"",
          "streams": [
            "64804345385b9b5903fc2f82"
          ],
          "filters": []
        }
      },
      "index_name": "gl-events_25",
      "index_type": "message"
    },
    {
      "event": {
        "id": "01JTXCXRGSB51MXGA34Z5KCEQT",
        "event_definition_type": "aggregation-v1",
        "event_definition_id": "66d5d3117cc5852aad4b50a1",
        "origin_context": "urn:graylog:message:es:lfops-default_741:0f7ab021-2db3-11f0-985e-005056a1bf10",
        "timestamp": "2025-05-10T15:25:52.052Z",
        "timestamp_processing": "2025-05-10T15:27:22.649Z",
        "timerange_start": null,
        "timerange_end": null,
        "streams": [
          "000000000000000000000002"
        ],
        "source_streams": [
          "64804345385b9b5903fc2f82"
        ],
        "message": "HE PDF 4 heights time-server error",
        "source": "localhost",
        "key_tuple": [],
        "key": null,
        "priority": 2,
        "scores": {},
        "associated_assets": [],
        "alert": true,
        "fields": {},
        "group_by_fields": {},
        "replay_info": {
          "timerange_start": "2025-05-10T15:22:16.200Z",
          "timerange_end": "2025-05-10T15:27:16.200Z",
          "query": "source:icthepdf* AND message:\"response from time-stamp server\"",
          "streams": [
            "64804345385b9b5903fc2f82"
          ],
          "filters": []
        }
      },
      "index_name": "gl-events_25",
      "index_type": "message"
    },
    {
      "event": {
        "id": "01JTXCXRGSBS4GQ9B09P1SCQG0",
        "event_definition_type": "aggregation-v1",
        "event_definition_id": "66d5d3117cc5852aad4b50a1",
        "origin_context": "urn:graylog:message:es:lfops-default_741:ad07bf02-2db2-11f0-985e-005056a1bf10",
        "timestamp": "2025-05-10T15:23:06.883Z",
        "timestamp_processing": "2025-05-10T15:27:22.649Z",
        "timerange_start": null,
        "timerange_end": null,
        "streams": [
          "000000000000000000000002"
        ],
        "source_streams": [
          "64804345385b9b5903fc2f82"
        ],
        "message": "HE PDF 4 heights time-server error",
        "source": "localhost",
        "key_tuple": [],
        "key": null,
        "priority": 2,
        "scores": {},
        "associated_assets": [],
        "alert": true,
        "fields": {},
        "group_by_fields": {},
        "replay_info": {
          "timerange_start": "2025-05-10T15:22:16.200Z",
          "timerange_end": "2025-05-10T15:27:16.200Z",
          "query": "source:icthepdf* AND message:\"response from time-stamp server\"",
          "streams": [
            "64804345385b9b5903fc2f82"
          ],
          "filters": []
        }
      },
      "index_name": "gl-events_25",
      "index_type": "message"
    },
    {
      "event": {
        "id": "01JTXCXRGS4DE4G87D995CMXTW",
        "event_definition_type": "aggregation-v1",
        "event_definition_id": "66d5d3117cc5852aad4b50a1",
        "origin_context": "urn:graylog:message:es:lfops-default_741:ad07bf04-2db2-11f0-985e-005056a1bf10",
        "timestamp": "2025-05-10T15:23:06.883Z",
        "timestamp_processing": "2025-05-10T15:27:22.649Z",
        "timerange_start": null,
        "timerange_end": null,
        "streams": [
          "000000000000000000000002"
        ],
        "source_streams": [
          "64804345385b9b5903fc2f82"
        ],
        "message": "HE PDF 4 heights time-server error",
        "source": "localhost",
        "key_tuple": [],
        "key": null,
        "priority": 2,
        "scores": {},
        "associated_assets": [],
        "alert": true,
        "fields": {},
        "group_by_fields": {},
        "replay_info": {
          "timerange_start": "2025-05-10T15:22:16.200Z",
          "timerange_end": "2025-05-10T15:27:16.200Z",
          "query": "source:icthepdf* AND message:\"response from time-stamp server\"",
          "streams": [
            "64804345385b9b5903fc2f82"
          ],
          "filters": []
        }
      },
      "index_name": "gl-events_25",
      "index_type": "message"
    },
    {
      "event": {
        "id": "01JTXCMK06XJQ0EPMR7B850WS4",
        "event_definition_type": "aggregation-v1",
        "event_definition_id": "66d5d3117cc5852aad4b50a1",
        "origin_context": "urn:graylog:message:es:lfops-default_741:6d71d297-2db2-11f0-985e-005056a1bf10",
        "timestamp": "2025-05-10T15:21:20.205Z",
        "timestamp_processing": "2025-05-10T15:22:22.086Z",
        "timerange_start": null,
        "timerange_end": null,
        "streams": [
          "000000000000000000000002"
        ],
        "source_streams": [
          "64804345385b9b5903fc2f82"
        ],
        "message": "HE PDF 4 heights time-server error",
        "source": "localhost",
        "key_tuple": [],
        "key": null,
        "priority": 2,
        "scores": {},
        "associated_assets": [],
        "alert": true,
        "fields": {},
        "group_by_fields": {},
        "replay_info": {
          "timerange_start": "2025-05-10T15:17:16.200Z",
          "timerange_end": "2025-05-10T15:22:16.200Z",
          "query": "source:icthepdf* AND message:\"response from time-stamp server\"",
          "streams": [
            "64804345385b9b5903fc2f82"
          ],
          "filters": []
        }
      },
      "index_name": "gl-events_25",
      "index_type": "message"
    },
    {
      "event": {
        "id": "01JTXCMK06K4QQRWK10M18XYGA",
        "event_definition_type": "aggregation-v1",
        "event_definition_id": "66d5d3117cc5852aad4b50a1",
        "origin_context": "urn:graylog:message:es:lfops-default_741:6d71f9a0-2db2-11f0-985e-005056a1bf10",
        "timestamp": "2025-05-10T15:21:20.205Z",
        "timestamp_processing": "2025-05-10T15:22:22.086Z",
        "timerange_start": null,
        "timerange_end": null,
        "streams": [
          "000000000000000000000002"
        ],
        "source_streams": [
          "64804345385b9b5903fc2f82"
        ],
        "message": "HE PDF 4 heights time-server error",
        "source": "localhost",
        "key_tuple": [],
        "key": null,
        "priority": 2,
        "scores": {},
        "associated_assets": [],
        "alert": true,
        "fields": {},
        "group_by_fields": {},
        "replay_info": {
          "timerange_start": "2025-05-10T15:17:16.200Z",
          "timerange_end": "2025-05-10T15:22:16.200Z",
          "query": "source:icthepdf* AND message:\"response from time-stamp server\"",
          "streams": [
            "64804345385b9b5903fc2f82"
          ],
          "filters": []
        }
      },
      "index_name": "gl-events_25",
      "index_type": "message"
    }
  ],
  "used_indices": [
    "gl-events_25",
    "gl-system-events_18"
  ],
  "parameters": {
    "page": 1,
    "per_page": 10,
    "timerange": {
      "from": 864000,
      "type": "relative"
    },
    "query": "description:he \"time-server\"",
    "filter": {
      "alerts": "include",
      "event_definitions": [],
      "priority": [],
      "aggregation_timerange": null,
      "key": [],
      "id": [],
      "extra_filters": {}
    },
    "sort_by": "timestamp",
    "sort_direction": "desc",
    "sort_unmapped_type": null
  },
  "total_events": 6,
  "duration": 4,
  "context": {
    "event_definitions": {
      "66d5d3117cc5852aad4b50a1": {
        "id": "66d5d3117cc5852aad4b50a1",
        "title": "HE PDF 4 heights time-server error",
        "description": "PDF-Tools 4heights time-server not reached",
        "remediation_steps": null
      }
    },
    "streams": {
      "000000000000000000000002": {
        "id": "000000000000000000000002",
        "title": "All events",
        "description": "Stream containing all events created by Graylog",
        "remediation_steps": null
      }
    }
  }
}

And

{"sort_direction": "desc", "timerange": { "type": "relative", "from" : "864000"},  "query": "description:PDF",  "sort_by": "timestamp"}

Results in:

{
  "events": [],
  "used_indices": [
    "gl-events_25",
    "gl-system-events_18"
  ],
  "parameters": {
    "page": 1,
    "per_page": 10,
    "timerange": {
      "from": 864000,
      "type": "relative"
    },
    "query": "description:PDF",
    "filter": {
      "alerts": "include",
      "event_definitions": [],
      "priority": [],
      "aggregation_timerange": null,
      "key": [],
      "id": [],
      "extra_filters": {}
    },
    "sort_by": "timestamp",
    "sort_direction": "desc",
    "sort_unmapped_type": null
  },
  "total_events": 0,
  "duration": 2,
  "context": {
    "event_definitions": {},
    "streams": {}
  }
}

Which is consistent with the web view.

Topic		Replies	Views
Question regarding alerts and monitoring Graylog Central (peer support)	11	759	November 27, 2019
[Graylog_2.3.2] No result in search Graylog Central (peer support)	8	1220	January 19, 2018
Event definitions not create alerts Graylog Central (peer support) alert	7	650	June 27, 2022
Alerts & Events empty "No Events found for the current search criteria." Graylog Central (peer support) alert	5	901	October 7, 2022
Alert log via internal log or by rsyslog Graylog Central (peer support)	10	1426	May 28, 2019

Search in Alerts & Events behaving unexpectedly

Related topics