Alerts not firing, alert filter not showing results

Graylog Central (peer support)
1

1. Describe your incident:

Created a SSH brute force attempt alert

very simple filter, any message from Syslog stream with “Invalid user” should fire slack notification

Search on the Syslog stream shows hits

Alert definition is showing 0 hits though

Input incoming messages show the hits as well

I dont see any error messages from my GL docker stack

2. Describe your environment:

  • OS Information: Rocky 9 linux (EC2 instance - 32G ram, 8 CPU), running Graylog stack w Opensearch via docker compose, using about 12G of 32G of RAM

  • Package Version:
    Graylog 5.1.6
    Opensearch 2.4.0
    Mongo 5.0

  • Service logs, configurations, and environment variables:

opensearch:
    image: "opensearchproject/opensearch:2.4.0"
    environment:
      - "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g"
      - "bootstrap.memory_lock=true"
      - "discovery.type=single-node"
      - "action.auto_create_index=false"
      - "processbuffer_processors=10"
      - "plugins.security.ssl.http.enabled=false"
      - "plugins.security.disabled=true"

Graylog envs

    environment:
      GRAYLOG_NODE_ID_FILE: "/usr/share/graylog/data/config/node-id"
      GRAYLOG_PASSWORD_SECRET: "${GRAYLOG_PASSWORD_SECRET:?Please configure GRAYLOG_PASSWORD_SECRET in the .env file}"
      GRAYLOG_ROOT_PASSWORD_SHA2: "${GRAYLOG_ROOT_PASSWORD_SHA2:?Please configure GRAYLOG_ROOT_PASSWORD_SHA2 in the .env file}"
      GRAYLOG_HTTP_BIND_ADDRESS: "0.0.0.0:9000"
      GRAYLOG_HTTP_EXTERNAL_URI: "http://localhost:9000/"
      GRAYLOG_ELASTICSEARCH_HOSTS: "http://opensearch:9200"
      GRAYLOG_MONGODB_URI: "mongodb://mongodb:27017/graylog"
      GRAYLOG_SERVER_JAVA_OPTS: "-Xms16g -Xmx16g"
      GRAYLOG_TIMEZONE: "America/New_York"
      TZ: "America/New_York"

3. What steps have you already taken to try and solve the problem?

restarted GL docker stack

4. How can the community help?

Not sure how to troubleshoot this, Its not an issue with Slack connector, I ran a test alert and it works, the Alert filter is not picking up the Search query

2

Hi @perfecto25, your query is not matching as written. Has that event happened in the last 30 minutes? If not, it won’t appear on the right.

Test by using a standard search with the same query and see if you get any matches.

It’s always possible that the event condition is fine, but that those events just haven’t come in yet.

3

the search did match the alert.

weird, i ran the alert every 30 sec with lookback time of 30sec, I upped this to 2 min, with lookback 2min, and now its firing.

Not sure if its hitting some resource cap with 30 sec interval

4

I agree that 30 seconds is a very short window. Most people seem to use around five minutes. It shouldn’t make a difference, but if you are overloading your hardware, it may be taking longer than 30 seconds for events to get collected and processed and then the queries keep turning up empty because there’s nothing in that 30 second window.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.