Alerts/Events do not trigger

Graylog Central (peer support)
41

I just startet working graylog at the moment I try 5/min

2019-09-17T09:41:03.085Z INFO [InputStateListener] Input [Random HTTP message generator/5d80aa2ed2ab0e04ff8f7700] is now STARTING
2019-09-17T09:41:03.108Z INFO [InputStateListener] Input [Random HTTP message generator/5d80aa2ed2ab0e04ff8f7700] is now RUNNING
2019-09-17T09:41:03.249Z INFO [connection] Opened connection [connectionId{localValue:15, serverValue:34}] to localhost:27017
2019-09-17T09:41:03.249Z INFO [connection] Opened connection [connectionId{localValue:16, serverValue:35}] to localhost:27017
2019-09-17T09:41:03.250Z INFO [connection] Opened connection [connectionId{localValue:12, serverValue:31}] to localhost:27017
2019-09-17T09:41:03.250Z INFO [connection] Opened connection [connectionId{localValue:14, serverValue:33}] to localhost:27017
2019-09-17T09:41:03.250Z INFO [connection] Opened connection [connectionId{localValue:11, serverValue:30}] to localhost:27017
2019-09-17T09:41:03.250Z INFO [connection] Opened connection [connectionId{localValue:13, serverValue:32}] to localhost:27017
2019-09-17T09:43:12.072Z INFO [InputStateListener] Input [Random HTTP message generator/5d80aa2ed2ab0e04ff8f7700] is now STOPPING
2019-09-17T09:43:12.107Z INFO [InputStateListener] Input [Random HTTP message generator/5d80aa2ed2ab0e04ff8f7700] is now TERMINATED
2019-09-17T09:43:12.108Z INFO [InputStateListener] Input [Random HTTP message generator/5d80aa2ed2ab0e04ff8f7700] is now STOPPED
2019-09-17T09:43:12.112Z INFO [InputStateListener] Input [Random HTTP message generator/5d80aa2ed2ab0e04ff8f7700] is now STARTING
2019-09-17T09:43:12.115Z INFO [InputStateListener] Input [Random HTTP message generator/5d80aa2ed2ab0e04ff8f7700] is now RUNNING

output:
root@logsrv01:/home/hh# echo ‘db.processing_status.find()’ | mongo
MongoDB shell version v4.0.12
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“dcc342a0-47c8-480f-80e8-cf7d662e3797”) }
MongoDB server version: 4.0.12
bye
root@logsrv01:/home/hh#

42

this command is incomplete. you need to specify the name of your database. (default is graylog)
echo ‘db.processing_status.find()’ | mongo graylog

43

root@logsrv01:/home/hh# echo ‘db.processing_status.find()’ | mongo graylog
MongoDB shell version v4.0.12
connecting to: mongodb://127.0.0.1:27017/graylog?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“11cbeebd-4049-488b-8f9e-addd595e415a”) }
MongoDB server version: 4.0.12
{ “_id” : ObjectId(“5d72786f125fefc15ea1f45e”), “node_id” : “ef886ea0-70e4-4419-ab19-85ca6fd5457c”, “node_lifecycle_status” : “RUNNING”, “updated_at” : ISODate(“2019-09-17T09:52:54.854Z”), “receive_times” : { “ingest” : ISODate(“2019-09-17T09:52:54.817Z”), “post_processing” : ISODate(“2019-09-17T09:52:54.817Z”), “post_indexing” : ISODate(“2019-09-17T09:52:52.985Z”) }, “input_journal” : { “uncommitted_entries” : NumberLong(71), “read_messages_1m_rate” : 39.34059460793546, “written_messages_1m_rate” : 39.34059448504912 } }
bye
root@logsrv01:/home/hh#

44

In my server config file I have
processing_status_journal_write_rate_threshold=0

45

I changed Search within the last 5 min
Execute search every 1 secounds

and post following http:
curl -XPOST http://myserver.dk:12201/gelf -p0 -d ‘{“short_message”:“Hello hashed -----13:45------- “,”_Response_code”:400, “host”:“example.org”, “facility”:“test”, “_foo”:“bar”}’

I get somtimes 5 emails
I get emails which contains timestamp for 4 days ago
---------------------------- email response contains --------------------
— [Event Definition] ---------------------------
Title: response 400 new event
Description: response 200 event
Type: aggregation-v1
— [Event] --------------------------------------
Timestamp: 2019-09-13T06:49:22.074Z
Message: response 400 new event
Source: myserver.dk
Key:
Priority: 3
Alert: true
Timestamp Processing: 2019-09-13T06:49:22.074Z
Timerange Start:
Timerange End:
Fields:

— [Backlog] ------------------------------------
Last messages accounting for this alert:

{index=graylog_0, message=short message HASHED.4, timestamp=2019-09-13T06:49:22.074Z, fields={Path=/originals, Response_code=400, level=7, Customer=123, line=26, gl2_remote_ip=192.168.1.68, gl2_remote_port=60215, gl2_message_id=01DMMP4ACWYN7SX727BBZZEX41, Service=public-api, Time=100, gl2_source_input=5d761808d2ab0e05cdcf5539, Type=http request, full_message=full_message - HASHED.4, Verb=POST, Contextid=1, Environment=Dev, gl2_source_node=ef886ea0-70e4-4419-ab19-85ca6fd5457c}, id=9dfa4fa1-d5f2-11e9-9a03-005056a45128, source=HASHED.4, stream_ids=[5d7967d0d2ab0e1e1e966994, 000000000000000000000001]}
----------------------------end email --------------

is somting wrong width my timestamp set up ( debean and gray log is UTC)

46

Yeah i see them when i search way back to the 12th. (we are now at 13th)
regarding the 1 minute chunks … any way to speed this up ? or delete them all?

Thanks in advance and all the help

47

We are aware that this is problem and will address this sooner or later.
I think the easiest way to make it catch up, is to delete the Event Definition and create a new identical one.
That one should start processing from the current time onwards.

48

Heay
I did what you asked. There are no events in alert page but sometimes I get emails and sometimes not.
Is there any Graylag version in the past for Debian, which everything worked without problem (Version 2.X)? if yes do you have installitions url
Thanhs

49

Hello, please give an example how to configure http notification for slack now?

50

Hello,
Same problem here:
graylog 3.1.2 no events triggered

51

Having the same issue as well on 3.1.2. Alerts were working when we did initial testing with Graylog at version 3.0 a couple months back. We will receive email notifications for our alerts at random times or just not at all.

52

Exactly the same issue

53

@datamans could you please open a new thread - on such a long conversation just adding “I have the same issue” without writing what workarounds that are already given you have tried already and what your current status is does not help at all.

@kzimmerm what exactly did you already tried and what is the current result? For you the same as @datamans. while you could reference this posting in your own it does not really is helpful - because how should anybody from the outside know what your current state is and what you have already tried?

54

I have tried adding message_journal_enabled = false to the idle nodes but no dice.
I’ve tried processing_status_journal_write_rate_threshold = 0 and that did not help.

Should both of those parameters only exist on the master graylog node?

We have a 4 node cluster on CentOS 7.5.
Mongodb 4.0.12-1
Elasticsearch 6.8.3-1
Graylog 3.1.2

Right now we are building out a test cluster with version 3.0 until this can be resolved.

55

I guess that it is more that you do not have enough messages in the test cluster that the alerting is not triggered because of the low ingest rate in the test cluster @kzimmerm

But that is just guessing.

56

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.