Events, alerts fired only after a second message arrives

Hi everyone,

I’ve been working with Graylog 3.1.1 and have this issue (i will tell onward) so i upgraded to the latest 3.2.4 but still don’t get what it’s wrong:

  • only 1 node
    I receive syslog messages from routers, so i’m testing sending syslog messages using ‘logger’ from Ubuntu directly to the server with graylog to fire up Events (and then Alarms).
    The syslog messages are received OK, no problem with that, BUT the Event is not shown, then i can’t trigger Alarms…UNTIL i receive a second message.

Let me explain this:

  1. i send a first message with a text naming a fake BGP protocol problem (containing the text “BGP neighbor”)
  2. that message is received and routed into 2 Streams: All Messages and “BGP flaps”
  3. NO Event is shown…
  4. i send a 2nd message that has nothing to do with the first one, it is routed into “All messages” stream) and then the Event related to the previous message is shown (in Alerts/Events)

any idea or tip? what could be wrong with my setup?

Thank you very much!
Max!

he @maxplay

did you have regular inputs that receive messages?

How did you configure the event definitions?

Graylog will run the events definition search only if it receive messages for/in the given search window of the definition. If no messages are received at all no event search is done.

Hi @jan how are you?
Mi input is “syslog udp” and i receive messages normally.

Here is part of the Event definition:

(i didn’t set: Event Fields, Notifications)

Please, i don’t see what could be wrong. As i said before, the messages are displayed ok, but no event is shown, when a New message arrives (some minutes later than the 1st one) then the Event is shown.

Let me know if you need more information (print screen, or whatever)
thank you very much!

@jan maybe the problem here is also related to the one @DerBeton (found solution?) posted some days ago…they seem to be similar…

Hey @maxplay
I haven’t found a better solution yet. Means I’m still using the Random HTTP Message Generator from Graylog to create Random messages every 30 seconds.

With this I have at least enough traffic so that Graylog detects events correctly.
I’ve also created a custom Stream for this messages and a Index where they are stored.
This Index stores a maximum of 7200 messages, which prevents my Storage from filling up with unnecessary logs.

It’s not a beautiful solution but at least my critical Events are fired. Hopefully this issue gets fixed sometime. Most people probably won’t have this problem, because there servers are pretty busy and therfore they receive more than 1 log/minute.

he @DerBeton, @maxplay

that is true - this is an absolute edge case. We wanted to reduce not needed work and so Graylog checks if it has received messages in the period it should search in. Because if not, the event search will not be done. That a graylog does not receive that much messages is mostly seen only in POC or Test setups. Others have mostly a minimal stream of 1-10 messages per second. I already have that in my home lab setup where some servers and DNS traffic is send to…

From our point of view this is not a bug, it is more a feature request to run the event search anyways. But not sure if that is possible from a code perspective.

Hi @jan , @DerBeton

Thanks for your comments!

I’m not sure if it is the exactly the same problem we both have: in my case the message arrives OK, and no Event fired, then, after some time, let’s say wait 5 minutes (more than 1 min), receive a new message and the Event related to the first one is fired.

So it’s kind of “the Event was stored there -kind of hidden- and waiting to be shown”

I’ll try @DerBeton solution (…i wonder if this would work if i route those HTTP messages into a Pipeline and apply a rule to drop them instead of storing an X quantity of them…will try…).

@jan other thing i saw is that if you make a Search and ONLY ONE message is displayed/found the Graph that shows the message count in the timeline, doesn’t show anything at all. When more than one message is found (later) you can see both bars in the graph (so: 1 message = no bars, 2 messages separated in time = bars appears in the graph -size one ok-)

I’ll comment later, thank you very much!

Hi again,

Well, as i thought, routing those Random HTTP messages into a Stream and then into Pipeline and rule droping:

  • drops the message
  • makes the messages aren’t “get/take” (?) by ElasticSearch
  • so: no effect with Events…

So, i’m doing @DerBeton solution: WORKS OK!
Is there a way to “not show” that Stream (the one that gets the Random messages) into the Default Search screen?
(i would like to hide those messages)

I know can do this: Search with:
"NOT source:the_name_of_your_source" (as i override the name in the input)
BUT, i have to type that every time, i want to make it or see it automatically

I found i can accomplish the same (without using the HTTP random messages) this way:

(1st: Stop the input of Random HTTP Messages)

  • Create a Pipeline that executes a rule to create a new message (using create_message) and route_message

    • the message should be delayed, so i will have to add a delay to the actual timestamp (how?)
  • Test to see if it works (if the Event is triggered) --> if the message isn’t delayed, it won’t work (tested), need to investigate how to add “times” or make a delay or “wait time”

  • Edit the Pipeline:

    • create a new Stage inside the pipeline that will run a Rule to delete the NewMessage

@jan suggestions?

thanks!

Hey @maxplay thanks for your update.

So your idea to drop the message in the pipeline is quite good but in this case I don’t have to implement this if you say it doesen’t work.

Actually I think we have exactly the same issue because when a message enters graylog after the log which should fire an alert, it works. That’s why I created these Random Inputs.

If you have some news about your new approach please let me know. I’m quite busy right now but maybe I have time for a closer look next week or so.

right @DerBeton, it didn’t work

@jan is there a way to make a rule send a message with a defined delay?
let’s say the rule is executing, then it sleep for some seconds, then execute create_message and finally send the new message (that will trigger the Event)

i don’t see any way to make the delay, no while, for, or sleep() to use in there…(i couldn’t sum or add time to the timestamp and i think that idea won’t work).

thank you very much!
Max!

Hi @jan how are you?

To sum up, i have 2 questions:

  1. is there a way to generate and send a message with a defined delay after you receive a message?

  2. is there a way to log to STDOUT to a file other than server.log?

Thank you very much!

Max!

So, i’m doing @DerBeton solution: WORKS OK!
Is there a way to “not show” that Stream (the one that gets the Random messages) into the Default Search screen?
(i would like to hide those messages)

Do not use the “admin” search but use the stream search …

It is no way given to create “delayed” messages. But you could write a shell script (or similar that is just creating a single message a minute).

  1. is there a way to generate and send a message with a defined delay after you receive a message?

No - only the script notification could be used for that. As you can fire one script as notification in the events. That script can be anything …

  1. is there a way to log to STDOUT to a file other than server.log?

Sure - just modify the log4j2 configuration for your needs.

@jan, thanks, about your answers…i have a few questions:

ok, but first i have to select all the Streams…i was trying to do a “default search” when i press “Search” in the top bar

Is there a “script Notification”? if there is …i don’t see it within Notifications. Or you mean write a script outside graylog?

Note that i just want the output of some Streams not all the loggin of graylog…

So:
My intention was to write the output of a Stream to a new “file.log” and with a bash script read that file every 30 secs or 1 minute, and if the content is what i expect then send a message to a graylog input.

he @maxplay

your intention will not work with a vanilla Graylog. Because “output to file” is not given. The log4j is to configure the logging behaviour of Graylog itself.

The script notification plugin ( https://docs.graylog.org/en/3.2/pages/alerts.html#legacy-script-alert-notification ) is one additional plugin and part of the enterprise plugins.

I guess the idea that you have is not working. I would think more how you can have regular ingested messages into Graylog.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.