The configured event was not setting off any alarms
Even so, I was also having endlessly many email notifications every time the alarm SHOULD have been triggered.
I had to also search within the past 5+ hours to see any events in search or in event definition filter.
It looks like this was all due to the graylog server time zone being in UTC
I knew it was set to be in UTC by default, but I thought syncing the linux server to a local NTP server would resolve that. Which it did for the linux time, just not for the graylog time.
Windows 10 Event Logs were being collected via the SolarWinds log forwarder(super easy to use and setup by the way) and the time stamp being applied were the Windows system times (EST in 24 hour format)
To resolve this do the following
- Open “server.conf” in your favorite text editor
- Look for “root_timezone = UTC”
- Remove the comment out (#) from the beginning
- replace UTC with your own time zone listed here
- Save and exit the text editor
- Restart the graylog server service (not sure if this is necessary, but i did it)
All this can be found in the following link about the server.conf file, however documentation is not always very clear on specific issues.
Hopefully my lovely, mildly unguided, kinda simple to solve issue and solution will help someone else down the road