Error: cannot PUT notification (500)

[kkplein]

We seem to have a corrupted notification or something on Graylog 3.1.4+1149fe1: Menu alerts, notifications.

There is only one (old) notification present, and it also generates errors in server.log. I cannot delete or edit it in the GUI, because it gives:

Error updating Notification “too many fatals on sophos” failed with status: Error: cannot PUT https://gl.FQDN.com/api/events/notifications/5e4aa8abe1b7e23519b6ee61 (500)

I am not interested in keeping that notification. Is there a way to delete it outside the graylog GUI?

Thanks!

[kkplein]

For completeness, the error logged in server.log is:
2021-04-07T13:28:45.075+02:00 ERROR [EventProcessorExecutionJob] Event processor <5e4aa8abe1b7e23519b6ee63> failed to execute: parameters=AggregationEventProcessorParameters{type=aggregation-v1, timerange=AbsoluteRange{type=absolute, from=2020-02-18T07:43:27.781Z, to=2020-02-18T07:48:27.780Z}, streams=, batchSize=500} (retry in 5000 ms)

[kkplein]

Apache logs a similar line:
1.2.3.4 - - [07/Apr/2021:15:01:23 +0200] “DELETE /api/events/notifications/5e4aa8abe1b7e23519b6ee61 HTTP/1.1” 500 702 “https://gl.FQDN.com/alerts/notifications” “Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0”

[gsmith]

@kkplein
Hello,
I see your using HTTPS. The errors you see on the GUI might be from your Certificates and/or the Graylog server.conf file. I cant tell because the lack of information of your envirnoment.
If you could explain in greater detail how you set up Graylog Server, what configurations you made, etc… that would be very helpfull to troubleshoot your issue.

Have a look here.
https://community.graylog.org/t/community-guidelines/6649#before-ask

Thanks

[kkplein]

hi @gsmith!

Thanks for your response!

Yes, we’re running through apache2, configured like this:

<VirtualHost *:443>
	ServerName gl.FQDN.com
	DocumentRoot /var/www/html

    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    <Location />
        RequestHeader set X-Graylog-Server-URL "https://gl.FQDN.com/"
        ProxyPass http://127.0.0.1:9000/
        ProxyPassReverse http://127.0.0.1:9000/
    </Location>

	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
	# error, crit, alert, emerg.
	# It is also possible to configure the loglevel for particular
	# modules, e.g.
	LogLevel info ssl:warn

	ErrorLog ${APACHE_LOG_DIR}/graylog-error.log
	CustomLog ${APACHE_LOG_DIR}/graylog-access.log combined


	#   SSL Engine Switch:
	#   Enable/Disable SSL for this virtual host.
	SSLEngine on


	SSLCertificateFile	/etc/ssl/apache2/cert.pem
	SSLCertificateKeyFile /etc/ssl/apache2/key.pem
	SSLCertificateChainFile /etc/ssl/apache2/fullchain.pem

</VirtualHost>

The ssl certificate is valid, and since the whole of graylog works perfectly, I didn’t think of including the apache config. Sorry.

It is just this paticular notification under “Alerts” that is causing the server.log errors, and that I cannot delete. I created and deleted a new test notification, just to make sure that there is no bigger global issue.

It seems some kind of corruption exists with this particular notification, and as I no longer need it, simply getting rid of i would be fine. But how…?

Let me know if there is anything else needed…?

Thanks again!

[kkplein]

I noticed only now that the whole of the “Alerts” menu does not seem to work:

• Loading Events information… never completes (keeps spinning)
• Loading Event Definitions information…never completes (keeps spinning)

graylog logs at the time:

2021-04-08T11:52:19.337+02:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource

The (single) notification we have under Alerts, IS displayed, but I cannot delete it.

For the rest, the graylog system seems to work.

Any ideas.?

[kkplein]

I solved it.

In the past we tried out the 3.2 version, and did not like it. So we changed back to 3.1. But I guess that upgraded something (incompatible with 3.1) in the database.
I now upgraded via 3.2 to 3.3, and: we like the newer version, the problem above is gone, and the reason why we reverted to 3.1 is also solved. (easy one-click reverse-sort of the search results)

So, this issue can be closed.

[gsmith]

@kkplein
Nice, glad you figured it out.