1. Describe your incident:
A few days ago, we had to rebuild our MongoDB replica set. Since then we are no longer able to log into Graylog web interface, neither with ‘admin’ account, nor with personal accounts.
When attempting to log in, an error is displayed:
“Error - the server returned: 500 - There was an error fetching a resource: Internal Server Error. Additional information: Couldn’t load session”
No changes to graylog configuration were made. Until changes were made to the replica set, graylog login worked.
2. Describe your environment:
- OS Information:
# grep ^VERSION= /etc/os-release
VERSION="20.04.5 LTS (Focal Fossa)"
- Package Version:
# apt list --installed | grep -E 'graylog|elasticsearch'
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
elasticsearch-oss/stable,now 7.10.2 amd64 [installed]
graylog-4.2-repository/stable,now 1-4 all [installed]
graylog-server/stable,now 4.2.13-1 all [installed]
MongoDB is 4.4.13 (we use the MongoDB Cloud Manager to install and maintain the replica set).
From mongod host1:
$> mongod --version
db version v4.4.13
Build Info: {
"version": "4.4.13",
"gitVersion": "df25c71b8674a78e17468f48bcda5285decb9246",
"openSSLVersion": "OpenSSL 1.1.1 11 Sep 2018",
"modules": [],
"allocator": "tcmalloc",
"environment": {
"distmod": "ubuntu1804",
"distarch": "x86_64",
"target_arch": "x86_64"
}
}
- Service logs, configurations, and environment variables:
- /etc/graylog/server/server.conf
# grep ^http /etc/graylog/server/server.conf
http_bind_address = 0.0.0.0:9000
# grep ^mongodb_uri /etc/graylog/server/server.conf
mongodb_uri = mongodb://host1:27017,host2:27017,host3:27017/graylog?replicaSet=replSet01
- /var/log/graylog-server/server.log:
2022-11-29T11:02:44.704Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: Couldn't load session
at org.graylog2.security.MongoDbSessionDAO.doUpdate(MongoDbSessionDAO.java:101) ~[graylog.jar:?]
at org.apache.shiro.session.mgt.eis.CachingSessionDAO.update(CachingSessionDAO.java:277) ~[graylog.jar:?]
at org.apache.shiro.session.mgt.DefaultSessionManager.onChange(DefaultSessionManager.java:212) ~[graylog.jar:?]
at org.apache.shiro.session.mgt.AbstractNativeSessionManager.applyGlobalSessionTimeout(AbstractNativeSessionManager.java:125) ~[graylog.jar:?]
at org.apache.shiro.session.mgt.AbstractNativeSessionManager.start(AbstractNativeSessionManager.java:100) ~[graylog.jar:?]
at org.apache.shiro.mgt.SessionsSecurityManager.start(SessionsSecurityManager.java:152) ~[graylog.jar:?]
at org.apache.shiro.subject.support.DelegatingSubject.getSession(DelegatingSubject.java:340) ~[graylog.jar:?]
at org.apache.shiro.subject.support.DelegatingSubject.getSession(DelegatingSubject.java:316) ~[graylog.jar:?]
at org.graylog2.shared.security.SessionCreator.create(SessionCreator.java:80) ~[graylog.jar:?]
at org.graylog2.rest.resources.system.SessionsResource.newSession(SessionsResource.java:146) ~[graylog.jar:?]
at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_352]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_352]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_352]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_352]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_352]
3. What steps have you already taken to try and solve the problem?
- verified, that no changes were made to
/etc/graylog/server/server.conf- especially:
mongodb_uri
- especially:
- verified, hostnames in
mongodb_uriresolve to the correct IP addresses of mongod nodes - verified, replica set is set up properly and can be written to and read from (tested from various internal applications relying on MongoDB)
-
host1is primary,host2andhost3are secondaries
-
- verified,
graylogdatabase exists in replica set - verified, that no auth is required against replica set
4. How can the community help?
Any advise is appreciated
- if deleting the replica set and creating a new one with same name is the cause of the server error 500 when attempting to log in
- if there are any information about the replica set (i.e.: UUID, …) stored in graylog and if so,
- how to change it or any other way to (re-)connect the graylog instance to the replica set,
- helping to interpret the above error in graylog’s server.log (google did not help very much
), especially:
- what the exact cause is,
- how and where to find further information and
- how to solve the issue.
Thanks in advance!
but your log tends to be from Mongo.