Unable to load user list after upgrade to 2.2.2

Graylog Central (peer support)
1

Hi all, got problem after upgrade.Few days ago we upgraded our Graylog server from 2.0.1 to 2.2.2. Now when we go to Authentication -> Users it stuck on loading animation showing red error below:

Could not load user list
Loading user list failed with status: Error: cannot GET http://hostname:12900/users (500)

Got following in graylog server log:

 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.ClassCastException: com.mongodb.BasicDBObject cannot be cast to [B
        at org.graylog2.security.MongoDbSession.getAttributes(MongoDbSession.java:62) ~[graylog.jar:?]
        at org.graylog2.security.MongoDbSession.getUsernameAttribute(MongoDbSession.java:92) ~[graylog.jar:?]
        at org.graylog2.rest.resources.users.UsersResource.lambda$listUsers$0(UsersResource.java:146) ~[graylog.jar:?]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) ~[?:1.8.0_91]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_91]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_91]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_91]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_91]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_91]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_91]
        at org.graylog2.rest.resources.users.UsersResource.listUsers(UsersResource.java:147) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_91]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_91]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_91]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_91]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[gra                                                                    ylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]

Launching: curl -u user:user -H 'Accept: application/json' http://host:12900/users?pretty=true i receive following error:

{
  "type" : "ApiError",
  "message" : "com.mongodb.BasicDBObject cannot be cast to [B"
}

Other REST API calls working as expected.

Software i use:
Graylog Version: 2.2.2
Elasticsearch Version: 2.3.4
MongoDB Version: 3.2.8
Operating System: Oracle Linux Server release 7.2

2

Please post the contents of the users collection in MongoDB (without the password fields).

For reference:

3 
{ "_id" : ObjectId("578dde6603c25a4df2996610"), "password" : "User synced from LDAP.", "full_name" : "User1", "timezone" : "UTC", "permissions" : [ ], "roles" : [ ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "sss@mail.mail", "username" : "a.aa" }
{ "_id" : ObjectId("57ac642b03c25a7b54de6573"), "password" : "User synced from LDAP.", "full_name" : "User2", "startpage" : { "type" : "dashboard", "id" : "578dee0c03c25a4df299768d" }, "timezone" : "UTC", "permissions" : [ "users:passwordchange:a.bb", "users:edit:a.bb" ], "roles" : [ ObjectId("578733f003c25a2744927425"), ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "a.bb@mail.mail", "username" : "a.bb" }
{ "_id" : ObjectId("57bbdae803c25a7b54eea2ea"), "password" : "User synced from LDAP.", "full_name" : "User3", "timezone" : "UTC", "permissions" : [ ], "roles" : [ ObjectId("578733f003c25a2744927425"), ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "a.a@mail.mail", "username" : "a.a" }
{ "_id" : ObjectId("57bc0e9a03c25a7b54eed933"), "password" : "User synced from LDAP.", "full_name" : "User4", "session_timeout_ms" : NumberLong(28800000), "timezone" : "UTC", "permissions" : [ "users:passwordchange:a.s", "users:edit:a.s" ], "roles" : [ ObjectId("578733f003c25a2744927425"), ObjectId("57c5515e03c25a7b54f897e2"), ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "a.s@mail.mail", "username" : "a.s" }
{ "_id" : ObjectId("57c4013903c25a7b54f73396"), "password" : "User synced from LDAP.", "full_name" : "User5" "session_timeout_ms" : NumberLong(28800000), "startpage" : { "type" : "dashboard", "id" : "578dee0c03c25a4df299768d" }, "timezone" : "UTC", "permissions" : [ ], "roles" : [ ObjectId("578733f003c25a2744927425"), ObjectId("57c5515e03c25a7b54f897e2"), ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "ss@mail.mail", "username" : "sss.sss" }
{ "_id" : ObjectId("57c563b003c25a7b54f8ab89"), "password" : "User synced from LDAP.", "full_name" : "User6", "timezone" : "UTC", "permissions" : [ ], "roles" : [ ObjectId("578733f003c25a2744927425"), ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "gg@mail.mail", "username" : "g.g" }
{ "_id" : ObjectId("57cd811c03c25a7b54013a96"), "password" : "User synced from LDAP.", "full_name" : "User7", "timezone" : "UTC", "permissions" : [ ], "roles" : [ ObjectId("57c5515e03c25a7b54f897e2"), ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "h.h@mail.mail", "username" : "h.h" }
{ "_id" : ObjectId("57d11dff03c25a7b540507ca"), "password" : "User synced from LDAP.", "full_name" : "User8", "timezone" : "UTC", "permissions" : [ "users:passwordchange:j.j", "users:edit:j.j" ], "roles" : [ ObjectId("57c550d003c25a7b54f89748"), ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "j.j@mail.mail", "username" : "j.jj" }
{ "_id" : ObjectId("57f36eba03c25a7b5429238d"), "password" : "User synced from LDAP.", "full_name" : "User9", "timezone" : "UTC", "permissions" : [ "users:passwordchange:o.o", "users:edit:o.o" ], "roles" : [ ObjectId("57c550d003c25a7b54f89748"), ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "o.o@mail.mail", "username" : "o.o" }
{ "_id" : ObjectId("57fcaafe03c25a063d2c5a6e"), "password" : "User synced from LDAP.", "full_name" : "user10", "timezone" : "UTC", "permissions" : [ ], "roles" : [ ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "A.K@mail.mail", "username" : "a.k" }
{ "_id" : ObjectId("580a033d03c25a063d3a718a"), "password" : "User synced from LDAP.", "full_name" : "User11", "timezone" : "UTC", "permissions" : [ ], "roles" : [ ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "h.h@mail.mail", "username" : "hh" }
{ "_id" : ObjectId("586e631e03c25a063da44540"), "password" : "User synced from LDAP.", "full_name" : "User12", "timezone" : "UTC", "permissions" : [ ], "roles" : [ ObjectId("57c5515e03c25a7b54f897e2"), ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "o.on@mail.mail", "username" : "o.on" }
{ "_id" : ObjectId("587cbb7803c25a063db35bce"), "password" : "User synced from LDAP.", "full_name" : "User13", "timezone" : "UTC", "permissions" : [ ], "roles" : [ ObjectId("57c5515e03c25a7b54f897e2"), ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "R.K@mail.mail", "username" : "r.ko" }
{ "_id" : ObjectId("589d9a0203c25a063dd5f542"), "password" : "User synced from LDAP.", "full_name" : "User14", "timezone" : "UTC", "permissions" : [ ], "roles" : [ ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "e.go@mail.mail", "username" : "e.go" }
{ "_id" : ObjectId("58b0057603c25a7e8fb37c37"), "password" : "User synced from LDAP.", "full_name" : "User15", "timezone" : "UTC", "permissions" : [ ], "roles" : [ ObjectId("58b0054003c25a7e8fb37bf6"), ObjectId("578733f003c25a2744927426") ], "external_user" : true, "email" : "S.Ko@mail.mail", "username" : "s.ko" }
{ "_id" : ObjectId("58d8d08603c25a0636838566"), "password" : "", "full_name" : "TEST_USER", "session_timeout_ms" : NumberLong(3600000), "permissions" : [ ], "roles" : [ ObjectId("578733f003c25a2744927426") ], "email" : "asa@aa.lc", "username" : "test" }
4

Try purging the “sessions” collection in MongoDB. It looks like there was some incompatibility between the old session serialization format and the new serialization session format (at least concerning the “attributes” field).

5

Thanks jochen it helped. Now i can see list of users.

6

@wilfrredhariden just to clarify when connected to the graylog mongodb run this?:

db.sessions.remove({})
7

Yeap, exactly. It got me logged out from Graylog and after I logged back in it was rainbows and unicorns.