GET /api/system/bundles returns "ApiError" with empty message and HTTP code 500

[EpiqSty]

Hi,

I’m facing with such responce on GET /api/system/bundles :

500

{
  "type": "ApiError",
  "message": ""
}

Looks like it happend after PUT request with the incorrect payload. I’ve got return code 201 on that PUT, but after this, I’m not able to list content packs anymore.

I’ve tried already to restart graylog-server process on one of nodes, but wthout success, GET /api/system/bundles still returning ApiError and HTTP code 500.

Maybe someone has ideas, how to fix it?

ps: unluckily, I’ve lost original response message after uploading incorrect content pack, so I can’t delete it by id.

So there is any bypass mechanism to get the list of content packs rather than via API?

Thanks in advance!

p.s. Graylog version is 2.4.7+9116ead

at the graylog server log, I am getting this at the moment of request:

2019-12-10T13:27:06.351+01:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.NullPointerException: null
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:787) ~[graylog.jar:?]
        at com.google.common.collect.ImmutableSetMultimap$Builder.put(ImmutableSetMultimap.java:268) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.bundles.BundleResource.listBundles(BundleResource.java:102) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

[jan (Jan Doberstein)]

it could be that you hit a bug in that old version. You might need to cleanup the mongodb collection for that.

But sorry I do not have such an old version running to check which collection that actually is. It might be named like content_packs or similar.

[EpiqSty]

That was actually the only my idea, but I am bit lacking of documentation regarding, how to access local mongodb instance for Graylog node. Maybe you could sent me some starting points/keywords for it?
And another question, should I make such cleanup for all graylog nodes in the cluster, or they are replicating auto-magically?
Thanks!

[jan (Jan Doberstein)]

he @epiqsty

if you have multiple MongoDB instances in your setup that are part of a cluster - build a replica set or replicate you only need to make the write to the mongoDB master server and it will be replicated to all secondary servers.

you might want to spin up one instance of nosqlclient to help you with the task - it is a mongoDB client ( https://www.nosqlclient.com/ )

This would make it also easier to identify the malicious parts in the database.

[EpiqSty]

Thanks a lot!
With nosqlclient and additional help of https://blockdev.io/connecting-to-a-mongo-replica-set-via-ssh/
I was able to connect to MongoDb, and reach out the ID of incorrect content pack bundle in content_packs collection.
Then I’ve deleted such content pack via Graylog API and /api/system/bundles endpoint started working again!