Archiv3 catalog broken after upgrade, 2.2 to 2.3.1


(Saulius Zilis) #1

I upgraded graylog from 2.2 to 2.3.1 via extracting from tar. That upgrade went smoothly and I restarted the graylog-server, I then realized that I hadn’t upgraded the plugins as well. I shut it all down and extracted the 2.3.0 plugins into the plugin directory and got the server back online.

Once I did I lost all the archive settings and graylog reports that it cannot find the archive catalog with the following error displayed in the web console:

Could not retrieve the archive catalog
Fetching archive catalog failed: cannot GET https://graylog2.wiland.com/api/plugins/org.graylog.plugins.archive/archives/catalog?page=1&per_page=20 (500)

I unfortunately don’t have a backup from Mongo prior to these upgrades but when I issue show collections I have these two displayed; archive_backends, archive_catalog.

I’m at a loss for what to do short of a full rebuild, but then I lose a year of Archives.

Here are java errors I have pulled from the log files as well.

When I attempt to access System > Archives this error pops up in the logs.

2017-10-09T16:02:24.248-06:00 WARN  [Errors] The following warnings have been detected: WARNING: Unknown HK2 failure detected:
MultiException stack 1 of 2
java.lang.IllegalStateException: No backend configured yet!
        at org.graylog.plugins.archive.backends.fs.FSBackendFactory.create(FSBackendFactory.java:29)
        at org.graylog.plugins.archive.rest.ArchivesResource.<init>(ArchivesResource.java:157)
        at sun.reflect.GeneratedConstructorAccessor233.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1375)
        at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:272)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:366)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
        at org.glassfish.jersey.process.internal.RequestScope.findOrCreate(RequestScope.java:162)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2022)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:774)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:737)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:707)
        at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:172)
        at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284)
        at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:74)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:109)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:92)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:61)
        at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:318)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384)
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
MultiException stack 2 of 2
java.lang.IllegalStateException: Unable to perform operation: create on org.graylog.plugins.archive.rest.ArchivesResource
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:393)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
        at org.glassfish.jersey.process.internal.RequestScope.findOrCreate(RequestScope.java:162)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2022)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:774)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:737)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:707)
        at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:172)
        at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284)
        at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:74)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:109)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:92)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:61)
        at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:318)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384)
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

While I just let the server run normally, this error pops up occasionally.

2017-10-09T15:58:44.377-06:00 INFO  [SystemJobManager] Submitted SystemJob <04d50890-ad3d-11e7-acfc-0cc47a54414b> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob]
2017-10-09T15:58:44.379-06:00 ERROR [SystemJobManager] Unhandled error while running SystemJob <04d50890-ad3d-11e7-acfc-0cc47a54414b> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob]
java.lang.IllegalStateException: No backend configured yet!
        at org.graylog.plugins.archive.backends.fs.FSBackendFactory.create(FSBackendFactory.java:29) ~[?:?]
        at org.graylog.plugins.archive.job.ArchiveCreateJob.run(ArchiveCreateJob.java:140) ~[?:?]
        at org.graylog.plugins.archive.job.ArchiveCreateSystemJob.execute(ArchiveCreateSystemJob.java:33) ~[?:?]
        at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_141]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

(bernd) #2

@Lightazurik It looks like your archive configuration doesn’t include a backend_id. This should have been created automatically on upgrade.

The archive plugin is an enterprise feature so you should open a ticket with our support system. Please include the full Graylog server log when you open a ticket to add more context. Thank you!


(system) #3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.