I upgraded graylog from 2.2 to 2.3.1 via extracting from tar. That upgrade went smoothly and I restarted the graylog-server, I then realized that I hadn’t upgraded the plugins as well. I shut it all down and extracted the 2.3.0 plugins into the plugin directory and got the server back online.
Once I did I lost all the archive settings and graylog reports that it cannot find the archive catalog with the following error displayed in the web console:
Could not retrieve the archive catalog
Fetching archive catalog failed: cannot GET https://graylog2.wiland.com/api/plugins/org.graylog.plugins.archive/archives/catalog?page=1&per_page=20 (500)
I unfortunately don’t have a backup from Mongo prior to these upgrades but when I issue show collections I have these two displayed; archive_backends, archive_catalog.
I’m at a loss for what to do short of a full rebuild, but then I lose a year of Archives.
Here are java errors I have pulled from the log files as well.
When I attempt to access System > Archives this error pops up in the logs.
2017-10-09T16:02:24.248-06:00 WARN [Errors] The following warnings have been detected: WARNING: Unknown HK2 failure detected:
MultiException stack 1 of 2
java.lang.IllegalStateException: No backend configured yet!
at org.graylog.plugins.archive.backends.fs.FSBackendFactory.create(FSBackendFactory.java:29)
at org.graylog.plugins.archive.rest.ArchivesResource.<init>(ArchivesResource.java:157)
at sun.reflect.GeneratedConstructorAccessor233.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1375)
at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:272)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:366)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
at org.glassfish.jersey.process.internal.RequestScope.findOrCreate(RequestScope.java:162)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2022)
at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:774)
at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:737)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:707)
at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:172)
at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284)
at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:74)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:109)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:92)
at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:61)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:318)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384)
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
MultiException stack 2 of 2
java.lang.IllegalStateException: Unable to perform operation: create on org.graylog.plugins.archive.rest.ArchivesResource
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:393)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
at org.glassfish.jersey.process.internal.RequestScope.findOrCreate(RequestScope.java:162)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2022)
at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:774)
at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:737)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:707)
at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:172)
at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284)
at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:74)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:109)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:92)
at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:61)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:318)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384)
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
While I just let the server run normally, this error pops up occasionally.
2017-10-09T15:58:44.377-06:00 INFO [SystemJobManager] Submitted SystemJob <04d50890-ad3d-11e7-acfc-0cc47a54414b> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob]
2017-10-09T15:58:44.379-06:00 ERROR [SystemJobManager] Unhandled error while running SystemJob <04d50890-ad3d-11e7-acfc-0cc47a54414b> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob]
java.lang.IllegalStateException: No backend configured yet!
at org.graylog.plugins.archive.backends.fs.FSBackendFactory.create(FSBackendFactory.java:29) ~[?:?]
at org.graylog.plugins.archive.job.ArchiveCreateJob.run(ArchiveCreateJob.java:140) ~[?:?]
at org.graylog.plugins.archive.job.ArchiveCreateSystemJob.execute(ArchiveCreateSystemJob.java:33) ~[?:?]
at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_141]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]