Index failures after upgrading to 2.3.1 from 2.2.3


(Julien) #1

I basically get the same problem as this post: Indexer Failures after upgrade to 2.3.1
After upgrading omnibus package from 2.2.3 to 2.3.1 this started for no apparent reason; I don’t recall making custom mappings.

Problem is I can’t even rotate the index, neither by API or UI… Nothing happens and I can’t find the cause in the logs… any help appreciated.

EDIT: After troubleshooting, here is the log when trying to rotate the index:

2017-09-27_19:49:43.15525 WARN  [Indices] Couldn't create index graylog_1548. Error: {"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [String] declared on field [ProcessName]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [message]: No handler for type [String] declared on field [ProcessName]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [String] declared on field [ProcessName]"}}
2017-09-27_19:49:43.15618 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
2017-09-27_19:49:43.15712 java.lang.RuntimeException: Could not create new target index <graylog_1548>.
2017-09-27_19:49:43.16039       at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:294) ~[graylog.jar:?]
2017-09-27_19:49:43.16102       at org.graylog2.rest.resources.system.DeflectorResource.cycle(DeflectorResource.java:124) ~[graylog.jar:?]
2017-09-27_19:49:43.16188       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
2017-09-27_19:49:43.16373       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
2017-09-27_19:49:43.16492       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
2017-09-27_19:49:43.16623       at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
2017-09-27_19:49:43.16716       at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
2017-09-27_19:49:43.16871       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
2017-09-27_19:49:43.16961       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
2017-09-27_19:49:43.17198       at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:143) ~[graylog.jar:?]
2017-09-27_19:49:43.17286       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
2017-09-27_19:49:43.17424       at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
2017-09-27_19:49:43.17512       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
2017-09-27_19:49:43.17662       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
2017-09-27_19:49:43.17753       at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
2017-09-27_19:49:43.17899       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
2017-09-27_19:49:43.18028       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
2017-09-27_19:49:43.18145       at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
2017-09-27_19:49:43.18218       at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
2017-09-27_19:49:43.18358       at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
2017-09-27_19:49:43.18435       at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
2017-09-27_19:49:43.18604       at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
2017-09-27_19:49:43.18695       at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
2017-09-27_19:49:43.18862       at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
2017-09-27_19:49:43.18942       at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
2017-09-27_19:49:43.19145       at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
2017-09-27_19:49:43.19223       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
2017-09-27_19:49:43.19390       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
2017-09-27_19:49:43.19474       at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

(Julien) #2

Ok found out some coworker added custom mappings without telling anyone and deleting it allowed to rotate the index (we’ll reformat the mapping to ES 5.5 standards eventually).

The index failures stopped after rotation of the index.


(system) #3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.