I basically get the same problem as this post: Indexer Failures after upgrade to 2.3.1
After upgrading omnibus package from 2.2.3 to 2.3.1 this started for no apparent reason; I don’t recall making custom mappings.
Problem is I can’t even rotate the index, neither by API or UI… Nothing happens and I can’t find the cause in the logs… any help appreciated.
EDIT: After troubleshooting, here is the log when trying to rotate the index:
2017-09-27_19:49:43.15525 WARN [Indices] Couldn't create index graylog_1548. Error: {"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [String] declared on field [ProcessName]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [message]: No handler for type [String] declared on field [ProcessName]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [String] declared on field [ProcessName]"}}
2017-09-27_19:49:43.15618 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
2017-09-27_19:49:43.15712 java.lang.RuntimeException: Could not create new target index <graylog_1548>.
2017-09-27_19:49:43.16039 at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:294) ~[graylog.jar:?]
2017-09-27_19:49:43.16102 at org.graylog2.rest.resources.system.DeflectorResource.cycle(DeflectorResource.java:124) ~[graylog.jar:?]
2017-09-27_19:49:43.16188 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
2017-09-27_19:49:43.16373 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
2017-09-27_19:49:43.16492 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
2017-09-27_19:49:43.16623 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
2017-09-27_19:49:43.16716 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
2017-09-27_19:49:43.16871 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
2017-09-27_19:49:43.16961 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
2017-09-27_19:49:43.17198 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:143) ~[graylog.jar:?]
2017-09-27_19:49:43.17286 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
2017-09-27_19:49:43.17424 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
2017-09-27_19:49:43.17512 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
2017-09-27_19:49:43.17662 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
2017-09-27_19:49:43.17753 at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
2017-09-27_19:49:43.17899 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
2017-09-27_19:49:43.18028 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
2017-09-27_19:49:43.18145 at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
2017-09-27_19:49:43.18218 at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
2017-09-27_19:49:43.18358 at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
2017-09-27_19:49:43.18435 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
2017-09-27_19:49:43.18604 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
2017-09-27_19:49:43.18695 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
2017-09-27_19:49:43.18862 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
2017-09-27_19:49:43.18942 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
2017-09-27_19:49:43.19145 at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
2017-09-27_19:49:43.19223 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
2017-09-27_19:49:43.19390 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
2017-09-27_19:49:43.19474 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]