Fresh installation on RH8 - index default error

Graylog Central (peer support)
1

Hi. I’ve installed graylog 4 with required packages:

elasticsearch.x86_64 7.0.1-1
graylog-server.noarch 4.0.5-1
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.282.b08-2.el8_3
mongodb-org.x86_64 4.2.13-1.el8
mongodb-org-mongos.x86_64 4.2.13-1.el8
mongodb-org-server.x86_64 4.2.13-1.el8
mongodb-org-shell.x86_64 4.2.13-1.el8
mongodb-org-tools.x86_64 4.2.13-1.el8

It looks like it should work but not at all works. I have problem with index default. I’ve tried to recreate environment, but problem still occurs. Can you give some advises? Firewall and selinux are disabled.
Gui works with errors:

Could not retrieve global index stats.
Fetching global index stats failed: cannot GET http://<IP_address>:9000/api/system/indices/index_sets/stats (500)
Could not retrieve index sets.
Fetching index sets list failed: Unable to retrieve list of closed indices for [graylog_*]

ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to retrieve list of closed indices for [graylog_]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:136) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:99) ~[?:?]
at org.graylog.storage.elasticsearch7.cat.CatApi.perform(CatApi.java:89) ~[?:?]
at org.graylog.storage.elasticsearch7.cat.CatApi.requestIndices(CatApi.java:85) ~[?:?]
at org.graylog.storage.elasticsearch7.cat.CatApi.indices(CatApi.java:60) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.closedIndices(IndicesAdapterES7.java:295) ~[?:?]
at org.graylog2.indexer.indices.Indices.getClosedIndices(Indices.java:266) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.getClosedIndices(Indices.java:270) ~[graylog.jar:?]
at org.graylog2.indexer.IndexSetStatsCreator.getForIndexSet(IndexSetStatsCreator.java:38) ~[graylog.jar:?]
at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) ~[?:1.8.0_282]
at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_282]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_282]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_282]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_282]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_282]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_282]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_282]
at org.graylog2.rest.resources.system.indexer.IndexSetsResource.list(IndexSetsResource.java:153) ~[graylog.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [GET], host [http://127.0.0.1:9200], URI [/cat/indices/graylog?format=json&h=index%2Cstatus&s=index%2Cstatus&expand_wildcards=all], status line [HTTP/1.1 400 Bad Request]
{“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“request [/cat/indices/graylog] contains unrecognized parameter: [expand_wildcards]"}],“type”:“illegal_argument_exception”,“reason”:"request [/cat/indices/graylog] contains unrecognized parameter: [expand_wildcards]”},“status”:400}
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.storage.elasticsearch7.cat.CatApi.lambda$perform$4(CatApi.java:92) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:97) ~[?:?]
… 43 more
2021-03-29T14:51:01.698+02:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to retrieve list of closed indices for [gl-system-events_, graylog_, gl-events_]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:136) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:99) ~[?:?]
at org.graylog.storage.elasticsearch7.cat.CatApi.perform(CatApi.java:89) ~[?:?]
at org.graylog.storage.elasticsearch7.cat.CatApi.requestIndices(CatApi.java:85) ~[?:?]
at org.graylog.storage.elasticsearch7.cat.CatApi.indices(CatApi.java:60) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.closedIndices(IndicesAdapterES7.java:295) ~[?:?]
at org.graylog2.indexer.indices.Indices.getClosedIndices(Indices.java:266) ~[graylog.jar:?]
at org.graylog2.rest.resources.system.indexer.IndexSetsResource.globalStats(IndexSetsResource.java:175) ~[graylog.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [GET], host [http://127.0.0.1:9200], URI [/cat/indices/gl-system-events,graylog_,gl-events_?format=json&h=index%2Cstatus&s=index%2Cstatus&expand_wildcards=all], status line [HTTP/1.1 400 Bad Request]
{“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“request [/cat/indices/gl-system-events,graylog_,gl-events_] contains unrecognized parameter: [expand_wildcards]"}],“type”:“illegal_argument_exception”,“reason”:"request [/cat/indices/gl-system-events,graylog_,gl-events_] contains unrecognized parameter: [expand_wildcards]”},“status”:400}
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.storage.elasticsearch7.cat.CatApi.lambda$perform$4(CatApi.java:92) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:97) ~[?:?]
… 33 more

2

Hi @dkryszak

Running a manual “curl” in my end with the same URI in the error log I get this:

curl -s -H 'Content-Type: application/json' -XGET "x.x.x.x:9200/cat/indices/graylog?pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [cat]",
        "resource.type" : "index_expression",
        "resource.id" : "cat",
        "index_uuid" : "_na_",
        "index" : "cat"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [cat]",
    "resource.type" : "index_expression",
    "resource.id" : "cat",
    "index_uuid" : "_na_",
    "index" : "cat"
  },
  "status" : 404
}

With an error in “cat”, as you can see. But even with the righ “command”, it runs with error:

curl -s -H 'Content-Type: application/json' -XGET "x.x.x.x:9200/_cat/indices/graylog?pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [graylog]",
        "index_uuid" : "_na_",
        "resource.type" : "index_or_alias",
        "resource.id" : "graylog",
        "index" : "graylog"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [graylog]",
    "index_uuid" : "_na_",
    "resource.type" : "index_or_alias",
    "resource.id" : "graylog",
    "index" : "graylog"
  },
  "status" : 404
}

The right way is:

curl -s -H ‘Content-Type: application/json’ -XGET “x.x.x.x:9200/_cat/indices/graylog_*?pretty”

Try to modify this in your end to see if works.

1 Like
3

Hi @reimlima . Thanks for reply. I’ve resolved the issue. Problem was with elastic search version. After upgrade from 7.0.1 to 7.12.0 problem has disappeared.

4

FYI, Elasticseach version 7.11 and up is not officially suported:

https://docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html#elasticsearch-versions

1 Like
5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.