Hello everyone,
I hope someone can help me on this topic because it’s driving me crazy.
I manually rotated an index on my Graylog node and when I try to query something, it returns this:
While retrieving data for this widget, the following error(s) occurred:
Unable to perform search query Fielddata is disabled on text fields by default. Set fielddata=true on [timestamp] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
This happens just pointing on the new rotated index (index_4), if I query the old ones (index_n-1), it works.
Now, I know it could be something related to index mapping, but if it really was, I’d see the same error on the “normal” Search queries too, this happens just on Extended Search View.
Have you any idea to solve this without removing the index and creating a new one with a new mapping?
what did you do exactly? Did you run a query? Do you create a widget? What kind of search did you run? What did you try to do to get that result? Does that happens always?
Hi @jan,
thanks for your quick reply.
I will try to answer you point to point:
I manually rotated an index (es: graylog_1 to graylog_2). Messages correctly forwarded in the new elastic index.
Going to “Views” feature, I created a new View launching a simple query like “EventID:4624”
As I create a Widget (for example, “message table”), the error occurs.
If I launch a query including an old index (EXAMPLE: _index:graylog_1 AND EventID:4624), it works, the widget shows me the messages correctly.
So, I think the widget is not able to show me the message included in the new index graylog_2 giving me that error. I checked the mappings and they are exactly the same.
Yes, It happens always, everytime I try to query the new index.
It does not happen doing the same query going to Search tab.
I Just updated to 3.1.4, same story.
Is there a way to understand how the extended searches actually work?
As I create the view, the first thing I see is the error occuring on both “message count” and “all messages” widgets.
If I delete the widgets already there, creating them again, the error still occurs.
If I query an old index (graylog_3 for example), selecting “Search in all messages” as time arriving, the widgets work.
I don’t know why, but it seems that graylog identifies the Timestamp field with “text” type instead of “date”, asking me to enable Fielddata on it.
This does not happen if I query the messages on Search tab.
Hi @gsmith,
thank you for the reply.
My graylog setup includes:
one graylog node and one elasticsearch node installed on two different servers.
There’s no cluster, the environment is small.
We use this graylog/elasticsearch environment as a repository, we dump messages from another elasticsearch node to this node filtered by DSL Queries, so no input is enable.
The system was working fine until this index rotation.
Here what I can see in the server.log file:
2020-01-27T14:47:22.702+01:00 WARN [IndexFieldTypePollerPeriodical] Active write index for index set “Default index set” (5c38b17d4d9e030a78612c47) doesn’t exist yet
2020-01-27T14:47:27.702+01:00 WARN [IndexFieldTypePollerPeriodical] Active write index for index set “Default index set” (5c38b17d4d9e030a78612c47) doesn’t exist yet
2020-01-27T14:47:30.730+01:00 INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2020-01-27T14:47:30.731+01:00 INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2020-01-27T14:47:30.734+01:00 INFO [MongoIndexSet] Cycling from to <graylog_0>.
2020-01-27T14:47:30.734+01:00 INFO [MongoIndexSet] Creating target index <graylog_0>.
2020-01-27T14:47:30.749+01:00 INFO [Indices] Successfully created index template graylog-internal
2020-01-27T14:47:30.752+01:00 WARN [Indices] Couldn’t create index graylog_0. Error: {“root_cause”:[{“type”:“resource_already_exists_exception”,“reason”:“index [graylog_0/cDuQ0alwTF-Oqjt6OcIC6A] already exists”,“index_uuid”:“cDuQ0alwTF-Oqjt6OcIC6A”,“index”:“graylog_0”}],“type”:“resource_already_exists_exception”,“reason”:“index [graylog_0/cDuQ0alwTF-Oqjt6OcIC6A] already exists”,“index_uuid”:“cDuQ0alwTF-Oqjt6OcIC6A”,“index”:“graylog_0”}
2020-01-27T14:47:30.753+01:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
java.lang.RuntimeException: Could not create new target index <graylog_0>.
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:294) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:261) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_232]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_232]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_232]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
graylog_0 index is closed since we use a custom index to which we forward the logs to.
@tanzagraylog85
Hello
I’m unsure how to fix your problem, I have not configured an environment like yours before and never had to import/dump messages from another elasticsearch node. We have cluster setups for High volume environments, and just a “All in one” for small environments.
Looking at your posted logs some of them caught my attention.
2020-01-27T14:47:22.702+01:00 WARN [IndexFieldTypePollerPeriodical] Active write index for index set “Default index set” (5c38b17d4d9e030a78612c47) doesn’t exist yet
2020-01-27T14:47:30.753+01:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index java.lang.RuntimeException: Could not create new target index <graylog_0>.
What I don’t understand is Mongo tries to Cycle indexes” [MongoIndexSet] Cycling from to <graylog_0>” then Mongo creates and Index “Creating target index <graylog_0>” now you have a warning “Couldn’t create index graylog_0”
Have you looked at the Elasticsearch logs for errors and MongoDB logs for errors?
Sorry I can’t be more help.
Ok, PROBLEM SOLVED.
As I thought, graylog had something wrong in cache, but It was related to a particular index that I deleted some time ago from the console, but querying Elasticsearch, it was still there.
Apparently, this index had Timestamp field with “text” type, thus the widgets showed me that error (strange thing, because I was expected the same error on Search widgets).
I deleted the index and launched a db.index_ranges.drop() from mongodb shell to drop the ranges, after that I recalculated the index ranges.
Everything is all right now
