After many months of problem-free working, I’ve now got a problem that has blocked basically all use of graylog for search.
Now, any time I try to do a “Quick Value” sort on some data, I get the dreaded red popup at the bottom and the ES cluster starts reporting
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [application_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
(that is an example of me trying to sort some data by the “application_name” field)
I’ve now altered the “/_template/graylog-custom-mapping” mapping to add “fielddata: true” to that field (and a couple of others causing the same error), confirmed with curl that the change occurred, and rotated the index, but even though it’s now been 4 hours since doing this and the system has itself rotated to a new index, it still can’t sort on “application_name” even over the last five minute period. That doesn’t sound right?
If I look at “/system/index_sets”, I also notice that now the old indices don’t appear correct time-wise. I’m pushing syslog+GELF data into graylog and I’d expect the “newest” older index to show something like “Contains messages from 2 hours ago up to in 5 hours”, but instead the first 20+ indices all say “Contains messages from 2 months ago up to in 6 months”. ie the timestamps seems completely wrong.
Actually, I just ran a standard search over the past 5 minutes, and now notice the comment “Search result Found 975,472 messages in 761 ms, searched in 655 indices.”. 655 indices? Surely that should be 1 or maybe 2 indices?
Any ideas what’s gone wrong? These are CentOS-7 systems running graylog-server-3.0.1-2.noarch and elasticsearch-5.6.16-1.noarch (4-node cluster) from official repos.