Message Table Can't Sort by Compound Field Type

[cmccabe (Chris)]

Hey All,
I had a working message table widget in a dashboard but I changed the field that the table was sorting on from a string data type to a datetime, rotated the active write index so Graylog would recognize the field as a datetime value, and the table no longer worked. Graylog now recognizes the field as datetime but the message table is for all messages in the stream so the field is compound(date,string) since the field messages before the switch were strings and after the switch they are dates. The table doesn’t know what to think of it and it is throwing this error:

While retrieving data for this widget, the following error(s) occurred:
OpenSearch exception [type=class_cast_exception, reason=class java.lang.Long cannot be cast to class org.apache.lucene.util.BytesRef (java.lang.Long is in module java.base of loader ‘bootstrap’; org.apache.lucene.util.BytesRef is in unnamed module of loader ‘app’)].

Environment:

• OS Information: Graylog docker image running in an AWS container

• Package Version:
  Graylog 5.0.5
  OpenSearch 2.3
  MongoDB 5.0.0

3. What steps have you already taken to try and solve the problem?
The field being used for the table sort is an os_datetime so the messages are in order by run time of the Python process they came from. If I switch to a different datetime field like the default Graylog Timestamp field the table works as intended. If I change the date range to a range entirely before or after the field type switch the table works as intended. Thus leading me to believe the issue is the attempt to sort the table on a compound field type.

4. How can the community help?
Is there a way I can keep the date range for the table widget set to all messages and still sort by the same field? I can’t delete the index with the old field type as the default index is used for other streams as well.
Thanks in Advance!

[joe.gross]

Hi @cmccabe,

Short answer, yes. You can still view all messages and sort by the same field. BUT, it’s going to take some work. I’ll lay out your choices, you can pick what works.

Because you cannot use compound data types in an aggregation, you have two choices. You can either restrict your searches to either side of the “switch”, making one aggregation on the indices containing the string data type, and another with the indices containing the datetime data type, or you can reindex the indices containing the string data type, so they will all match again and the switch line disappears.

I usually recommend not reindexing an index, but here are the docs if you would like to consider it.

Reindex data

Good luck!