JSON extractor creates long field from epoch data value

Hi folks, I’m very new to graylog and I still have difficulties to understand all the connections between opensearch, mongodb and graylog itself.

I have graylog 5.1.8 running and send data from a few applications to it. So far it’s looking good, but I have one issue I’m unable to solve

1. Describe your incident:
I have a gelf input running, which is getting data in json format. I create a json extractor which is working fine excpect for a few date fields, which are in epoch format. They are stored as long.
I tried to convince chatgpt to solve my issue, where I stumbled over index templates, index creating with fixed values, reindexing data, but no matter what I do, I don’t succeed.

2. Describe your environment:

  • OS Information:
    Ubuntu 22.04.3 LTS \n \l
  • Package Version:
    ii graylog-5.1-repository 1-2 all Package to install Graylog 5.1 GPG key and repository
    ii graylog-server 5.1.8-1 amd64 Graylog server
    ii opensearch 2.11.0 amd64 An open source distributed and RESTful search engine

3. What steps have you already taken to try and solve the problem?
I tried to create an index template where I only list up the fields which should be date and not long

{
“index_patterns”: [“my_index*”],
“template”: {
“mappings”: {
“properties”: {
“InitialisedDate”: {
“type”: “date”,
“format”: “epoch_millis”
},
“ModificationDate”: {
“type”: “date”,
“format”: “epoch_millis”
},
“CreationDate”: {
“type”: “date”,
“format”: “epoch_millis”
},
}
}
}
}

After rotating the index it only consist of exactly the above, all the g2* stuff, which is normally added isn’t there and after sending log data in, the structure changes how data is represented in the index.

Before having a template it looks like this

            "CardNumber": {
                "type": "keyword"
            },

and after

            "CardNumber": {
                "type": "text",
                "fields": {
                    "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                    }
                }

Which also leads to no data is shown on the search at al with the error message:

While retrieving data for this widget, the following error(s) occurred:

  • Unable to perform search query: OpenSearch exception [type=illegal_argument_exception, reason=Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [timestamp] in order to load field data by uninverting the inverted index. Note that this can use significant memory.].

4. How can the community help?
Maybe give me a hint how I can simply convince graylog to treat a specific list of fields as epoch instead of being a long number :slight_smile:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.