Datatype for a custom field is wrong, expects object I want string

[tuaris (Daniel Morante)]

1. Describe your incident:

It seems that I messed up the datatype for a custom field. It’s is now an object rather than a string.

2. Describe your environment:

• OS Information: Ubuntu 20.04

• Package Version: graylog-4.2

3. What steps have you already taken to try and solve the problem?

Documentation, search engine, these forums trial and error. The issue is too specific to yield any results.

4. How can the community help?

I created a pipeline rule to add a field for environment and FQDN of the host. The value for environment is derived from the domain of the host. I created a CSV lookup table for mapping domains to environments and I am using the built-in PTR lookup table. If for whatever reason the domain is not in the CSV lookup table, I have the rule fallback on auto-detecting the environment from the sub-domain.

This is the pipeline script that is currently working minus the environment field. I have those lines (as well as some troubleshooting) commented out since that’s where my problem is.

rule "set environment"
when
  true
then
  let fqdn = lookup_value("ip-to-hostname", $message.gl2_remote_ip, $message.source);
  let host = regex_replace("\\.$", to_string(fqdn), "", false);
  let domain = regex("[^.]*[.]([^.]*[.]mydomain[.]net)[.]?", host);
  let auto_detected_env = regex("[^.]*[.]([^.]*)[.]mydomain[.]net[.]", host);
  //let environment = lookup_value("domain-to-environment", domain["0"], auto_detected_env["0"]);
  //let environment = lookup_value("domain-to-environment", "prod.mydomain.net", "production");
  
  set_field("host", host);
  //set_field("environment", environment);
  //set_field("environment", "production");
end

As soon as I uncomment the environment related fields it makes Graylog unhappy and starts throwing some indexer errors:

ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=object mapping for [environment] tried to parse field [environment] as object, but found a concrete value]]

Even if I just hard code the value (as seen in the last line above).

I suspect this is related to an error I made where I used the result of the regex match function directly as the field value. It seems that the datatype is now an object rather than a string.

How can I fix this if that’s the issue. Also would welcome feedback/suggestions on the pipeline rule (still new to it).

[Joel_Duffield]

If you manually rotate the index that the messages are being stored in that should reset those field types. It will automatically detect the data type on first write. Rotating the index will reset back to nothing and let you start over.

Now after you do that if you run a search you can have two data types on the same field, and that can cause weird things until that old index is completely deleted.