OpenSearch 2.5 graphs broken

[SteveU]

Now that OpenSearch has a Debian apt package with the 2.5.0 release, started testing migrating from ES 7.10.2. Overall it’s been smooth, but I just noticed some of my graphs are broken. Specifically, ones that have 2 fields with column selected. As an example, I have Linux servers put df info into syslog and parse with a pipeline rule. I then create a chart with the average value of df-use (type: long), and have source and df-filesystem as columns to graph each filesystem per server. This worked fine with ES, but with OpenSearch I now get this error in the graph box:

* OpenSearch exception [type=null_pointer_exception, reason=Cannot invoke "org.opensearch.search.aggregations.InternalAggregations.getSerializedSize()" because "reducePhase.aggregations" is null].

I don’t know if this is one of those things that shouldn’t have worked in the first place, if there is a better way to get what I’m looking for, bug/feature of Graylog or OpenSearch, etc. I know I can add something to concat these fields in the pipeline rule and adjust the graph, but I got a lot of graphs like this and figured I’d ask before making a lot of changes. Thanks!

• OS Information: Ubuntu 20.04.5
• Package Version: Graylog 5.0.2, OpenSearch 2.5.0

logs generated with:

df -t ext4 | sed 1d | tr -s ' ' | logger -t "df-output"

produces output like:

/dev/sda2 40970464 10839600 28017484 28% /
/dev/sdb1 267240240 31523240 222174088 13% /var/lib/opensearch

pipeline rule:

rule "df-output logs"
when
  has_field("application_name") &&
  $message.application_name == "df-output"
then
  let parsed_fields = grok(pattern: "%{UNIXPATH:df-filesystem} %{NUMBER:df-size} %{NUMBER:df-used} %{NUMBER:df-avail} %{NUMBER:df-use}% %{UNIXPATH:df-mount}", value: to_string($message.message), only_named_captures: true);
  set_fields(parsed_fields);
end

cat graylog-custom-mapping.json

{
  "template": "graylog_*",
  "mappings" : {
      "properties" : {
        "df-size" : {
          "type" : "long"
        },
        "df-used" : {
          "type" : "long"
        },
        "df-avail" : {
          "type" : "long"
        },
        "df-use" : {
          "type" : "long"
        }
      }
  }
}

Graph settings:

Working graph example:

[gsmith]

Hello @SteveU

By chance is this similar to your issue?

Multiple Widgets Error: OpenSearch exception type=null_pointer_exception

Curious, can you recreate those fields again? or do you still get errors?

Was the index set rotated after the migration? Sometime this does help along with recalulating the index set.

[SteveU]

This looks exactly like my issue.

I’ve tried creating new widgets, and it works if I have source or df-filesystem, but not with both.

I have rotated the index and verified custom mapping is still in place.

Thanks. -Steve

[SteveU]

Just found Graylog 5.0.3 was released. Not sure which of the changes helped (maybe Supporting nested and tupled groupings in aggregations. by dennisoelkers · Pull Request #14446 · Graylog2/graylog2-server · GitHub), but I’m no longer seeing the error and the graph is working as expected.

[gsmith]

Awesome, if you could mark this post as resolved this would help for future searches