Pipeline rule editor error when updating rules

Graylog Central (peer support)
,
1

Hi everyone! I hope someone can help.

1. Describe your incident:
There’s an error when updating pipeline rules. I can create new rules with the rule editor from scratch just fine but as soon as I want to update a rule I get the following error:

grafik
grafik1805×101 6.46 KB

The same happens to months old rules. If I open them and only click “Update rule” I immediately get the same error. When I click “Convert to source code” and create a new rule from there, I can keep editing the rule from the source code editor without the error.

I’m not sure when it started happening as I haven’t had to touch the rules in a while and everything else seems works just fine.

2. Describe your environment:

  • 3 VMs
  • 1x Graylog Server 6.2.2-1 with Nginx 1.22.1-9+deb12u1
  • 1x Graylog Datanode 6.2.2-1
  • 1x MongoDB 7.0.20
  • latest Debian 12.10 Bookworm

3. What steps have you already taken to try and solve the problem?
I was on version 6.2.0-10 or 6.2.1-1 when I noticed it happening. I restarted everything and the error still appeared. Then I noticed that 6.2.2-1 is released but without changelog yet and hoped it could fix it, so I upgraded graylog-server and graylog-datanode but it kept happening.
The logs don’t log anything when the error appears. I looked into the following log files:

/var/log/graylog-server/server.log
/var/log/graylog-datanode/datanode.log
/var/log/mongodb/mongod.log

Then I looked into the browser developer tools and saw such entries whenever I tried to hit “Update Rule”:

XHR PUT https://graylog.our.domain/api/system/pipelines/rulebuilder/681b5aa22cf1a268a728e93f [HTTP/2 400  18ms]

I wasn’t sure if it could be browser related so I tested Chrome as well (I usually use Firefox) and tried again without caches but it’s still happening. I’m using the Nginx reverse proxy configuration from the docs.

I read through the changelogs and haven’t noticed anything I would have to do manually in case it started happening after one of the latest updates.

4. How can the community help?
I’m not sure where to look to get more info as I haven’t seen anyone else with this problem.
I saw a post from 2020 with someone having a similar problem in a different editor but they solved it themselves by updating.
If needed I can post and anonymise the configurations but seems weird as everything else works fine.

Does anyone have a hint where to look or even recognises the problem?

2

I nearly forgot to mention that the rule editor as a whole breaks too once I want to edit existing rules to add something new:

grafik
grafik1259×421 10.3 KB

As can be seen in the screenshot, the newly added steps stay blank.

4

@msmoak please start a new topic for your 6.2.2 issues

5

@elster I cannot reproduce this. This would happen if the JSON data does not match the Java class. But then everybody that uses Rulebuilder on 6.2.2 should be seeing the issue.

Can you capture the JSON request data from the browser and share that? Here’s a screenshot what that looks like on Firefox:

image
image878×411 63 KB

6

Hi @patrickmann

I created a new rule (which worked fine) and then hit update twice with the same error:

grafik
grafik1443×833 86 KB

If it’s of help I also tried to add a new entry (which stays blank) and hit update for comparison:

grafik
grafik1743×939 75.3 KB

Here’s the error in question in JSON format too:

{
  "type": "RequestError",
  "message": "Unable to map property _scope.\nKnown properties include: title, modified_at, created_at, id, description, rule_builder, source, simulator_message",
  "line": 1,
  "column": 923,
  "path": "_scope",
  "reference_path": "org.graylog.plugins.pipelineprocessor.rulebuilder.rest.AutoValue_RuleBuilderDto[\"_scope\"]"
}
7

Still trying to understand what’s going on … do you see the same error if you click Update rule & close?

8

I was able to reproduce the issue on 6.2.2. Debugging it now …
It’s interesting that the error only seems to show up on repeated update requests. The first one succeeds but following requests fail, with exactly the same request data.

It seems to be a timing issue. If I wait about 10s or more between updates, there is no error. Clicking quickly in succession triggers the bug.

I filed a bug to track:

9

I’m glad you could reproduce something but I’m not sure if it’s the same problem or maybe a similar cause. For me it doesn’t matter how long I wait apparently. I waited roughly 30s between every single step but the error still occurs.

What baffles me is that once I saved a new rule, I can’t add new conditions or actions or at the very least I don’t see them. It looks like it’s included in the JSON message but won’t be displayed.

I added some marking to make it more clear:

grafik
grafik1562×878 70 KB

Maybe it’s related to the same root cause though.

10

I just want to dump some information here in case it helps. We took snapshots of all VMs and did some testing.

  1. We dropped the database, rebooted the graylog-server and graylog-datanode and after the preflight the issue still occured.
  2. We replaced the graylog-server only and the issue still occured.
  3. Emptying the folders config and data under /var/lib/graylog-datanode/opensearch and restarting the graylog-datanode service did not help either.
  4. We tried to add a new datanode to replace the existing one but we were unsuccessful to do so. We get the same error messages as seen in the logs from this post here:
    Failled to add new Datanode - #14 by Etny
  5. Accessing the webinterface without reverse proxy produced the same error.

We are eyeing to just build it from scratch but it seems really tedious to migrate all the rules and stuff.