Odd Pipeline/Stream Behavior (Part Deux)

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I am ingesting Office365 logs via @ddbnl’s very awesome officelogcollector. That part seems to be functioning fine.

I use the recommended JSON extractor. I then have a pipeline that strips unwanted/malformed fields and runs the message against some lookup tables.

All this was working fine until the other day when suddenly* messages were no longer being committed to the indices.

*(what I mean is, I know of no event that could have caused this change, nor had I made any configuration changes)

The issue is almost EXACTLY like the one resolved, here: Odd Pipeline/Stream Behavior - #6 by drewmiranda-gl

Problem is, there are no such similar errors to guide me.

2. Describe your environment:

  • OS Information:
    Debian 11
  • Package Version:
    Graylog 5.2.5+7eaa89d
  • Service logs, configurations, and environment variables:

Maybe this:

OpenSearchException[OpenSearch exception [type=cluster_manager_not_discovered_exception, reason=FailedToCommitClusterStateException[publication failed]; 
nested: OpenSearchException[publication cancelled before committing: timed out after 30s];]]; 
nested: OpenSearchException[OpenSearch exception [type=failed_to_commit_cluster_state_exception, reason=publication failed]]; 
nested: OpenSearchException[OpenSearch exception [type=exception, reason=publication cancelled before committing: timed out after 30s]];

3. What steps have you already taken to try and solve the problem?

I’ve changed the Stream rule so that it does NOT remove the message from the Default index. If I do this, the messages are retained. As soon as I switch it back so that the messages are only to be stored in the purpose specific “o365_” index, the messages are lost, again.

Hey @accidentaladmin

This looks like something with OpenSearch. I have seen an issue similar while back.
Is a single node or cluster. Is it possible to show your configuration settings?

Normally when that error occurs its when OpenSearch due to issues like network problems, node failures, or configuration.

Hey @gsmith; I just posted a response to Joel in my other thread. Likely the same problem is causing multiple issues. Check it out here: Cluster manager not discovered exception - #3 by accidentaladmin

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.