Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!
1. Describe your incident:
I am ingesting Office365 logs via @ddbnl’s very awesome officelogcollector. That part seems to be functioning fine.
I use the recommended JSON extractor. I then have a pipeline that strips unwanted/malformed fields and runs the message against some lookup tables.
All this was working fine until the other day when suddenly* messages were no longer being committed to the indices.
*(what I mean is, I know of no event that could have caused this change, nor had I made any configuration changes)
The issue is almost EXACTLY like the one resolved, here: Odd Pipeline/Stream Behavior - #6 by drewmiranda-gl
Problem is, there are no such similar errors to guide me.
2. Describe your environment:
- OS Information:
Debian 11 - Package Version:
Graylog 5.2.5+7eaa89d - Service logs, configurations, and environment variables:
Maybe this:
OpenSearchException[OpenSearch exception [type=cluster_manager_not_discovered_exception, reason=FailedToCommitClusterStateException[publication failed];
nested: OpenSearchException[publication cancelled before committing: timed out after 30s];]];
nested: OpenSearchException[OpenSearch exception [type=failed_to_commit_cluster_state_exception, reason=publication failed]];
nested: OpenSearchException[OpenSearch exception [type=exception, reason=publication cancelled before committing: timed out after 30s]];
3. What steps have you already taken to try and solve the problem?
I’ve changed the Stream rule so that it does NOT remove the message from the Default index. If I do this, the messages are retained. As soon as I switch it back so that the messages are only to be stored in the purpose specific “o365_” index, the messages are lost, again.