3.1.1 Upgrade problem

We upgraded today during our monthly patch cycle from 3.0.2 to 3.1.1. We also upgraded our Elasticsearch cluster from 6.8.2 to 6.8.3. All upgrades were done serially and Elasticsearch as a rolling upgrade. Afterwards, all connections and traffic flow appears normal to the three Graylog and three Elasticsearch nodes. The problem is that we cannot see any logs after the last Graylog node was upgraded. They appear (from connection and traffic volume) to be loading but do not appear in any query. Logs on both Graylog and Elasticsearch show nothing that would explain this.

Any thoughts?

Graylog does not support rolling upgrades.

Did you recalculate the index range in Graylog? That can be done via API or System > Indices in the UI

We have used this same procedure for all upgrades, including from 2.x to 3.0. We disable shard replication while updating and rebooting Elasticsearch nodes and then enable and wait for green before proceeding to the next node. Of course, we cannot recalc an active index, so that cannot be done.

Traffic volume to both Graylog and Elasticsearch nodes appears normal and all the expected connections are present. The messages just don’t appear in searches. The current Elasticsearch index continues to grow at expected rate. All of our pipelines show typical message processing.

Of course, we cannot recalc an active index, so that cannot be done

Why did you think that is not possible? Just hit the button - it should take only milliseconds.

It fixed the issue, as noted above. A bit more than milliseconds to process ~5TB of indices.