It’s been a while, but we are finally getting back to working on upgrading our Graylog from 3.1 to the latest version 4.3. We are currently taking the approach of upgrading to each minor version.
3.1 -> 3.2
3.2 -> 3.3
3.3 -> 4.0
etc...
We were having issues with our dashboards not being migrated and were unable to complete the upgrade at that time. After revisiting the issue, we were finally able to get around that issue by deleting the views collection for our MongoDB. I assume this is because the migration failed, but didn’t roll back and cause issues with invalid IDs. Even though we created a backup of MongoDB, the “bad” views collection was still in the production database.
We did a mongodump and removed the views collections and were able to get the dashboards migrated properly on our dev cluster. However, the saved searches did not get migrated. We have a saved_searches collection in the database and a searches collection. When doing some testing on the 3.2 dev cluster by creating a new saved search, we noticed that the new saved search appears in the views collection.
2. Describe your environment:
OS Information: Ubuntu 20.04
Package Version: Graylog Server 3.2.6-1
3. What steps have you already taken to try and solve the problem?
When looking at the logs, we are not able to find any errors in regards to the saved searches at all. So, we are not sure if this is another issue with the saved searches not being migrated. It doesn’t seem to be an issue with the views collection, as it did not exist until we performed the Graylog server version upgrade.
4. How can the community help?
We have attempted to lookup related issues, but only seem to find info about the failed migrations. We even look into the following report issue, but since that issue mentioned the bug fix in 3.2.1 and we are using 3.2.6 and not getting an errors in the logs on the initial upgrade. Unfortunately, there doesn’t seem to be a method to import saved searches via API and Graylog 3.1 doesn’t contain the saved search as an option for the Content Packs.
Any advice and next steps would be incredibly helpful as we can finally get upgraded to the latest version of Graylog.
It been a while since I did a extensive upgrade like that. Not sure how you went about doing this upgrade but you have a couple parts. I assume you upgrade Graylog 2.5.x which allows you to use Elasticsearch 6.0. I also assume you’re on MongoDb latest 3.x.x. If this is correct then installing Graylog 3.0.x first and check it out. Remember there were a few different changes in Authentication and configurations. Dashboard are called views at this point and authentications also has change so any configuration or customization for these will create a problem this is just GL 3.0 you still have more changes in Graylog 4.0.x.
Yea, it’s been a while since we have done this too . We are already at GL 3.1 and are working towards 4.x. We are using MongoDB 4.2 and upgraded to our ES cluster to 6.8. Everything appears to be working fine.
After the getting passed the initial issue with the failed Dashboard migrations, we are in a better place and able to move forward. We really just need to more info about how to handle/re-run migrations or how we can get the saved searches re-added to Graylog via MongoDB or something. So far it appears that the latter may need to be done by hand.
We are going to keep pushing forwarding to get to 4.x and will see if anyone has anything other tips. We probably won’t do the prod Graylog upgrade until after the holidays. So helpfully we can get some ideas before then.
I would highly recommend when you get to Graylog 4.3.9 migrate to OpenSearch what ever you do don’t pass Elasticsearch 7.10. I think in couple weeks they should have .deb packages for Ubuntu. I have a couple graylog server I need to migrate from CentOS7 to Ubuntu, just waiting right now.
Don’t forget you can always create a content pack and download it (i.e., dashboards, streams, etc…) then perhaps upload.
I think this idea sounds smart in the first place, but the contentpack created with 3.x will not be able to be imported by 4.x. Or at least this is what I would fear.
My saved searches survived the upgrade from 2.x to 3.x and from 3.x to 4.x - or my colleagues did some magic I am not aware of.
I concur…
but I think this really does depend on what the content pack is made of, meaning if you have stream/s , indices I don’t see a problem yet like you stated, what I have seen is custom settings like permissions/authentications create in 3.3 that are no longer are available in 4.0. The only way around it that I see is removed those settings and then create content pack.
During my upgrade possess from 2.0 to 4.3 these past few years, the main issues I had were from
the new naming convection of Dashboards → Views, Authentication changes, etc…
I still create a content pack even thou it may not work on a newer version , I still can read the JSON file to re-create what could not be upload, just incase all hell breaks loose
@gsmith Thanks for the info ! I think we were going to stick to the Elasticsearch 6.8. Is there a need to switch to Opensearch vs Elsaticsearch?
@ihe Unfortunately because the saved searches didn’t migrate from 3.1 to 3.2, we don’t have a way to import them using the content pack because the saved searches is not available as a content pack sources in 3.1.
While doing some more poking around on graylog MongoDB, we found an entry in the cluster_config collection that had the saved_searches migration listed as complete. The migration for the dashboard failed before, so we rolled back to 3.1, which we have been using for over 2 years now.
We restored selective collections from the MongoDB dump and excluded the cluster_config collection. Here’s the list of the collections we included in the mongorestore.
After performing the upgrade from 3.1 to 3.2, the migration was successful and everything showed up as expected.
It appears our issues was related to cluster_config collection having incorrect data which didn’t trigger another migration, even though we rolled back our deployment from the failed MongoDB migration.
I think we are all good here and added details from what we experienced so others can quickly move forward if they encounter a similar issue.
It depends, Graylog will be phasing Elasticsearch for later versions of GL and for the newer version Of Graylog , I believe 5.0 there not supporting 6.8 but they will be support for ES 7.10. If this is a production setup, I would look into the Release and Change logs for Graylog for Security updates, and again this will depend on this environment.
. We are already at GL 3.1 and are working towards 4.x. We are using MongoDB 4.2 and upgraded to our ES cluster to 6.8. Everything appears to be working fine.
