Hi,
We are attempting to either upgrade or migrate our graylog from version 3 to version 5. Unfortunately there are alot of data stored in our instance (over 250GB) and minimal downtime is preferred. We are running graylog in a single centos box with 1 node.
If we are to migrate, how do we go about doing this while retaining our data and settings/configurations?
If we are to upgrade, what are the steps? I had a quick look and the hardest part seems to keep mongo, elasticsearch and graylog in sync and in line; but which should be done first? How can I tell what version of mongo and elastic I have?
TL;DR: lots of planning and testing.
I’m not gonna sugar coat it, its gonna be cumbersome. IMO, if you can, build a new cluster and migrate your log sources over to it. I say this because its more or less there will be some downtime and that downtime can be unpredictable, meaning its really difficult to anticipate any and every possible issue ahead of time.
With that preface, you’ll have the following challenges ahead of you:
- Upgrading Graylog
- Upgrading MongoDB
- Migrating to OpenSearch
The doc page here goes into a lot of detail and includes a very important version matrix: Upgrading Graylog
What versions of mongo and elastic are you running?
The guidance is to upgrade graylog in major version increments, so
3.x → 3.x latest version
→ 4.3 (req Mongo >=3.6 and elasticsearch >=6.8)
→ 5.1(req mongo >= 5.0 and elasticsearch 7.10.2)
IF you really do want to upgrade what you have in place, as is, i strongly reccomend you build a test lab (even if it is a single vm, doesn’t need data sources), build a version that matches your current version (you can find old versions via Graylog Package Repository and Graylog Package Repository )
export your existing mongodb and import into the test lab. If you can take a snapshot of the VM, then test the upgrade process for all the components and see where you may get stuck or run into issues.
I’ve done the above process and while it can be a lot of work it made the upgrades go much smoother and gave me more time to resolve issues. I’ve also done ugprades without testing and ended up having to spend a couple of hours to fix, and during that time graylog was down.
I know this is a lot of info so let me know if you have any questions.
Thanks for that.
Yeah, the graylog machine is a virtual machine and I do intend to build a test lab but even before that, it seems that the effort of upgrading is harder than just exporting all 250GB+ of data and import them back in. On the other hand, it seems that the difference is significant enough that this might be a problem. Is this correct?
For the upgrade steps, I thought the process was upgrading the mongo db and then upgrading graylog?
@stantor, You may have another option. Is it absolutely necessary that the data be available in the new system? How long do you retain your log data? Many of our customers opt instead to run the two systems alongside each other until the existing data ages out. Basically, any searches on data prior to upgrade are performed on the old system. Any searches on data collected after the cut over are performed on the new system.
Some even turn off the old system after a short while, and spin it back up only when they need to search historical data. It is by far the least amount of effort, and should result in minimal downtime as you switch sources to the new system and they pick up where they left off.
Thats an interesting one but unfortunately our data retention is about 2 years. So as enticing as this option is, I dont think we can run 2 graylog systems side by side for 2 years.
Before I wish you luck, I will point out that you don’t really have to run them side by side for the full two years. The old one will get queried less and less as time goes by. Most customers say that 90% of their searches go back 30 days or less.
Even if you run two systems for three months, after that, you will only have to spin the old one back up when you need to run queries. By the end of two years, you likely won’t start it very often.
In actual practice, how often do you search logs older than 90 days?
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.