Grayog Upgrade to 2.4.6 with Elasticsearch Upgrade to 5.x


(theresa) #1

Hello everyone,

I was finally looking into upgrading my (t)rusty Graylog server which has been running fine since I installed it. This meant that I didn’t touch it since first installing it.

The version that I initially installed was 2.3.0+81f822 and the elasticsearch version that was popular then elasticsearch-2.4.5-1

The reason I didn’t update until now was that I didn’t want to update elasticsearch to the 5.x version because it’s a bit of a pain to update it.
Also I have quite a bit of storage, as a I have to keep 90 days worth of logs, which brings me to an index of 3.3 TB (!)
Each index is rotated daily.

I read before updating elasticsearch to 5.x it is mandatory that you have to re-index it using the API.
Does this mean I have to re-index each of those 90 indices through the API? Otherwise I won’t be able to search through this data?

What is the recommended procedure here? I’d really love to use the benefits of elasticsearch 5.x and the latest Graylog server version.

Unfortunately the problem is that I only have 1 master and 1 datanode (that carries all of the 3.3TB by itself). Which doesn’t help the situation altogether.

I’d appreciate any best-practice & recommendations, even if it’s “don’t touch it…or you’re better off re-installing it from scratch”. Although I’d hate to lose all the data (due to Audit/Compliance reasons).

many thanks in advance,
micsnare


(Jan Doberstein) #2

He @micsnare

reindex is not mandatory - only if the index is created with elasticsearch 1.x.

You can do the update in two steps. First make the update to 2.4 ( http://docs.graylog.org/en/2.4/pages/upgrade/graylog-2.4.html ) after that is finished, make the Elasticsearch upgrade. We have covered the important notice in the documentation ( http://docs.graylog.org/en/2.4/pages/upgrade.html#upgrading-elasticsearch )

Does that make it less scary?


(theresa) #3

Thank you @jan

I like the idea of updating in two steps.
Ok, then I’ll ditch the idea of reindexing, since it didn’t work for me anyhow.

I have checked the documentation before, however I have a question regarding this bit:

After the upgrade you must rotate the indices once manually.

Since I have 91 indices (due to data retention period being 90 days) does this mean I have to rotate the indices manually only once via the maintenance (rotate active write index).
Then all the old indices will still be readable/searchable with the new elasticsearch 5.x version?

Maybe I’m overthinking this too much… I’m still scared, but that’s what snapshots are for, right? :wink:


(theresa) #4

Also, the question is I’m not sure how this error/warning will effect the upgrade process of elasticsearch altogether:

[2018-10-16 13:39:22,936][INFO ][cluster.routing.allocation.decider] [Supremor] low disk watermark [85%] exceeded on [hvnWS-kwQXucrrJJMfbh6A][Supremor][/opt/elasticsearch/graylog/nodes/0] free: 601gb[14.9%], replicas will not be assigned to this node

for now the elasticsearch process is running fine, despite the warning…


(Jan Doberstein) #5

Since I have 91 indices (due to data retention period being 90 days) does this mean I have to rotate the indices manually only once via the maintenance (rotate active write index).
Then all the old indices will still be readable/searchable with the new elasticsearch 5.x version?

after the update of Graylog you run “rotate indices” once for every index and the then newly created indices are fine - the older can be searched without problems. So basically the answer is yes.

Also, the question is I’m not sure how this error/warning will effect the upgrade process of elasticsearch altogether:

I would raise the low water mark a little - because warnings might make your debugging harder. Just search for the low-watermark and high watermark warning. This is regarding the available disk space.


(John Buchanan) #6

I too felt uneasy about upgrading our Elasticsearch cluster from 2.4.6 to (5.6.9 back in June). We have around 1 year and 100TB of data. I ran the upgrade assistant plugin and addressed the issues which popped up, then just followed their full cluster restart upgrade procedure. Disabled shard allocation, performed a flush/synced, shutdown all nodes, then started by upgrading my dedicated masters, brought them online and allowed them to form a cluster, upgraded my data nodes and brought them online, waited for them to recover primary shards and for the cluster to reach status of yellow, then re-enabled shard allocation and waited for the cluster to return to green. It understandably took some time for our cluster to reach yellow and then green, but Graylog was performing normal indexing and search during that time.


(system) #7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.