I was finally looking into upgrading my (t)rusty Graylog server which has been running fine since I installed it. This meant that I didn’t touch it since first installing it.
The version that I initially installed was 2.3.0+81f822 and the elasticsearch version that was popular then elasticsearch-2.4.5-1
The reason I didn’t update until now was that I didn’t want to update elasticsearch to the 5.x version because it’s a bit of a pain to update it.
Also I have quite a bit of storage, as a I have to keep 90 days worth of logs, which brings me to an index of 3.3 TB (!)
Each index is rotated daily.
I read before updating elasticsearch to 5.x it is mandatory that you have to re-index it using the API.
Does this mean I have to re-index each of those 90 indices through the API? Otherwise I won’t be able to search through this data?
What is the recommended procedure here? I’d really love to use the benefits of elasticsearch 5.x and the latest Graylog server version.
Unfortunately the problem is that I only have 1 master and 1 datanode (that carries all of the 3.3TB by itself). Which doesn’t help the situation altogether.
I’d appreciate any best-practice & recommendations, even if it’s “don’t touch it…or you’re better off re-installing it from scratch”. Although I’d hate to lose all the data (due to Audit/Compliance reasons).
Since I have 91 indices (due to data retention period being 90 days) does this mean I have to rotate the indices manually only once via the maintenance (rotate active write index).
Then all the old indices will still be readable/searchable with the new elasticsearch 5.x version?
Maybe I’m overthinking this too much… I’m still scared, but that’s what snapshots are for, right?
Also, the question is I’m not sure how this error/warning will effect the upgrade process of elasticsearch altogether:
[2018-10-16 13:39:22,936][INFO ][cluster.routing.allocation.decider] [Supremor] low disk watermark [85%] exceeded on [hvnWS-kwQXucrrJJMfbh6A][Supremor][/opt/elasticsearch/graylog/nodes/0] free: 601gb[14.9%], replicas will not be assigned to this node
for now the elasticsearch process is running fine, despite the warning…
Since I have 91 indices (due to data retention period being 90 days) does this mean I have to rotate the indices manually only once via the maintenance (rotate active write index).
Then all the old indices will still be readable/searchable with the new elasticsearch 5.x version?
after the update of Graylog you run “rotate indices” once for every index and the then newly created indices are fine - the older can be searched without problems. So basically the answer is yes.
Also, the question is I’m not sure how this error/warning will effect the upgrade process of elasticsearch altogether:
I would raise the low water mark a little - because warnings might make your debugging harder. Just search for the low-watermark and high watermark warning. This is regarding the available disk space.
I too felt uneasy about upgrading our Elasticsearch cluster from 2.4.6 to (5.6.9 back in June). We have around 1 year and 100TB of data. I ran the upgrade assistant plugin and addressed the issues which popped up, then just followed their full cluster restart upgrade procedure. Disabled shard allocation, performed a flush/synced, shutdown all nodes, then started by upgrading my dedicated masters, brought them online and allowed them to form a cluster, upgraded my data nodes and brought them online, waited for them to recover primary shards and for the cluster to reach status of yellow, then re-enabled shard allocation and waited for the cluster to return to green. It understandably took some time for our cluster to reach yellow and then green, but Graylog was performing normal indexing and search during that time.
