Failed elasticsearch upgrade

[OlympiaLady (Beth)]

I want to move from Graylog 2.2.3 to 2.3.0. I got things working in my test environment, and now started on Production. Things seem different, however. My first step has been to try to upgrade Elasticsearch from 2.4.5 to 5.6 on the latest Debian. I did the upgrade, and after the upgrade, Elasticsearch would not start. I have no Elasticsearch plugins installed, so that is not an issue. I kept my old elasticsearch.yml and init script. I updated the data directory, and the node.master and node.data settings so that is not an issue. The frustrating part is there is nothing in the /var/log/elasticsearch logs or /var/log/messages. The main problem seems to center around:

ERROR: D is not a recognized option

The full output when I do service elasticsearch status is:

root@myserver:~# service elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: failed (Result: exit-code) since Tue 2017-12-26 15:39:44 PST; 1min 21s ago
Docs: http://www.elastic.co
Process: 940 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile={PID_DIR}/elasticsearch.pid -Des.default.path.home={ES_HOME} -Des.default.path.logs={LOG_DIR} -Des.default.path.data={DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=64)
Process: 821 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 940 (code=exited, status=64)

Dec 26 15:39:43 dshsapoly808glp elasticsearch[940]: -V, --version Prints elasticsearch version information and exits
Dec 26 15:39:43 dshsapoly808glp elasticsearch[940]: -d, --daemonize Starts Elasticsearch in the background
Dec 26 15:39:43 dshsapoly808glp elasticsearch[940]: -h, --help show help
Dec 26 15:39:43 dshsapoly808glp elasticsearch[940]: -p, --pidfile Creates a pid file in the specified path on start
Dec 26 15:39:43 dshsapoly808glp elasticsearch[940]: -q, --quiet Turns off standard ouput/error streams logging in console
Dec 26 15:39:43 dshsapoly808glp elasticsearch[940]: -s, --silent show minimal output
Dec 26 15:39:43 dshsapoly808glp elasticsearch[940]: -v, --verbose show verbose output
Dec 26 15:39:43 dshsapoly808glp elasticsearch[940]: ERROR: D is not a recognized option
Dec 26 15:39:44 dshsapoly808glp systemd[1]: elasticsearch.service: main process exited, code=exited, status=64/n/a
Dec 26 15:39:44 dshsapoly808glp systemd[1]: Unit elasticsearch.service entered failed state.

Any help would be appreciated.

[jan (Jan Doberstein)]

IMHO you have somewhere a single D in the config files - that need to be found.

[jtkarvo]

You probably have to use the new init script to get it working. The old yml file might work, but you might need to modify that a bit, too. I had to, but I don’t remember how, anymore. But anyway, after switching to the new init scripts, elasticsearch will tell in the error messages, what parts of the yml file need to change.

[aolague (Alex Olague)]

Was the original init script for ES 2.4.5 modified? I ran into the same issue with the upgrade not overwriting the old init script that had been modified during the our initial Graylog deployment.

See the systemd setting here. You can use the new init script and migrate your modifications to an override config file:

https://www.elastic.co/guide/en/elasticsearch/reference/current/setting-system-settings.html

For me on Centos the new init script was in the same directory as elasticsearch.service (but named elasticsearch.service.rpmnew.) No experience with Debian though, sorry.

As for the elasticsearch.yml file running the migration plugin helped here as it detects options that are no longer valid or have changed.

Alex

[OlympiaLady (Beth)]

Thank you everyone. What I did to solve the problem was to:

1. Upgrade to the latest Java (thought this may have had nothing to do with it).
2. Upgrade to the latest 2.x elasticsearch.
3. Accept the new init script during the elasticsearch 5.x upgrade (I think this was the real fix).