After upgrading Graylog to the most current version (2.4.1-1), elasticsearch fails to start and I’m unable to figure out why or how to correct it. This is on a CentOS 7 machine, 16GB RAM, 10TB storage. Please advise. Thank you!
Jamen McGranahan
How exactly did you install and upgrade Graylog?
What’s in the logs of Graylog and Elasticsearch?
I updated it with yum update:
yum install graylog-server-2.4
Graylog Log (for just one minute):
2018-01-22T16:06:06.858-06:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #496).
2018-01-22T16:06:10.779-06:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
2018-01-22T16:06:10.779-06:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2018-01-22T16:06:10.990-06:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
2018-01-22T16:06:10.990-06:00 WARN [Aggregates] Indexer is not running, not checking any rules this run.
2018-01-22T16:06:10.990-06:00 INFO [AggregatesMaintenance] removed 0 history items
2018-01-22T16:06:11.239-06:00 WARN [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2018-01-22T16:06:18.499-06:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #496).
2018-01-22T16:06:18.813-06:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #496).
2018-01-22T16:06:20.765-06:00 ERROR [AlertScanner] Skipping alert check <Aggregate rule [Invalid logins] triggered an alert./75089c50-1ca7-41a0-a983-13aa6c86de7e>: Unable to perform terms query (ElasticsearchException)
2018-01-22T16:06:20.770-06:00 ERROR [AlertScanner] Skipping alert check <IEEE alert/11f427e3-972f-47db-9ddd-a7695b0a7127>: Unable to perform terms query (ElasticsearchException)
2018-01-22T16:06:25.305-06:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #501).
Elasticsearch Log (for one minute):
[2018-01-22T12:04:52,564][INFO ][o.e.n.Node ] JVM arguments [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-01-22T12:04:54,005][INFO ][o.e.p.PluginsService ] [CRnzwwY] loaded module [aggs-matrix-stats]
[2018-01-22T12:04:54,005][INFO ][o.e.p.PluginsService ] [CRnzwwY] loaded module [ingest-common]
[2018-01-22T12:04:54,005][INFO ][o.e.p.PluginsService ] [CRnzwwY] loaded module [lang-expression]
[2018-01-22T12:04:54,005][INFO ][o.e.p.PluginsService ] [CRnzwwY] loaded module [lang-groovy]
[2018-01-22T12:04:54,005][INFO ][o.e.p.PluginsService ] [CRnzwwY] loaded module [lang-mustache]
[2018-01-22T12:04:54,005][INFO ][o.e.p.PluginsService ] [CRnzwwY] loaded module [lang-painless]
[2018-01-22T12:04:54,006][INFO ][o.e.p.PluginsService ] [CRnzwwY] loaded module [parent-join]
[2018-01-22T12:04:54,006][INFO ][o.e.p.PluginsService ] [CRnzwwY] loaded module [percolator]
[2018-01-22T12:04:54,006][INFO ][o.e.p.PluginsService ] [CRnzwwY] loaded module [reindex]
[2018-01-22T12:04:54,006][INFO ][o.e.p.PluginsService ] [CRnzwwY] loaded module [transport-netty3]
[2018-01-22T12:04:54,006][INFO ][o.e.p.PluginsService ] [CRnzwwY] loaded module [transport-netty4]
[2018-01-22T12:04:54,007][INFO ][o.e.p.PluginsService ] [CRnzwwY] no plugins loaded
[2018-01-22T12:04:55,241][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalArgumentException: unknown setting [script.indexed] did you mean any of [script.inline, script.ingest]?
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:293) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:256) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:139) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.node.Node.<init>(Node.java:344) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) [elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.6.jar:5.6.6]
[2018-01-22T12:04:55,250][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [script.indexed] did you mean any of [script.inline, script.ingest]?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.6.jar:5.6.6]
Caused by: java.lang.IllegalArgumentException: unknown setting [script.indexed] did you mean any of [script.inline, script.ingest]?
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:293) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:256) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:139) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.node.Node.<init>(Node.java:344) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.6.jar:5.6.6]
... 6 moredid you checked if you additional update Elasticsearch and have only settings in the configuration file that are valid?
current it does not look like.
You’ve probably been using Elasticsearch 2.x before and now upgraded to Elasticsearch 5.6.6.
In this case, make sure to read the upgrade notes of Elasticsearch and especially the sections about breaking changes:
I didn’t realize it had upgraded until you had mentioned it, but you are right - Elasticsearch was upgraded. I tried running the Python script to migrate my data, but it fails:
[root@graylog jamen]# python migrate.py
Traceback (most recent call last):
File "migrate.py", line 7, in <module>
for doc in helpers.scan(es, index=".scripts", preserve_order=True):
File "/usr/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 364, in scan
request_timeout=request_timeout, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 636, in search
doc_type, '_search'), params=params, body=body)
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 314, in perform_request
status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 158, in perform_request
raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x1330410>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x1330410>: Failed to establish a new connection: [Errno -2] Name or service not known)
And I am using the code provided on the page (first without the port; then by adding port 9200 to the script):
from elasticsearch import Elasticsearch,helpers
es = Elasticsearch([
{‘host’: ‘localhost:9200’}
])for doc in helpers.scan(es, index=“.scripts”, preserve_order=True):
es.put_script(lang=doc[‘_type’], id=doc[‘_id’], body=doc[‘_source’])
Now what? Can I revert back Elasticsearch? Or will that corrupt the data?
If no data was written by Elasticsearch 5.x yet, you could roll back to Elasticsearch 2.x and keep using your old data.
For questions about the breaking changes in Elasticsearch 5.x and problems with the migration scripts, please consult https://discuss.elastic.co/.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.