Change of Elasticsearch data/log path causing error

[zmann (Zac Mann)]

Graylog version 2.4.6 and Elasticsearch version 5.6.12 on Centos 7. I made changes to the data and log path in my elasticsearch.yml file. The reason I made the change is because I was out of memory in the root/var directory which caused elasticsearch to crash. I moved the data and logs path to point to the home directory where I have 700G available. When I check the home directory, the directory is blank. When I check the status of elasticsearch I get the following:

 Jan 30 14:21:46 ElasticSearch1 systemd[1]: Starting Elasticsearch...
Jan 30 14:21:46 ElasticSearch1 systemd[1]: Started Elasticsearch.
Jan 30 14:21:47 ElasticSearch1 elasticsearch[12017]: starts elasticsearch
Jan 30 14:21:47 ElasticSearch1 elasticsearch[12017]: Option                Description
Jan 30 14:21:47 ElasticSearch1 elasticsearch[12017]: ------                -----------
Jan 30 14:21:47 ElasticSearch1 elasticsearch[12017]: -E <KeyValuePair>     Configure a setting
Jan 30 14:21:47 ElasticSearch1 elasticsearch[12017]: -V, --version         Prints elasticsearch version information and exits
Jan 30 14:21:47 ElasticSearch1 systemd[1]: elasticsearch.service: main process exited, code=exited, status=64/n/a
Jan 30 14:21:47 ElasticSearch1 systemd[1]: Unit elasticsearch.service entered failed state.
Jan 30 14:21:47 ElasticSearch1 systemd[1]: elasticsearch.service failed.

Here is my config for elasticsearch.yml

# ======================== Elasticsearch Configuration =========================

#

# NOTE: Elasticsearch comes with reasonable defaults for most settings.

# Before you set out to tweak and tune the configuration, make sure you

# understand what are you trying to accomplish and the consequences.

#

# The primary way of configuring a node is via this file. This template lists

# the most important settings you may want to configure for a production cluster.

#

# Please consult the documentation for further information on configuration options:

# https://www.elastic.co/guide/en/elasticsearch/reference/index.html

#

# ---------------------------------- Cluster -----------------------------------

#

# Use a descriptive name for your cluster:

#

cluster.name: Graylog

#

# ------------------------------------ Node ------------------------------------

#

# Use a descriptive name for the node:

#

#node.name: node-1

#

# Add custom attributes to the node:

#

#node.attr.rack: r1

#

# ----------------------------------- Paths ------------------------------------

#

# Path to directory where to store the data (separate multiple locations by comma):

#

path.data: /home/elasticsearch/data

#

# Path to log files:

#

path.logs: /home/elasticsearch/logs

#

# ----------------------------------- Memory -----------------------------------

#

# Lock the memory on startup:

#

#bootstrap.memory_lock: true

#

# Make sure that the heap size is set to about half the memory available

# on the system and that the owner of the process is allowed to use this

# limit.

#

# Elasticsearch performs poorly when the system is swapping the memory.

#

# ---------------------------------- Network -----------------------------------

#

# Set the bind address to a specific IP (IPv4 or IPv6):

#

network.host: 10.0.38.52

#

# Set a custom port for HTTP:

#

http.port: 9200

#

# For more information, consult the network module documentation.

#

# --------------------------------- Discovery ----------------------------------

#

# Pass an initial list of hosts to perform discovery when new node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

# discovery.zen.ping.unicast.hosts: 10.0.38.52:9300

#

# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):

#

#discovery.zen.minimum_master_nodes: 3

#

# For more information, consult the zen discovery module documentation.

#

# ---------------------------------- Gateway -----------------------------------

#

# Block initial recovery after a full cluster restart until N nodes are started:

#

#gateway.recover_after_nodes: 3

#

# For more information, consult the gateway module documentation.

#

# ---------------------------------- Various -----------------------------------

#

# Require explicit names when deleting indices:

#

#action.destructive_requires_name: true

Thanks in advance.

[jan (Jan Doberstein)]

are you sure that you did not modify any other file, like the JVM options or similar?

Check if the directory you have are accessable by the elasticsearch user and have the same access rights as the default. Depending on your OS you might need to adjust the SELINUX Settings to match.

[zmann (Zac Mann)]

Jan,

I did change the jvm.options file to attempt to update the heap size. Originally, the initial heap size and maximum heap size was set to “Xms2g”. I updated both values to “Xms4g” but I have since reverted back to the original config. Could this modification have caused a corruption?

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms2g
-Xmx2g

I also disabled SELinux on this server.

[zmann (Zac Mann)]

Unfortunately changing the jvm options back to the default did not solve the issue. I’ve confirmed that all of my directories and accessible by the elasticsearch user, and SE Linux is disabled. Do you think I need to start from scratch and reinstall elasticsearch?