Error : Could not connect to ElasticSearch

Graylog Central (peer support)
,
1

Hey there,
My graylog was working perfect until I ran a search and then it failed. I have checked the following:
Service status for graylog and ElasticSerach: both running, I reloaded both.
Logs for GrayLog : Could not connect to http://10.111.111.224/9200
Logs for ES: Just Info no errors
It takes a long time to load the web interface. My assumption was the memory but its 4GB and I guess it should be enough. (All the servers -graylog,ES,mongoDB- are on same machine)
All elasticsearch configs were in default which then I changed the network_hosts to the IP.
The results of elasticSearch helath is:
{
“cluster_name” : “graylog”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 4,
“active_shards” : 4,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 100.0

2

Are you sure this URI is correct?

3

@jochen
yes when I run curl -XGET ‘http://10.111.111.224:9200/_cluster/health?pretty=true’
the results looks fine:

{
“cluster_name” : “graylog”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 4,
“active_shards” : 4,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 100.0
}
But have to say that it goes of and on, it goes green when I run (Debian8):
sudo systemctl start elasticsearch.service

4

That’s not the URI mentioned in the error message.

5

Ah, my bad then, sorry I’m newbie. I assumed if the health check is fine then the elasticsearch_hosts URI is working.
But how I can check it?

6

@jochen
Here are my config files:
Graylog-server.conf:
elasticsearch_hosts = http://10.111.111.224:9200/

elasticsearch.yml:
cluster.name: graylog
network.host: 10.111.111.224
http.port: 9200

7

Update: I have tried to reinstall the elasticsearch using:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo “deb https://artifacts.elastic.co/packages/5.x/apt stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
$ sudo apt update && sudo apt install elasticsearch

but no success

8

What’s the complete output of these commands?
What’s in the logs of your Elasticsearch node(s)?

9

@jochen
Here is the last page from graylog.log in ES logs

    [2018-04-05T14:08:59,258][INFO ][o.e.n.Node               ] [] initializing ...
[2018-04-05T14:08:59,554][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [93gb], net total_space [99.9gb], spins? [unknown], types [rootfs]
[2018-04-05T14:08:59,554][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-04-05T14:08:59,632][INFO ][o.e.n.Node               ] node name [Z2kVnoJ] derived from node ID [Z2kVnoJTTF-pc7MTXrx_rQ]; set [node.name] to override
[2018-04-05T14:08:59,633][INFO ][o.e.n.Node               ] version[5.6.8], pid[2713], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.16.0-5-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_162/25.162-b12]
[2018-04-05T14:08:59,633][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [aggs-matrix-stats]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [ingest-common]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-expression]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-groovy]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-mustache]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-painless]
[2018-04-05T14:09:02,032][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [parent-join]
[2018-04-05T14:09:02,032][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [percolator]
[2018-04-05T14:09:02,032][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [reindex]
[2018-04-05T14:09:02,033][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty3]
[2018-04-05T14:09:02,034][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty4]
[2018-04-05T14:09:02,035][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] no plugins loaded
[2018-04-05T14:09:06,624][INFO ][o.e.d.DiscoveryModule    ] [Z2kVnoJ] using discovery type [zen]
[2018-04-05T14:09:08,067][INFO ][o.e.n.Node               ] initialized
[2018-04-05T14:09:08,067][INFO ][o.e.n.Node               ] [Z2kVnoJ] starting ...
[2018-04-05T14:09:08,872][INFO ][o.e.t.TransportService   ] [Z2kVnoJ] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-04-05T14:09:12,042][INFO ][o.e.c.s.ClusterService   ] [Z2kVnoJ] new_master {Z2kVnoJ}{Z2kVnoJTTF-pc7MTXrx_rQ}{lZ3VHz0ISFqqszMYEexAyg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-04-05T14:09:12,108][INFO ][o.e.h.n.Netty4HttpServerTransport] [Z2kVnoJ] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-04-05T14:09:12,108][INFO ][o.e.n.Node               ] [Z2kVnoJ] started
[2018-04-05T14:09:12,452][INFO ][o.e.g.GatewayService     ] [Z2kVnoJ] recovered [1] indices into cluster_state
[2018-04-05T14:09:13,679][INFO ][o.e.c.r.a.AllocationService] [Z2kVnoJ] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][2]] ...]).
[2018-04-05T14:17:29,450][WARN ][o.e.m.j.JvmGcMonitorService] [Z2kVnoJ] [gc][young][467][86] duration [1s], collections [1]/[2.2s], total [1s]/[7s], memory [398.6mb]->[168.4mb]/[1.9gb], all_pools {[young] [237.3mb]->[4mb]/[266.2mb]}{[survivor] [19mb]->[22.9mb]/[33.2mb]}{[old] [142.3mb]->[142.4mb]/[1.6gb]}
[2018-04-05T14:17:29,789][INFO ][o.e.m.j.JvmGcMonitorService] [Z2kVnoJ] [gc][467] overhead, spent [1s] collecting in the last [2.2s]
[2018-04-05T14:22:34,809][INFO ][o.e.n.Node               ] [] initializing ...
[2018-04-05T14:22:35,104][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [92.6gb], net total_space [99.9gb], spins? [unknown], types [rootfs]
[2018-04-05T14:22:35,105][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-04-05T14:22:35,209][INFO ][o.e.n.Node               ] node name [Z2kVnoJ] derived from node ID [Z2kVnoJTTF-pc7MTXrx_rQ]; set [node.name] to override
[2018-04-05T14:22:35,209][INFO ][o.e.n.Node               ] version[5.6.8], pid[3397], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.16.0-5-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_162/25.162-b12]
[2018-04-05T14:22:35,210][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [aggs-matrix-stats]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [ingest-common]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-expression]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-groovy]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-mustache]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-painless]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [parent-join]
[2018-04-05T14:22:37,581][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [percolator]
[2018-04-05T14:22:37,581][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [reindex]
[2018-04-05T14:22:37,582][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty3]
[2018-04-05T14:22:37,582][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty4]
[2018-04-05T14:22:37,583][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] no plugins loaded
[2018-04-05T14:22:41,237][INFO ][o.e.d.DiscoveryModule    ] [Z2kVnoJ] using discovery type [zen]
[2018-04-05T14:22:42,887][INFO ][o.e.n.Node               ] initialized
[2018-04-05T14:22:42,887][INFO ][o.e.n.Node               ] [Z2kVnoJ] starting ...
[2018-04-05T14:22:43,529][INFO ][o.e.t.TransportService   ] [Z2kVnoJ] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-04-05T14:22:46,701][INFO ][o.e.c.s.ClusterService   ] [Z2kVnoJ] new_master {Z2kVnoJ}{Z2kVnoJTTF-pc7MTXrx_rQ}{ibATvcUSQB-tIMrV8ueCqw}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-04-05T14:22:46,750][INFO ][o.e.h.n.Netty4HttpServerTransport] [Z2kVnoJ] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-04-05T14:22:46,751][INFO ][o.e.n.Node               ] [Z2kVnoJ] started
[2018-04-05T14:22:47,122][INFO ][o.e.g.GatewayService     ] [Z2kVnoJ] recovered [1] indices into cluster_state
[2018-04-05T14:23:27,361][INFO ][o.e.c.r.a.AllocationService] [Z2kVnoJ] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][1], [graylog_0][2]] ...]).

after running the commands to install the ES:

    
root@microsoft-graylog:/var/log/elasticsearch# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
OK
root@microsoft-graylog:/var/log/elasticsearch# echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
deb https://artifacts.elastic.co/packages/5.x/apt stable main
root@microsoft-graylog:/var/log/elasticsearch# sudo apt update && sudo apt install elasticsearch
Ign http://ftp.uni-erlangen.de jessie InRelease
Hit http://security.debian.org jessie/updates InRelease
Hit http://ftp.uni-erlangen.de jessie-updates InRelease
Hit http://ppa.launchpad.net xenial InRelease
Ign http://ppa.launchpad.net jessie InRelease
Get:1 http://ftp.debian.org jessie-backports InRelease [166 kB]
Ign http://repo.mongodb.org jessie/mongodb-org/3.6 InRelease
Get:2 http://repo.percona.com jessie InRelease [15.9 kB]
Hit http://repo.saltstack.com jessie InRelease
Hit http://ftp.uni-erlangen.de jessie Release.gpg
Hit http://ftp.uni-erlangen.de jessie Release
Ign http://ppa.launchpad.net jessie Release.gpg
Ign http://ppa.launchpad.net jessie Release
Hit http://repo.mongodb.org jessie/mongodb-org/3.6 Release.gpg
Hit http://security.debian.org jessie/updates/main Sources
Hit http://security.debian.org jessie/updates/main amd64 Packages
Get:3 https://artifacts.elastic.co stable InRelease [310 B]
Ign https://artifacts.elastic.co stable InRelease
Get:4 https://packages.graylog2.org stable InRelease [318 B]
Ign https://packages.graylog2.org stable InRelease
Hit http://repo.mongodb.org jessie/mongodb-org/3.6 Release
Hit http://security.debian.org jessie/updates/main Translation-en
Get:5 http://ftp.uni-erlangen.de jessie-updates/main amd64 Packages/DiffIndex [11.3 kB]
Get:6 http://ftp.uni-erlangen.de jessie-updates/main Translation-en/DiffIndex [3688 B]
Hit http://ftp.uni-erlangen.de jessie-updates/main Sources
Hit https://artifacts.elastic.co stable Release.gpg
Err http://ppa.launchpad.net jessie/main amd64 Packages
  404  Not Found
Hit https://artifacts.elastic.co stable Release
Get:7 http://ftp.debian.org jessie-backports/main amd64 Packages/DiffIndex [27.8 kB]
Ign http://ppa.launchpad.net jessie/main Translation-en
Hit http://ppa.launchpad.net xenial/main amd64 Packages
Hit http://ppa.launchpad.net xenial/main Translation-en
Hit https://packages.graylog2.org stable Release.gpg
Get:8 http://ftp.debian.org jessie-backports/main Translation-en/DiffIndex [27.8 kB]
Ign http://repo.percona.com jessie InRelease
Hit http://ftp.uni-erlangen.de jessie/main Sources
Hit http://ftp.uni-erlangen.de jessie/main amd64 Packages
Ign http://repo.percona.com jessie/main Sources/DiffIndex
Hit http://ftp.uni-erlangen.de jessie/main Translation-en
Hit http://repo.saltstack.com jessie/main amd64 Packages
Get:9 http://repo.mongodb.org jessie/mongodb-org/3.6/main amd64 Packages [4051 B]
Get:10 https://artifacts.elastic.co stable/main Translation-en [329 B]
Ign http://repo.percona.com jessie/main amd64 Packages/DiffIndex
Ign http://repo.mongodb.org jessie/mongodb-org/3.6/main Translation-en
Hit https://packages.graylog2.org stable Release
Hit https://artifacts.elastic.co stable/main amd64 Packages
Get:11 https://artifacts.elastic.co stable/main Translation-en [329 B]
Ign http://repo.saltstack.com jessie/main Translation-en
Get:12 https://artifacts.elastic.co stable/main Translation-en [329 B]
Get:13 https://artifacts.elastic.co stable/main Translation-en [329 B]
Get:14 https://artifacts.elastic.co stable/main Translation-en [329 B]
Ign https://artifacts.elastic.co stable/main Translation-en
Ign http://repo.percona.com jessie/main Translation-en
Get:15 https://packages.graylog2.org stable/2.4 Translation-en [336 B]
Hit http://repo.percona.com jessie/main Sources
Hit http://repo.percona.com jessie/main amd64 Packages
Hit https://packages.graylog2.org stable/2.4 amd64 Packages
Get:16 https://packages.graylog2.org stable/2.4 Translation-en [336 B]
Get:17 https://packages.graylog2.org stable/2.4 Translation-en [336 B]
Get:18 https://packages.graylog2.org stable/2.4 Translation-en [336 B]
Get:19 https://packages.graylog2.org stable/2.4 Translation-en [336 B]
Ign https://packages.graylog2.org stable/2.4 Translation-en
Fetched 257 kB in 4s (61.6 kB/s)
W: GPG error: http://repo.percona.com jessie InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9334A25F8507EFA5
W: Failed to fetch http://ppa.launchpad.net/openjdk-r/ppa/ubuntu/dists/jessie/main/binary-amd64/Packages  404  Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.
root@microsoft-graylog:/var/log/elasticsearch# sudo systemctl daemon-reload
root@microsoft-graylog:/var/log/elasticsearch# sudo systemctl enable elasticsearch.service
Synchronizing state for elasticsearch.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d elasticsearch defaults
Executing /usr/sbin/update-rc.d elasticsearch enable
root@microsoft-graylog:/var/log/elasticsearch# sudo systemctl restart elasticsearch.service
10

127.0.0.1 is the local loopback interface and only accessible from the same machine.

11

@jochen
It was the default setting and it was working before until I made a wrong query string in search on web interface.
What should I change now?

12

Make sure that your Graylog nodes (elasticsearch_hosts in the configuration file) are able to communicate with the HTTP API of your Elasticsearch nodes (network.host and related settings in the configuration file).

1 Like
13

Hey @jochen,
Appreciate your help but still no chance.Here are my config files:

    #elasticsearch.yml:
    cluster.name: graylog
    network.host: 10.1.11.224

    #graylog.conf

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxxx
root_password_sha2 = xxxxx
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://10.1.11.224:9000/api/
web_listen_uri = http://10.1.11.224:9000/
rest_enable_cors = true
elasticsearch_hosts = http://10.1.11.224:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5

I start the elastisearch the Curl returns GREEN after some seconds the whole system dies and the the elasticsearch service stops. (please ignore the IP difference from previous comments as 10.111.111.240)

    [2018-04-06T14:37:19,609][INFO ][o.e.n.Node               ] [] initializing ...
[2018-04-06T14:37:19,924][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [90.8gb], net total_space [99.9gb], spins? [unknown], types [rootfs]
[2018-04-06T14:37:19,924][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-04-06T14:37:20,029][INFO ][o.e.n.Node               ] node name [Z2kVnoJ] derived from node ID [Z2kVnoJTTF-pc7MTXrx_rQ]; set [node.name] to override
[2018-04-06T14:37:20,030][INFO ][o.e.n.Node               ] version[5.6.8], pid[10669], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.16.0-5-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_162/25.162-b12]
[2018-04-06T14:37:20,030][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [aggs-matrix-stats]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [ingest-common]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-expression]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-groovy]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-mustache]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-painless]
[2018-04-06T14:37:22,125][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [parent-join]
[2018-04-06T14:37:22,125][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [percolator]
[2018-04-06T14:37:22,126][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [reindex]
[2018-04-06T14:37:22,126][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty3]
[2018-04-06T14:37:22,126][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty4]
[2018-04-06T14:37:22,127][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] no plugins loaded
[2018-04-06T14:37:25,978][INFO ][o.e.d.DiscoveryModule    ] [Z2kVnoJ] using discovery type [zen]
[2018-04-06T14:37:27,127][INFO ][o.e.n.Node               ] initialized
[2018-04-06T14:37:27,127][INFO ][o.e.n.Node               ] [Z2kVnoJ] starting ...
[2018-04-06T14:37:27,653][INFO ][o.e.t.TransportService   ] [Z2kVnoJ] publish_address {10.1.11.224:9300}, bound_addresses {10.1.11.224:9300}
[2018-04-06T14:37:27,678][INFO ][o.e.b.BootstrapChecks    ] [Z2kVnoJ] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-04-06T14:37:30,835][INFO ][o.e.c.s.ClusterService   ] [Z2kVnoJ] new_master {Z2kVnoJ}{Z2kVnoJTTF-pc7MTXrx_rQ}{_iEE5FH2Sn2LH5sAVmXghg}{10.1.11.224}{10.1.11.224:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-04-06T14:37:30,886][INFO ][o.e.h.n.Netty4HttpServerTransport] [Z2kVnoJ] publish_address {10.1.11.224:9200}, bound_addresses {10.1.11.224:9200}
[2018-04-06T14:37:30,886][INFO ][o.e.n.Node               ] [Z2kVnoJ] started
[2018-04-06T14:37:31,282][INFO ][o.e.g.GatewayService     ] [Z2kVnoJ] recovered [1] indices into cluster_state
[2018-04-06T14:37:37,055][INFO ][o.e.c.r.a.AllocationService] [Z2kVnoJ] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][1]] ...]).

    #Graylog Log

2018-04-06T14:36:44.735Z INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2018-04-06T14:37:07.961Z ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://10.1.11.224:9200, retrying (attempt #72).
2018-04-06T14:37:10.264Z ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://10.1.11.224:9200, retrying (attempt #159).
2018-04-06T14:37:13.276Z ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://10.1.11.224:9200, retrying (attempt #73).
2018-04-06T14:37:14.735Z ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://10.1.11.224:9200)
2018-04-06T14:37:14.735Z INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2018-04-06T14:37:14.763Z WARN  [KafkaJournal] Journal utilization (101.0%) has gone over 95%.
2018-04-06T14:38:23.411Z ERROR [Messages] Caught exception during bulk indexing: org.apache.http.NoHttpResponseException: 10.1.11.224:9200 failed to respond, retrying (attempt #73).
2018-04-06T14:38:23.395Z WARN  [KafkaJournal] Journal utilization (101.0%) has gone over 95%.
2018-04-06T14:38:23.867Z ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://10.1.11.224:9200)

I have tried different combination fot config files with http.port = 9200 and without it. But no success.

Thanks

14

What does that mean exactly?

15

@jochen After enabling elasticsearch in less than a minute the I cannot run any commands anymore because the system gets too slow, some times I get error: cannot allocate memory. until the service stops automatically or if I am lucky I can stop it manually. I don’t know what is wrong with this elasticsearch

16

What are the hardware specs of the machine running Elasticsearch?

17

@jochenits a VM with 4CPU,4GB Ram, 50GB Disc and the OS is Debian8.
are there any compatibility issues for Debian8 and Elasticsearch version?(except the one mentioned in documents)

18

Are Elasticsearch and Graylog running on the same machine?

19

@jochen yes they are. all on same machine including mongodb

20

In this case you’ll have to tune the JVM settings of Graylog and Elasticsearch accordingly. I’d recommend using 1 GB heap for Elasticsearch and 1 GB heap for Graylog. MongoDB should be happy with 64 MB of memory.

Make sure that there is some memory left for the operating system to use as a disk buffer. Otherwise the performance will be abysmal.

2 Likes