Error : Could not connect to ElasticSearch


(shayz) #1

Hey there,
My graylog was working perfect until I ran a search and then it failed. I have checked the following:
Service status for graylog and ElasticSerach: both running, I reloaded both.
Logs for GrayLog : Could not connect to http://10.111.111.224/9200
Logs for ES: Just Info no errors
It takes a long time to load the web interface. My assumption was the memory but its 4GB and I guess it should be enough. (All the servers -graylog,ES,mongoDB- are on same machine)
All elasticsearch configs were in default which then I changed the network_hosts to the IP.
The results of elasticSearch helath is:
{
“cluster_name” : “graylog”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 4,
“active_shards” : 4,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 100.0


(Jochen) #2

Are you sure this URI is correct?


(shayz) #3

@jochen
yes when I run curl -XGET ‘http://10.111.111.224:9200/_cluster/health?pretty=true
the results looks fine:

{
“cluster_name” : “graylog”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 4,
“active_shards” : 4,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 100.0
}
But have to say that it goes of and on, it goes green when I run (Debian8):
sudo systemctl start elasticsearch.service


(Jochen) #4

That’s not the URI mentioned in the error message.


(shayz) #5

Ah, my bad then, sorry I’m newbie. I assumed if the health check is fine then the elasticsearch_hosts URI is working.
But how I can check it?


(shayz) #6

@jochen
Here are my config files:
Graylog-server.conf:
elasticsearch_hosts = http://10.111.111.224:9200/

elasticsearch.yml:
cluster.name: graylog
network.host: 10.111.111.224
http.port: 9200


(shayz) #7

Update: I have tried to reinstall the elasticsearch using:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo “deb https://artifacts.elastic.co/packages/5.x/apt stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
$ sudo apt update && sudo apt install elasticsearch

but no success


(Jochen) #8

What’s the complete output of these commands?
What’s in the logs of your Elasticsearch node(s)?


(shayz) #9

@jochen
Here is the last page from graylog.log in ES logs

    [2018-04-05T14:08:59,258][INFO ][o.e.n.Node               ] [] initializing ...
[2018-04-05T14:08:59,554][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [93gb], net total_space [99.9gb], spins? [unknown], types [rootfs]
[2018-04-05T14:08:59,554][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-04-05T14:08:59,632][INFO ][o.e.n.Node               ] node name [Z2kVnoJ] derived from node ID [Z2kVnoJTTF-pc7MTXrx_rQ]; set [node.name] to override
[2018-04-05T14:08:59,633][INFO ][o.e.n.Node               ] version[5.6.8], pid[2713], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.16.0-5-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_162/25.162-b12]
[2018-04-05T14:08:59,633][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [aggs-matrix-stats]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [ingest-common]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-expression]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-groovy]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-mustache]
[2018-04-05T14:09:02,031][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-painless]
[2018-04-05T14:09:02,032][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [parent-join]
[2018-04-05T14:09:02,032][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [percolator]
[2018-04-05T14:09:02,032][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [reindex]
[2018-04-05T14:09:02,033][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty3]
[2018-04-05T14:09:02,034][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty4]
[2018-04-05T14:09:02,035][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] no plugins loaded
[2018-04-05T14:09:06,624][INFO ][o.e.d.DiscoveryModule    ] [Z2kVnoJ] using discovery type [zen]
[2018-04-05T14:09:08,067][INFO ][o.e.n.Node               ] initialized
[2018-04-05T14:09:08,067][INFO ][o.e.n.Node               ] [Z2kVnoJ] starting ...
[2018-04-05T14:09:08,872][INFO ][o.e.t.TransportService   ] [Z2kVnoJ] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-04-05T14:09:12,042][INFO ][o.e.c.s.ClusterService   ] [Z2kVnoJ] new_master {Z2kVnoJ}{Z2kVnoJTTF-pc7MTXrx_rQ}{lZ3VHz0ISFqqszMYEexAyg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-04-05T14:09:12,108][INFO ][o.e.h.n.Netty4HttpServerTransport] [Z2kVnoJ] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-04-05T14:09:12,108][INFO ][o.e.n.Node               ] [Z2kVnoJ] started
[2018-04-05T14:09:12,452][INFO ][o.e.g.GatewayService     ] [Z2kVnoJ] recovered [1] indices into cluster_state
[2018-04-05T14:09:13,679][INFO ][o.e.c.r.a.AllocationService] [Z2kVnoJ] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][2]] ...]).
[2018-04-05T14:17:29,450][WARN ][o.e.m.j.JvmGcMonitorService] [Z2kVnoJ] [gc][young][467][86] duration [1s], collections [1]/[2.2s], total [1s]/[7s], memory [398.6mb]->[168.4mb]/[1.9gb], all_pools {[young] [237.3mb]->[4mb]/[266.2mb]}{[survivor] [19mb]->[22.9mb]/[33.2mb]}{[old] [142.3mb]->[142.4mb]/[1.6gb]}
[2018-04-05T14:17:29,789][INFO ][o.e.m.j.JvmGcMonitorService] [Z2kVnoJ] [gc][467] overhead, spent [1s] collecting in the last [2.2s]
[2018-04-05T14:22:34,809][INFO ][o.e.n.Node               ] [] initializing ...
[2018-04-05T14:22:35,104][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [92.6gb], net total_space [99.9gb], spins? [unknown], types [rootfs]
[2018-04-05T14:22:35,105][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-04-05T14:22:35,209][INFO ][o.e.n.Node               ] node name [Z2kVnoJ] derived from node ID [Z2kVnoJTTF-pc7MTXrx_rQ]; set [node.name] to override
[2018-04-05T14:22:35,209][INFO ][o.e.n.Node               ] version[5.6.8], pid[3397], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.16.0-5-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_162/25.162-b12]
[2018-04-05T14:22:35,210][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [aggs-matrix-stats]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [ingest-common]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-expression]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-groovy]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-mustache]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-painless]
[2018-04-05T14:22:37,580][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [parent-join]
[2018-04-05T14:22:37,581][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [percolator]
[2018-04-05T14:22:37,581][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [reindex]
[2018-04-05T14:22:37,582][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty3]
[2018-04-05T14:22:37,582][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty4]
[2018-04-05T14:22:37,583][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] no plugins loaded
[2018-04-05T14:22:41,237][INFO ][o.e.d.DiscoveryModule    ] [Z2kVnoJ] using discovery type [zen]
[2018-04-05T14:22:42,887][INFO ][o.e.n.Node               ] initialized
[2018-04-05T14:22:42,887][INFO ][o.e.n.Node               ] [Z2kVnoJ] starting ...
[2018-04-05T14:22:43,529][INFO ][o.e.t.TransportService   ] [Z2kVnoJ] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-04-05T14:22:46,701][INFO ][o.e.c.s.ClusterService   ] [Z2kVnoJ] new_master {Z2kVnoJ}{Z2kVnoJTTF-pc7MTXrx_rQ}{ibATvcUSQB-tIMrV8ueCqw}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-04-05T14:22:46,750][INFO ][o.e.h.n.Netty4HttpServerTransport] [Z2kVnoJ] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-04-05T14:22:46,751][INFO ][o.e.n.Node               ] [Z2kVnoJ] started
[2018-04-05T14:22:47,122][INFO ][o.e.g.GatewayService     ] [Z2kVnoJ] recovered [1] indices into cluster_state
[2018-04-05T14:23:27,361][INFO ][o.e.c.r.a.AllocationService] [Z2kVnoJ] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][1], [graylog_0][2]] ...]).

after running the commands to install the ES:

    
root@microsoft-graylog:/var/log/elasticsearch# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
OK
root@microsoft-graylog:/var/log/elasticsearch# echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
deb https://artifacts.elastic.co/packages/5.x/apt stable main
root@microsoft-graylog:/var/log/elasticsearch# sudo apt update && sudo apt install elasticsearch
Ign http://ftp.uni-erlangen.de jessie InRelease
Hit http://security.debian.org jessie/updates InRelease
Hit http://ftp.uni-erlangen.de jessie-updates InRelease
Hit http://ppa.launchpad.net xenial InRelease
Ign http://ppa.launchpad.net jessie InRelease
Get:1 http://ftp.debian.org jessie-backports InRelease [166 kB]
Ign http://repo.mongodb.org jessie/mongodb-org/3.6 InRelease
Get:2 http://repo.percona.com jessie InRelease [15.9 kB]
Hit http://repo.saltstack.com jessie InRelease
Hit http://ftp.uni-erlangen.de jessie Release.gpg
Hit http://ftp.uni-erlangen.de jessie Release
Ign http://ppa.launchpad.net jessie Release.gpg
Ign http://ppa.launchpad.net jessie Release
Hit http://repo.mongodb.org jessie/mongodb-org/3.6 Release.gpg
Hit http://security.debian.org jessie/updates/main Sources
Hit http://security.debian.org jessie/updates/main amd64 Packages
Get:3 https://artifacts.elastic.co stable InRelease [310 B]
Ign https://artifacts.elastic.co stable InRelease
Get:4 https://packages.graylog2.org stable InRelease [318 B]
Ign https://packages.graylog2.org stable InRelease
Hit http://repo.mongodb.org jessie/mongodb-org/3.6 Release
Hit http://security.debian.org jessie/updates/main Translation-en
Get:5 http://ftp.uni-erlangen.de jessie-updates/main amd64 Packages/DiffIndex [11.3 kB]
Get:6 http://ftp.uni-erlangen.de jessie-updates/main Translation-en/DiffIndex [3688 B]
Hit http://ftp.uni-erlangen.de jessie-updates/main Sources
Hit https://artifacts.elastic.co stable Release.gpg
Err http://ppa.launchpad.net jessie/main amd64 Packages
  404  Not Found
Hit https://artifacts.elastic.co stable Release
Get:7 http://ftp.debian.org jessie-backports/main amd64 Packages/DiffIndex [27.8 kB]
Ign http://ppa.launchpad.net jessie/main Translation-en
Hit http://ppa.launchpad.net xenial/main amd64 Packages
Hit http://ppa.launchpad.net xenial/main Translation-en
Hit https://packages.graylog2.org stable Release.gpg
Get:8 http://ftp.debian.org jessie-backports/main Translation-en/DiffIndex [27.8 kB]
Ign http://repo.percona.com jessie InRelease
Hit http://ftp.uni-erlangen.de jessie/main Sources
Hit http://ftp.uni-erlangen.de jessie/main amd64 Packages
Ign http://repo.percona.com jessie/main Sources/DiffIndex
Hit http://ftp.uni-erlangen.de jessie/main Translation-en
Hit http://repo.saltstack.com jessie/main amd64 Packages
Get:9 http://repo.mongodb.org jessie/mongodb-org/3.6/main amd64 Packages [4051 B]
Get:10 https://artifacts.elastic.co stable/main Translation-en [329 B]
Ign http://repo.percona.com jessie/main amd64 Packages/DiffIndex
Ign http://repo.mongodb.org jessie/mongodb-org/3.6/main Translation-en
Hit https://packages.graylog2.org stable Release
Hit https://artifacts.elastic.co stable/main amd64 Packages
Get:11 https://artifacts.elastic.co stable/main Translation-en [329 B]
Ign http://repo.saltstack.com jessie/main Translation-en
Get:12 https://artifacts.elastic.co stable/main Translation-en [329 B]
Get:13 https://artifacts.elastic.co stable/main Translation-en [329 B]
Get:14 https://artifacts.elastic.co stable/main Translation-en [329 B]
Ign https://artifacts.elastic.co stable/main Translation-en
Ign http://repo.percona.com jessie/main Translation-en
Get:15 https://packages.graylog2.org stable/2.4 Translation-en [336 B]
Hit http://repo.percona.com jessie/main Sources
Hit http://repo.percona.com jessie/main amd64 Packages
Hit https://packages.graylog2.org stable/2.4 amd64 Packages
Get:16 https://packages.graylog2.org stable/2.4 Translation-en [336 B]
Get:17 https://packages.graylog2.org stable/2.4 Translation-en [336 B]
Get:18 https://packages.graylog2.org stable/2.4 Translation-en [336 B]
Get:19 https://packages.graylog2.org stable/2.4 Translation-en [336 B]
Ign https://packages.graylog2.org stable/2.4 Translation-en
Fetched 257 kB in 4s (61.6 kB/s)
W: GPG error: http://repo.percona.com jessie InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9334A25F8507EFA5
W: Failed to fetch http://ppa.launchpad.net/openjdk-r/ppa/ubuntu/dists/jessie/main/binary-amd64/Packages  404  Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.
root@microsoft-graylog:/var/log/elasticsearch# sudo systemctl daemon-reload
root@microsoft-graylog:/var/log/elasticsearch# sudo systemctl enable elasticsearch.service
Synchronizing state for elasticsearch.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d elasticsearch defaults
Executing /usr/sbin/update-rc.d elasticsearch enable
root@microsoft-graylog:/var/log/elasticsearch# sudo systemctl restart elasticsearch.service

(Jochen) #10

127.0.0.1 is the local loopback interface and only accessible from the same machine.


(shayz) #11

@jochen
It was the default setting and it was working before until I made a wrong query string in search on web interface.
What should I change now?


(Jochen) #12

Make sure that your Graylog nodes (elasticsearch_hosts in the configuration file) are able to communicate with the HTTP API of your Elasticsearch nodes (network.host and related settings in the configuration file).


(shayz) #13

Hey @jochen,
Appreciate your help but still no chance.Here are my config files:

    #elasticsearch.yml:
    cluster.name: graylog
    network.host: 10.1.11.224
    #graylog.conf

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxxx
root_password_sha2 = xxxxx
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://10.1.11.224:9000/api/
web_listen_uri = http://10.1.11.224:9000/
rest_enable_cors = true
elasticsearch_hosts = http://10.1.11.224:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5

I start the elastisearch the Curl returns GREEN after some seconds the whole system dies and the the elasticsearch service stops. (please ignore the IP difference from previous comments as 10.111.111.240)

    [2018-04-06T14:37:19,609][INFO ][o.e.n.Node               ] [] initializing ...
[2018-04-06T14:37:19,924][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [90.8gb], net total_space [99.9gb], spins? [unknown], types [rootfs]
[2018-04-06T14:37:19,924][INFO ][o.e.e.NodeEnvironment    ] [Z2kVnoJ] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-04-06T14:37:20,029][INFO ][o.e.n.Node               ] node name [Z2kVnoJ] derived from node ID [Z2kVnoJTTF-pc7MTXrx_rQ]; set [node.name] to override
[2018-04-06T14:37:20,030][INFO ][o.e.n.Node               ] version[5.6.8], pid[10669], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.16.0-5-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_162/25.162-b12]
[2018-04-06T14:37:20,030][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [aggs-matrix-stats]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [ingest-common]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-expression]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-groovy]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-mustache]
[2018-04-06T14:37:22,124][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [lang-painless]
[2018-04-06T14:37:22,125][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [parent-join]
[2018-04-06T14:37:22,125][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [percolator]
[2018-04-06T14:37:22,126][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [reindex]
[2018-04-06T14:37:22,126][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty3]
[2018-04-06T14:37:22,126][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] loaded module [transport-netty4]
[2018-04-06T14:37:22,127][INFO ][o.e.p.PluginsService     ] [Z2kVnoJ] no plugins loaded
[2018-04-06T14:37:25,978][INFO ][o.e.d.DiscoveryModule    ] [Z2kVnoJ] using discovery type [zen]
[2018-04-06T14:37:27,127][INFO ][o.e.n.Node               ] initialized
[2018-04-06T14:37:27,127][INFO ][o.e.n.Node               ] [Z2kVnoJ] starting ...
[2018-04-06T14:37:27,653][INFO ][o.e.t.TransportService   ] [Z2kVnoJ] publish_address {10.1.11.224:9300}, bound_addresses {10.1.11.224:9300}
[2018-04-06T14:37:27,678][INFO ][o.e.b.BootstrapChecks    ] [Z2kVnoJ] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-04-06T14:37:30,835][INFO ][o.e.c.s.ClusterService   ] [Z2kVnoJ] new_master {Z2kVnoJ}{Z2kVnoJTTF-pc7MTXrx_rQ}{_iEE5FH2Sn2LH5sAVmXghg}{10.1.11.224}{10.1.11.224:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-04-06T14:37:30,886][INFO ][o.e.h.n.Netty4HttpServerTransport] [Z2kVnoJ] publish_address {10.1.11.224:9200}, bound_addresses {10.1.11.224:9200}
[2018-04-06T14:37:30,886][INFO ][o.e.n.Node               ] [Z2kVnoJ] started
[2018-04-06T14:37:31,282][INFO ][o.e.g.GatewayService     ] [Z2kVnoJ] recovered [1] indices into cluster_state
[2018-04-06T14:37:37,055][INFO ][o.e.c.r.a.AllocationService] [Z2kVnoJ] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][1]] ...]).
    #Graylog Log

2018-04-06T14:36:44.735Z INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2018-04-06T14:37:07.961Z ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://10.1.11.224:9200, retrying (attempt #72).
2018-04-06T14:37:10.264Z ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://10.1.11.224:9200, retrying (attempt #159).
2018-04-06T14:37:13.276Z ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://10.1.11.224:9200, retrying (attempt #73).
2018-04-06T14:37:14.735Z ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://10.1.11.224:9200)
2018-04-06T14:37:14.735Z INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2018-04-06T14:37:14.763Z WARN  [KafkaJournal] Journal utilization (101.0%) has gone over 95%.
2018-04-06T14:38:23.411Z ERROR [Messages] Caught exception during bulk indexing: org.apache.http.NoHttpResponseException: 10.1.11.224:9200 failed to respond, retrying (attempt #73).
2018-04-06T14:38:23.395Z WARN  [KafkaJournal] Journal utilization (101.0%) has gone over 95%.
2018-04-06T14:38:23.867Z ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://10.1.11.224:9200)

I have tried different combination fot config files with http.port = 9200 and without it. But no success.

Thanks


(Jochen) #14

What does that mean exactly?


(shayz) #15

@jochen After enabling elasticsearch in less than a minute the I cannot run any commands anymore because the system gets too slow, some times I get error: cannot allocate memory. until the service stops automatically or if I am lucky I can stop it manually. I don’t know what is wrong with this elasticsearch


(Jochen) #16

What are the hardware specs of the machine running Elasticsearch?


(shayz) #17

@jochenits a VM with 4CPU,4GB Ram, 50GB Disc and the OS is Debian8.
are there any compatibility issues for Debian8 and Elasticsearch version?(except the one mentioned in documents)


(Jochen) #18

Are Elasticsearch and Graylog running on the same machine?


(shayz) #19

@jochen yes they are. all on same machine including mongodb


(Jochen) #20

In this case you’ll have to tune the JVM settings of Graylog and Elasticsearch accordingly. I’d recommend using 1 GB heap for Elasticsearch and 1 GB heap for Graylog. MongoDB should be happy with 64 MB of memory.

Make sure that there is some memory left for the operating system to use as a disk buffer. Otherwise the performance will be abysmal.