Graylog not able to connect to Elasticsearch cluster

Graylog Central (peer support)
1

Issue is similar to Sources page cant connect to api
and I tried steps given in this thread, that didnt fix issue for me
Sorry I added complete post in parts as reply, reply as it wont allow adding links

I get below give error in graylog, even though elasticsearch itself works all good and am able to query on it
but graylog is not able to connect to ES cluster

org.graylog2.periodical.IndexRetentionThread - Elasticsearch cluster not available, skipping index retention checks

I am able to login to graylog web portal, error message from portal is

Could not retrieve index sets.
Fetching index sets list failed: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

any idea if I made mistake or missed any config entry ?

elasticsearch-2.4.4
graylog-2.2.3
mongodb-osx-x86_64-3.4.4

elasticsearch.yml

cluster.name: es_2.4.4
node.name: es_2.4.4_node-1
network.host: 127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
discovery.zen.minimum_master_nodes: 1

elasticsearch

{"name":"es_2.4.4_node-1","cluster_name":"es_2.4.4","cluster_uuid":"tGI1USE3QvOfXa4jCt9TXw","version":{"number":"2.4.4","build_hash":"fcbb46dfd45562a9cf00c604b30849a6dec6b017","build_timestamp":"2017-01-03T11:33:16Z","build_snapshot":false,"lucene_version":"5.5.2"},"tagline":"You Know, for Search"}

elasticsearch cluster health

{"cluster_name":"es_2.4.4","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":11,"active_shards":11,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":11,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":50.0}
2

server.conf

elasticsearch_cluster_name = es_2.4.4
rest_listen_uri = http://localhost:12900/api/
rest_transport_uri = http://localhost:12900/api/
3 
web_listen_uri = http://localhost:9000/
elasticsearch_config_file = /Documents/tools/ES/elasticsearch-2.4.4/config/elasticsearch.yml
elasticsearch_index_prefix = logstash
elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9200
mongodb_uri = mongodb://localhost/graylog

is_master = true
node_id_file = /Documents/tools/graylog-user/node-id
password_secret = xxxxx
root_username = xxxxx
root_password_sha2 = xxxxx
plugin_dir = plugin
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = data/journal
lb_recognition_period_seconds = 3
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

rest_listen_uri 12900 api

{"cluster_id":"bb9e8b96-9770-43e5-b4a3-2f8948a8b4b0","node_id":"a82459e5-b0d2-4ead-ba37-fa27e980eb86","version":"2.2.3+7adc951","tagline":"Manage your logs in the dark and have lasers going and make it look like you're from space!"}
4

[details=graylog logs]2017-05-13 18:33:37,509 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elastic Beats Input 2.2.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-05-13 18:33:37,511 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 2.2.3 [org.graylog.plugins.collector.CollectorPlugin]
2017-05-13 18:33:37,511 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Enterprise Integration Plugin 2.2.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-05-13 18:33:37,512 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: MapWidgetPlugin 2.2.3 [org.graylog.plugins.map.MapWidgetPlugin]
2017-05-13 18:33:37,519 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Pipeline Processor Plugin 2.2.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-05-13 18:33:37,519 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Anonymous Usage Statistics 2.2.3 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-05-13 18:33:37,680 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -Djava.library.path=./…/lib/sigar -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow
2017-05-13 18:33:37,849 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator null
2017-05-13 18:33:38,134 INFO : org.graylog2.shared.system.stats.SigarService - Failed to load SIGAR. Falling back to JMX implementations.
2017-05-13 18:33:39,533 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
2017-05-13 18:33:39,551 INFO : org.graylog2.plugin.system.NodeId - Node ID: a82459e5-b0d2-4ead-ba37-fa27e980eb86
2017-05-13 18:33:39,704 INFO : kafka.log.LogManager - Loading logs.
2017-05-13 18:33:39,748 INFO : kafka.log.LogManager - Logs loading complete.
2017-05-13 18:33:39,749 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at data/journal
2017-05-13 18:33:39,760 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2017-05-13 18:33:39,775 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2017-05-13 18:33:39,813 INFO : org.mongodb.driver.cluster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-05-13 18:33:39,834 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:70}] to localhost:27017
2017-05-13 18:33:39,837 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 4, 4]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=681458}
2017-05-13 18:33:39,842 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:71}] to localhost:27017
2017-05-13 18:33:40,106 INFO : org.elasticsearch.node - [es_2.4.4_node-1] version[2.4.4], pid[47979], build[fcbb46d/2017-01-03T11:33:16Z]
2017-05-13 18:33:40,106 INFO : org.elasticsearch.node - [es_2.4.4_node-1] initializing …
2017-05-13 18:33:40,111 INFO : org.elasticsearch.plugins - [es_2.4.4_node-1] modules [], plugins [graylog-monitor], sites []
2017-05-13 18:33:41,238 INFO : org.elasticsearch.node - [es_2.4.4_node-1] initialized
2017-05-13 18:33:41,286 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy .
2017-05-13 18:33:42,480 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2017-05-13 18:33:42,712 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-13 18:33:42,717 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy .
2017-05-13 18:33:42,769 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-13 18:33:42,821 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-13 18:33:42,879 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-13 18:33:42,930 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-13 18:33:43,467 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 2.2.3+7adc951 starting up
2017-05-13 18:33:43,468 INFO : org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_60-ea on Mac OS X 10.12.4
2017-05-13 18:33:43,468 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: unknown
2017-05-13 18:33:43,468 INFO : org.graylog2.bootstrap.ServerBootstrap - OS: macOS Sierra (10.12.4)
2017-05-13 18:33:43,468 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: x86_64
2017-05-13 18:33:43,470 WARN : org.graylog2.shared.events.DeadEventLoggingListener - Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-05-13 18:33:43,488 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 26 periodicals …
2017-05-13 18:33:43,489 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2017-05-13 18:33:43,490 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2017-05-13 18:33:43,491 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2017-05-13 18:33:43,491 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2017-05-13 18:33:43,491 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2017-05-13 18:33:43,492 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2017-05-13 18:33:43,492 INFO : org.elasticsearch.node - [es_2.4.4_node-1] starting …
2017-05-13 18:33:43,492 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2017-05-13 18:33:43,494 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2017-05-13 18:33:43,495 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2017-05-13 18:33:43,494 INFO : org.graylog2.periodical.IndexRetentionThread - Elasticsearch cluster not available, skipping index retention checks.
2017-05-13 18:33:43,496 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-05-13 18:33:43,496 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2017-05-13 18:33:43,497 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2017-05-13 18:33:43,497 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2017-05-13 18:33:43,497 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2017-05-13 18:33:43,497 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:72}] to localhost:27017
2017-05-13 18:33:43,498 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2017-05-13 18:33:43,498 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2017-05-13 18:33:43,498 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-05-13 18:33:43,499 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:73}] to localhost:27017
2017-05-13 18:33:43,501 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:74}] to localhost:27017
2017-05-13 18:33:43,503 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:75}] to localhost:27017
2017-05-13 18:33:43,505 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:7, serverValue:76}] to localhost:27017
2017-05-13 18:33:43,506 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:8, serverValue:77}] to localhost:27017
2017-05-13 18:33:43,514 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
2017-05-13 18:33:43,518 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2017-05-13 18:33:43,518 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-05-13 18:33:43,519 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-05-13 18:33:43,522 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2017-05-13 18:33:43,522 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-05-13 18:33:43,523 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-05-13 18:33:43,523 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-05-13 18:33:43,527 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2017-05-13 18:33:43,527 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-05-13 18:33:43,529 INFO : org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration - Legacy default stream has no connections, no migration needed.
2017-05-13 18:33:43,574 INFO : org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges - Cluster not connected yet, delaying migration until it is reachable.
2017-05-13 18:33:43,585 INFO : org.elasticsearch.transport - [es_2.4.4_node-1] publish_address {127.0.0.1:9350}, bound_addresses {127.0.0.1:9350}
2017-05-13 18:33:43,589 INFO : org.elasticsearch.discovery - [es_2.4.4_node-1] es_2.4.4/3G7ljvgPQGCiVvGIUctbwA
2017-05-13 18:33:43,780 INFO : org.graylog2.shared.initializers.JerseyService - Enabling CORS for HTTP endpoint
2017-05-13 18:33:46,594 WARN : org.elasticsearch.discovery - [es_2.4.4_node-1] waited for 3s and no initial state was set by the discovery
2017-05-13 18:33:46,594 INFO : org.elasticsearch.node - [es_2.4.4_node-1] started
2017-05-13 18:33:50,988 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [localhost:12900]
2017-05-13 18:33:50,989 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
2017-05-13 18:33:50,989 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at http://localhost:12900/api/
2017-05-13 18:33:51,603 WARN : org.graylog2.initializers.IndexerSetupService - Could not connect to Elasticsearch
2017-05-13 18:33:51,603 INFO : org.graylog2.initializers.IndexerSetupService - If you’re using multicast, check that it is working in your network and that Elasticsearch is accessible. Also check that the cluster name setting is correct.
2017-05-13 18:33:51,604 INFO : org.graylog2.initializers.IndexerSetupService - See
2017-05-13 18:33:52,716 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [localhost:9000]
2017-05-13 18:33:52,716 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer-1] Started.
2017-05-13 18:33:52,717 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at http://localhost:9000/
2017-05-13 18:33:52,717 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
2017-05-13 18:33:52,718 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-05-13 18:33:52,718 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {JournalReader [RUNNING]=3, InputSetupService [RUNNING]=3, BufferSynchronizerService [RUNNING]=3, ConfigurationEtagService [RUNNING]=3, KafkaJournal [RUNNING]=4, OutputSetupService [RUNNING]=5, StreamCacheService [RUNNING]=8, PeriodicalsService [RUNNING]=41, IndexerSetupService [RUNNING]=8114, JerseyService [RUNNING]=9229}
2017-05-13 18:33:52,723 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
2017-05-13 18:33:58,499 INFO : org.graylog2.periodical.IndexRangesCleanupPeriodical - Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy
2017-05-13 18:34:06,006 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:158) ~[graylog.jar:?]
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:70) ~[graylog.jar:?]
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:47) ~[graylog.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.(TransportBroadcastByNodeAction.java:260) ~[graylog.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:238) ~[graylog.jar:?]
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[graylog.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137) ~[graylog.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85) ~[graylog.jar:?]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58) ~[graylog.jar:?]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) ~[graylog.jar:?]
at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52) ~[graylog.jar:?]
at org.graylog2.indexer.elasticsearch.GlobalTimeoutClient.doExecute(GlobalTimeoutClient.java:58) ~[graylog.jar:?]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) ~[graylog.jar:?]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348) ~[graylog.jar:?]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1221) ~[graylog.jar:?]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1541) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.getAllDocCounts(Indices.java:212) ~[graylog.jar:?]
at org.graylog2.indexer.IndexSetStatsCreator.getForIndexSet(IndexSetStatsCreator.java:36) ~[graylog.jar:?]
at java.util.stream.Collectors.lambda$toMap$173(Collectors.java:1321) ~[?:1.8.0_60-ea]
at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_60-ea]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_60-ea]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_60-ea]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_60-ea]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_60-ea][/details]

things to consider from above logs:

2017-05-13 18:33:43,494 INFO : org.graylog2.periodical.IndexRetentionThread - Elasticsearch cluster not     available, skipping index retention checks.....
.
.
2017-05-13 18:34:06,006 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper -     Unhandled exception in REST resource
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not     recovered / initialized];
    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:158) ~    [graylog.jar:?]
    at     org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsA    ction.java:70) ~[graylog.jar:?]....
5

you should probably use a real IP address in the config files instead of the loopback address for the host.

6

tried with real IP address, error stays the same :frowning:

7

okies… this was missing entry
elasticsearch.yml
discovery.zen.ping.unicast.hosts: ["localhost:9300"]
graylog config
elasticsearch_discovery_zen_ping_unicast_hosts = localhost:9300

now there are no exceptions like:
2017-05-13 18:34:06,006 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

But I still see
2017-05-14 00:21:20,652 INFO : org.graylog2.periodical.IndexRetentionThread - Elasticsearch cluster not available, skipping index retention checks.

Question: log says INFO, but as it says cluster not available, what it really means?

[details=Summary]2017-05-14 00:21:13,764 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elastic Beats Input 2.2.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-05-14 00:21:13,766 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 2.2.3 [org.graylog.plugins.collector.CollectorPlugin]
2017-05-14 00:21:13,767 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Enterprise Integration Plugin 2.2.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-05-14 00:21:13,768 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: MapWidgetPlugin 2.2.3 [org.graylog.plugins.map.MapWidgetPlugin]
2017-05-14 00:21:13,776 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Pipeline Processor Plugin 2.2.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-05-14 00:21:13,776 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Anonymous Usage Statistics 2.2.3 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-05-14 00:21:14,937 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -Djava.library.path=./…/lib/sigar -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow
2017-05-14 00:21:15,101 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator null
2017-05-14 00:21:15,368 INFO : org.graylog2.shared.system.stats.SigarService - Failed to load SIGAR. Falling back to JMX implementations.
2017-05-14 00:21:16,750 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
2017-05-14 00:21:16,769 INFO : org.graylog2.plugin.system.NodeId - Node ID: a82459e5-b0d2-4ead-ba37-fa27e980eb86
2017-05-14 00:21:16,936 INFO : kafka.log.LogManager - Loading logs.
2017-05-14 00:21:16,999 INFO : kafka.log.LogManager - Logs loading complete.
2017-05-14 00:21:16,999 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at data/journal
2017-05-14 00:21:17,013 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2017-05-14 00:21:17,027 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[192.168.1.4:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2017-05-14 00:21:17,062 INFO : org.mongodb.driver.cluster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=192.168.1.4:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-05-14 00:21:17,081 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:62}] to 192.168.1.4:27017
2017-05-14 00:21:17,083 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=192.168.1.4:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 4, 4]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=495971}
2017-05-14 00:21:17,089 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:63}] to 192.168.1.4:27017
2017-05-14 00:21:17,322 INFO : org.elasticsearch.node - [es_2.4.4_node-1] version[2.4.4], pid[55742], build[fcbb46d/2017-01-03T11:33:16Z]
2017-05-14 00:21:17,322 INFO : org.elasticsearch.node - [es_2.4.4_node-1] initializing …
2017-05-14 00:21:17,326 INFO : org.elasticsearch.plugins - [es_2.4.4_node-1] modules , plugins [graylog-monitor], sites
2017-05-14 00:21:18,485 INFO : org.elasticsearch.node - [es_2.4.4_node-1] initialized
2017-05-14 00:21:18,535 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy .
2017-05-14 00:21:19,682 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2017-05-14 00:21:19,911 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-14 00:21:19,917 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy .
2017-05-14 00:21:19,973 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-14 00:21:20,058 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-14 00:21:20,092 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-14 00:21:20,123 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-05-14 00:21:20,627 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 2.2.3+7adc951 starting up
2017-05-14 00:21:20,627 INFO : org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_60-ea on Mac OS X 10.12.4
2017-05-14 00:21:20,627 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: unknown
2017-05-14 00:21:20,627 INFO : org.graylog2.bootstrap.ServerBootstrap - OS: macOS Sierra (10.12.4)
2017-05-14 00:21:20,627 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: x86_64
2017-05-14 00:21:20,629 WARN : org.graylog2.shared.events.DeadEventLoggingListener - Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-05-14 00:21:20,644 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 26 periodicals …
2017-05-14 00:21:20,644 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2017-05-14 00:21:20,646 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2017-05-14 00:21:20,646 INFO : org.elasticsearch.node - [es_2.4.4_node-1] starting …
2017-05-14 00:21:20,647 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2017-05-14 00:21:20,648 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2017-05-14 00:21:20,649 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2017-05-14 00:21:20,650 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2017-05-14 00:21:20,650 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2017-05-14 00:21:20,651 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2017-05-14 00:21:20,651 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2017-05-14 00:21:20,652 INFO : org.graylog2.periodical.IndexRetentionThread - Elasticsearch cluster not available, skipping index retention checks.
2017-05-14 00:21:20,652 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-05-14 00:21:20,652 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2017-05-14 00:21:20,653 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2017-05-14 00:21:20,653 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2017-05-14 00:21:20,654 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2017-05-14 00:21:20,653 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:64}] to 192.168.1.4:27017
2017-05-14 00:21:20,654 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2017-05-14 00:21:20,654 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2017-05-14 00:21:20,654 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-05-14 00:21:20,654 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:65}] to 192.168.1.4:27017
2017-05-14 00:21:20,657 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:66}] to 192.168.1.4:27017
2017-05-14 00:21:20,659 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:67}] to 192.168.1.4:27017
2017-05-14 00:21:20,659 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:7, serverValue:68}] to 192.168.1.4:27017
2017-05-14 00:21:20,660 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:9, serverValue:69}] to 192.168.1.4:27017
2017-05-14 00:21:20,660 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:8, serverValue:70}] to 192.168.1.4:27017
2017-05-14 00:21:20,669 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
2017-05-14 00:21:20,669 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2017-05-14 00:21:20,669 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-05-14 00:21:20,670 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-05-14 00:21:20,673 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2017-05-14 00:21:20,675 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-05-14 00:21:20,676 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-05-14 00:21:20,676 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-05-14 00:21:20,679 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2017-05-14 00:21:20,679 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-05-14 00:21:20,683 INFO : org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration - Legacy default stream has no connections, no migration needed.
2017-05-14 00:21:20,724 INFO : org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges - Cluster not connected yet, delaying migration until it is reachable.
2017-05-14 00:21:20,735 INFO : org.elasticsearch.transport - [es_2.4.4_node-1] publish_address {192.168.1.4:9350}, bound_addresses {192.168.1.4:9350}
2017-05-14 00:21:20,740 INFO : org.elasticsearch.discovery - [es_2.4.4_node-1] es_2.4.4/2efwyyU5Rp-hcPwn8a60pg
2017-05-14 00:21:21,011 INFO : org.graylog2.shared.initializers.JerseyService - Enabling CORS for HTTP endpoint
2017-05-14 00:21:23,746 WARN : org.elasticsearch.discovery - [es_2.4.4_node-1] waited for 3s and no initial state was set by the discovery
2017-05-14 00:21:23,746 INFO : org.elasticsearch.node - [es_2.4.4_node-1] started
2017-05-14 00:21:23,803 INFO : org.elasticsearch.cluster.service - [es_2.4.4_node-1] detected_master {es_2.4.4_node-1}{fkeTi0n0Rui5XJWrioBPGQ}{192.168.1.4}{192.168.1.4:9300}, added {{es_2.4.4_node-1}{fkeTi0n0Rui5XJWrioBPGQ}{192.168.1.4}{192.168.1.4:9300},}, reason: zen-disco-receive(from master [{es_2.4.4_node-1}{fkeTi0n0Rui5XJWrioBPGQ}{192.168.1.4}{192.168.1.4:9300}])
2017-05-14 00:21:27,757 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [192.168.1.4:12900]
2017-05-14 00:21:27,759 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
2017-05-14 00:21:27,759 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at http://192.168.1.4:12900/api/
2017-05-14 00:21:29,561 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [192.168.1.4:9000]
2017-05-14 00:21:29,561 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer-1] Started.
2017-05-14 00:21:29,561 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at http://192.168.1.4:9000/
2017-05-14 00:21:29,562 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
2017-05-14 00:21:29,562 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-05-14 00:21:29,562 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {OutputSetupService [RUNNING]=1, ConfigurationEtagService [RUNNING]=3, BufferSynchronizerService [RUNNING]=3, JournalReader [RUNNING]=3, InputSetupService [RUNNING]=3, KafkaJournal [RUNNING]=4, StreamCacheService [RUNNING]=15, PeriodicalsService [RUNNING]=38, IndexerSetupService [RUNNING]=3170, JerseyService [RUNNING]=8917}
2017-05-14 00:21:29,567 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
[/details]

8

try checking elasticsearch cluster health from command line with curl.

https://www.elastic.co/guide/en/elasticsearch/reference/2.2/cluster-health.html

9

Remove the (wrong) elasticsearch_config_file setting. This should not be the configuration file of your Elasticsearch node(s).

10

thx for pointing (wrong) elasticsearch_config_file setting, and I need to get ES health Yellow to Green. checking on these

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.