Hi,
Newly I have installed Graylog server, version Graylog v2.2.1+4829190 which is not connecting with Elastic cluster and getting below error messages.
In Webpage:
Elasticsearch cluster unavailable (triggered 2 days ago)
Graylog could not successfully connect to the Elasticsearch cluster. If you’re using multicast, check that it is working in your network and that Elasticsearch is accessible. Also check that the cluster name setting is correct. Read how to fix this in the Elasticsearch setup documentation.
tailf /var/log/graylog-server/server.log
2017-02-24T11:41:25.482+05:30 INFO [zen] [graylog-9961b166-b690-439d-8319-706ccfa8c3b3] failed to send join request to master [{Node-1}{CSmhvv7DScqPSDIVL4XZlg}{172.20.246.42}{10.0.0.2:9300}{master=true}], reason [RemoteTransportException[[Node-1][10.0.0.2:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-9961b166-b690-439d-8319-706ccfa8c3b3][127.0.0.1:9300] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9300]; ]
Configurations:
Graylog:
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxxxxxxxx
root_password_sha2 = xxxxxxxxxxxx
root_timezone = Asia/Kolkata
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://10.0.0.3:12900/api/
rest_thread_pool_size = 16
web_listen_uri = http://10.0.0.3:9000/
web_thread_pool_size = 16
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_time_per_index = 30d
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 2
elasticsearch_replicas = 0
elasticsearch_index_prefix = NetworkLogs
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_cluster_name = Analytics
elasticsearch_discovery_zen_ping_unicast_hosts = 10.0.0.2:9300
elasticsearch_discovery_zen_ping_multicast_enabled = false
elasticsearch_transport_tcp_port = 9300
elasticsearch_cluster_discovery_timeout = 5000
elasticsearch_discovery_initial_state_timeout = 10s
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://10.0.0.3:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
Elasticsearch:
Version - elasticsearch-2.4.3-1.noarch
cluster.name: Analytics
node.master: true
node.data: true
node.name: Node-1
path.data: /elastic/data
script.inline: on
script.indexed: on
bootstrap.mlockall: true
indices.store.throttle.max_bytes_per_sec: 150mb
network.host: 10.0.0.2
http.port: 9200
http.enabled: true
discovery.zen.ping.unicast.hosts: [“10.0.0.2:9300”]
discovery.zen.ping.multicast.enabled: false
Elastic cluster health:
1487917526 11:55:26 Analytics green 1 1 0 0 0 0 0 0 - 100.0%
After this when I added elasticsearch_network_host = 10.0.0.2 in graylog configuration the graylog web goes down and getting below error,
2017-02-24T11:59:00.200+05:30 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[JerseyService [STARTING]], FAILED=[IndexerSetupService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:742) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:555) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:304) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:147) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
2017-02-24T11:59:00.215+05:30 INFO [Server] SIGNAL received. Shutting down.
2017-02-24T11:59:00.229+05:30 INFO [GracefulShutdown] Graceful shutdown initiated.
2017-02-24T11:59:00.233+05:30 INFO [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2017-02-24T11:59:00.233+05:30 WARN [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-02-24T11:59:04.236+05:30 INFO [GracefulShutdown] Goodbye.
Thanks in advance,
rajkumar