Docker compose with elasticsearch fails to connect to cluster


(moire) #1

Hello all :slight_smile:

I am testing a docker compose setup with some basic configuration for testing purposes for now. We want docker for scalability purposes to span docker images upon increase of traffic.

I have followed the guides for the basic setup, but could not get the elasticsearch cluster to connect, it is timing out. Below is the compose file and some env info just to give you an idea and perhaps point me to what I am doing wrong.

The docker compose below:

version: '3'
services:
  graylog-mongo:
    image: mongo:3
    container_name: mongo
  graylog-elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.5.2
    container_name: elasticsearch
    environment:
      - cluster.name=elastics
      - bootstrap.memory_lock=true
      - network.host=_eth0_
    ulimits:
      nproc: 65536
      nofile:
        soft: 65536
        hard: 65536
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200
  graylog:
    image: graylog2/server
    environment:
       GRAYLOG_WEB_ENDPOINT_URI: http://192.168.0.3:9000
       GRAYLOG_REST_ENDPOINT_URI: http://192.168.0.3:9000/api
       GRAYLOG_ELASTICSEARCH_HOSTS: http://192.168.0.3:9200
    links:
      - graylog-mongo:mongo
      - graylog-elasticsearch:elasticsearch
    ports:
      - 9000:9000

After I bring all the containers up I can see on the host that the port forwarding is how it should be:

tcp6       0      0 :::9000                 :::*                    LISTEN      11590/docker-proxy
tcp6       0      0 :::5355                 :::*                    LISTEN      193/systemd-resolve
tcp6       0      0 :::9200                 :::*                    LISTEN      11406/docker-proxy
tcp6       0      0 :::22                     :::*                    LISTEN      240/sshd

The mongo container is up and running, ES one too, but the graylog does not see the ES one (from the logs):

elasticsearch            | [2017-08-21T14:03:39,162][INFO ][o.e.n.Node               ] initialized
elasticsearch            | [2017-08-21T14:03:39,163][INFO ][o.e.n.Node               ] [oh9tPs-] starting ...
graylog_1                | 2017-08-21 14:03:39,358 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
elasticsearch            | [2017-08-21T14:03:39,529][INFO ][o.e.t.TransportService   ] [oh9tPs-] publish_address {172.18.0.3:9300}, bound_addresses {172.18.0.3:9300}
elasticsearch            | [2017-08-21T14:03:39,540][INFO ][o.e.b.BootstrapChecks    ] [oh9tPs-] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
graylog_1                | 2017-08-21 14:03:41,583 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: docker
graylog_1                | 2017-08-21 14:03:41,668 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:5}] to mongo:27017
graylog_1                | 2017-08-21 14:03:53,656 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
graylog_1                | 2017-08-21 14:04:11,904 INFO : org.graylog2.periodical.IndexRetentionThread - Elasticsearch cluster not available, skipping index retention checks.
graylog_1                | 2017-08-21 14:04:11,904 INFO : org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges - Cluster not connected yet, delaying migration until it is reachable.
graylog_1                | 2017-08-21 14:04:22,476 INFO : org.graylog2.periodical.IndexRangesCleanupPeriodical - Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy
graylog_1                | 2017-08-21 14:04:32,478 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Connect to 192.168.0.3:9200 [/192.168.0.3] failed: connect timed out)

I tried different setups for the ES, even network host binding to localhost, but the graylog container always seem to time out while connecting to 9200.

I would appreciate any help with what I am missing. Thanks in advance.


(Jochen) #2

Please refer to http://docs.graylog.org/en/2.3/pages/installation/docker.html for a working Docker compose file.


(moire) #3

Thanks jochen,

I am a bit further, the moment I pasted the logs I saw the problem, but now I am stuck with the below error message:

graylog | 2017-08-21 14:35:03,056 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn’t read cluster health for indices [graylog_]
graylog |
graylog | missing authentication token for REST request [/cluster/health/graylog
] (n/a)
graylog | 2017-08-21 14:35:03,057 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
graylog | 2017-08-21 14:35:32,828 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn’t read cluster health for indices [graylog_*]
graylog

The time out was when I used the wrong IP, but when I used container name for ES: GRAYLOG_ELASTICSEARCH_HOSTS: http://graylog-elasticsearch:9200 it worked.


(system) #4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.