Graylog not talking to https Elasticsearch

(G) #1

Hello there,
I have successfully setup graylog unsecured in a docker environment and decided to go secured and make the switch to https. Unfortunately, I’m having trouble getting Graylog to talk to Elasticsearch. According to the logs everything comes up but it can’t find my Elasticsearch container and keeps trying to connect and times out. This results in a 502 on my NGINX proxy as it can’t connect. Hopefully someone can tell me what I have setup incorrectly. Here are the logs and details of my setup.

Below is the offensive log indicating it can’t connect to Elasticsearch.

graylog | 2019-04-11 16:29:53,261 INFO : org.graylog2.shared.initializers.JerseyService - Enabling CORS for HTTP endpoint
graylog | 2019-04-11 16:30:07,374 INFO : org.graylog2.periodical.IndexRangesCleanupPeriodical - Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy
graylog | 2019-04-11 16:30:13,148 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [127.0.0.1:9000]
graylog | 2019-04-11 16:30:13,150 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
graylog | 2019-04-11 16:30:13,152 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <127.0.0.1:9000>
graylog | 2019-04-11 16:30:13,154 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
graylog | 2019-04-11 16:30:13,155 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {OutputSetupService [RUNNING]=116, GracefulShutdownService [RUNNING]=136, BufferSynchronizerService [RUNNING]=158, KafkaJournal [RUNNING]=162, ConfigurationEtagService [RUNNING]=246, EtagService [RUNNING]=254, InputSetupService [RUNNING]=260, StreamCacheService [RUNNING]=264, JournalReader [RUNNING]=274, PeriodicalsService [RUNNING]=354, LookupTableService [RUNNING]=747, JerseyService [RUNNING]=20950}
graylog | 2019-04-11 16:30:13,157 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
graylog | 2019-04-11 16:30:13,163 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
graylog | 2019-04-11 16:30:22,271 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to https://127.0.0.1:9200)
graylog | 2019-04-11 16:30:22,272 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
graylog | 2019-04-11 16:30:52,272 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to https://127.0.0.1:9200)
graylog | 2019-04-11 16:30:52,272 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.

Some background about my environment: I run 4 containers via docker-compose version 1.8.0, build f3628c7. Other versions are listed below. As you can see, docker is showing unhealthy because it can’t talk to Elasticsearch.

CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                      PORTS                                                                                                                    NAMES
dc8cdssdyhsd        graylog/graylog:3.0.1                                 "/docker-entrypoint.…"   17 minutes ago      Up 17 minutes (unhealthy)   0.0.0.0:514->514/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:514->514/udp, 0.0.0.0:12201->12201/tcp, 0.0.0.0:12201->12201/udp   graylog
dsdsdfs7f744        mongo:3                                               "docker-entrypoint.s…"   17 minutes ago      Up 17 minutes               27017/tcp                                                                                                                mongodb
dsflkjdsf988        docker.elastic.co/elasticsearch/elasticsearch:6.6.0   "/usr/local/bin/dock…"   17 minutes ago      Up 17 minutes               9200/tcp, 9300/tcp                                                                                                       elasticsearch
aed4e4f45g55        nginx:latest                                          "nginx -g 'daemon of…"   17 minutes ago      Up 17 minutes               0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                                                 nginx

#docker-compose.yml

version: '2'
networks:
  gray1_network:
services:
#Load Balancer
  nginx:
    image: nginx:latest
    container_name: production_nginx
    volumes:
       - ./nginx/nginx.conf:/etc/nginx/nginx.conf
       - ./nginx/mychain.pem:/etc/nginx/mychain.pem
       - ./nginx/mykey.pem:/etc/nginx/mykey.pem
    ports:
      - 80:80
      - 443:443
# MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    image: mongo:3
    volumes:
      - mongo_data:/data/db
    networks:
      - gray1_network
# Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.6.0
    volumes:
      - es_data:/usr/share/elasticsearch/data
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    networks:
      - gray1_network
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      # Disable X-Pack security: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/security-settings.html#general-security-settings
      - xpack.security.enabled=false
      - xpack.watcher.enabled=false
      - xpack.monitoring.enabled=false
      - xpack.security.audit.enabled=false
      - xpack.ml.enabled=false
      - xpack.graph.enabled=false
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 4g
# Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:3.0.1
    volumes:
      - graylog_journal:/usr/share/graylog/data/journal
      - ./graylog.conf:/usr/share/graylog/data/config/graylog.conf
      - ./nginx/mychain.pem:/etc/nginx/mychain.pem
      - ./nginx/mykey.pem:/etc/nginx/mykey.pem
    networks:
      - gray1_network
    environment:
      # CHANGE ME!
      - GRAYLOG_PASSWORD_SECRET=xxxxxxx
          # Password: xxxxxx
      - GRAYLOG_ROOT_PASSWORD_SHA2=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      - GRAYLOG_WEB_ENDPOINT_URI=https://10.201.68.27:9000/api
      - rest_transport_uri = https://10.201.68.27:12900
        links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 514:514
      # Syslog UDP
      - 514:514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_journal:
    driver: local

#graylog.conf

#    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #
# GRAYLOG CONFIGURATION FILE
#    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #    #
#
# This is the Graylog configuration file. The file has to use ISO 8859-1/Latin-1 character encoding.
# Characters that cannot be directly represented in this encoding can be written using Unicode escapes
# as defined in https://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html    #jls-3.3, using the \u prefix.
# For example, \u002c.
#
# * Entries are generally expected to be a single line of the form, one of the following:
#
# propertyName=propertyValue
# propertyName:propertyValue
#
# * White space that appears between the property name and property value is ignored,
#   so the following are equivalent:
#
# name=Stephen
# name = Stephen
#
# * White space at the beginning of the line is also ignored.
#
# * Lines that start with the comment characters ! or     # are ignored. Blank lines are also ignored.
#
# * The property value is generally terminated by the end of the line. White space following the
#   property value is not ignored, and is treated as part of the property value.
#
# * A property value can span several lines if each line is terminated by a backslash (‘\’) character.
#   For example:
#
# targetCities=\
#         Detroit,\
#         Chicago,\
#         Los Angeles
#
#   This is equivalent to targetCities=Detroit,Chicago,Los Angeles (white space at the beginning of lines is ignored).
#
# * The characters newline, carriage return, and tab can be inserted with characters \n, \r, and \t, respectively.
#
# * The backslash character must be escaped as a double backslash. For example:
#
# path=c:\\docs\\doc1
#

# If you are running more than one instances of Graylog server you have to select one of these
# instances as master. The master will perform some periodical tasks that non-masters won't perform.
is_master = true

# The auto-generated node ID will be stored in this file and read after restarts. It is a good idea
# to use an absolute file path here if you are starting Graylog server from init scripts or similar.
node_id_file = /usr/share/graylog/data/config/node-id

# You MUST set a secret to secure/pepper the stored user passwords here. Use at least 64 characters.
# Generate one by using for example: pwgen -N 1 -s 96
password_secret = replacethiswithyourownsecret!

# The default root user is named 'admin'
#root_username = admin

# You MUST specify a hash password for the root user (which you only need to initially set up the
# system and in case you lose connectivity to your authentication backend)
# This password cannot be changed using the API or via the web interface. If you need to change it,
# modify it in this file.
# Create one by using for example: echo -n yourpassword | shasum -a 256
# and put the resulting hash value into the following line

# Default password: xxxxx
# CHANGE THIS!
root_password_sha2 = f8df8saf8ds8afu89auffjjfvldckjmv4u5984mvjg8905mt5mbv5t5908bmv50b

# The email address of the root user.
# Default is empty
#root_email = ""

# The time zone setting of the root user. See http://www.joda.org/joda-time/timezones.html for a list of valid time zones.
# Default is UTC
#root_timezone = UTC

# Set plugin directory here (relative or absolute)
plugin_dir = /usr/share/graylog/plugin

# REST API listen URI. Must be reachable by other Graylog server nodes if you run a cluster.
# When using Graylog Collectors, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
rest_listen_uri = https://127.0.0.1:9000/api/

# REST API transport address. Defaults to the value of rest_listen_uri. Exception: If rest_listen_uri
# is set to a wildcard IP address (0.0.0.0) the first non-loopback IPv4 system address is used.
# If set, this will be promoted in the cluster discovery APIs, so other nodes may try to connect on
# this address and it is used to generate URLs addressing entities in the REST API. (see rest_listen_uri)
# You will need to define this, if your Graylog server is running behind a HTTP proxy that is rewriting
# the scheme, host name or URI.
# This must not contain a wildcard address (0.0.0.0).
#rest_transport_uri = http://192.168.80.2:9000/api/

# Enable CORS headers for REST API. This is necessary for JS-clients accessing the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources from the server.
# This is enabled by default. Uncomment the next line to disable it.
#rest_enable_cors = false

# Enable GZIP support for REST API. This compresses API responses and therefore helps to reduce
# overall round trip times. This is enabled by default. Uncomment the next line to disable it.
#rest_enable_gzip = false

# Enable HTTPS support for the REST API. This secures the communication with the REST API with
# TLS to prevent request forgery and eavesdropping. This is disabled by default. Uncomment the
# next line to enable it.
rest_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the REST API.
rest_tls_cert_file = /etc/nginx/blahblahblah.pem

# The PKCS    #8 private key file in PEM format to use for securing the REST API.
rest_tls_key_file = /etc/nginx/blahblahblah.pem

# The password to unlock the private key used for securing the REST API.
#rest_tls_key_password = secret

# The maximum size of the HTTP request headers in bytes.
#rest_max_header_size = 8192

# The size of the thread pool used exclusively for serving the REST API.
#rest_thread_pool_size = 16

# Comma separated list of trusted proxies that are allowed to set the client address with X-Forwarded-For
# header. May be subnets, or hosts.
#trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128

# Enable the embedded Graylog web interface.
# Default: true
#web_enable = false

# Web interface listen URI.
# Configuring a path for the URI here effectively prefixes all URIs in the web interface. This is a replacement
# for the application.context configuration parameter in pre-2.0 versions of the Graylog web interface.
web_listen_uri = https://127.0.0.1:9000/

# Web interface endpoint URI. This setting can be overriden on a per-request basis with the X-Graylog-Server-URL header.
# Default: $rest_transport_uri
web_endpoint_uri = https://test-graylog.mydomain.com:9000/api

# Enable CORS headers for the web interface. This is necessary for JS-clients accessing the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources from the server.
#web_enable_cors = false

# Enable/disable GZIP support for the web interface. This compresses HTTP responses and therefore helps to reduce
# overall round trip times. This is enabled by default. Uncomment the next line to disable it.
#web_enable_gzip = false

# Enable HTTPS support for the web interface. This secures the communication of the web browser with the web interface
# using TLS to prevent request forgery and eavesdropping.
# This is disabled by default. Uncomment the next line to enable it and see the other related configuration settings.
web_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the web interface.
web_tls_cert_file = /etc/nginx/blahblahblah.pem

# The PKCS    #8 private key file in PEM format to use for securing the web interface.
web_tls_key_file = /etc/nginx/blahblahblah.pem

# The password to unlock the private key used for securing the web interface.
#web_tls_key_password = secret

# The maximum size of the HTTP request headers in bytes.
#web_max_header_size = 8192

# The size of the thread pool used exclusively for serving the web interface.
#web_thread_pool_size = 16

# List of Elasticsearch hosts Graylog should connect to.
# Need to be specified as a comma-separated list of valid URIs for the http ports of your elasticsearch nodes.
# If one or more of your elasticsearch hosts require authentication, include the credentials in each node URI that
# requires authentication.
#
# Default: http://127.0.0.1:9200
#elasticsearch_hosts = https://127.0.0.1:9200
elasticsearch_hosts = https://0.0.0.0:9200

# Maximum amount of time to wait for successfull connection to Elasticsearch HTTP port.
#
# Default: 10 Seconds
#elasticsearch_connect_timeout = 10s

# Maximum amount of time to wait for reading back a response from an Elasticsearch server.
#
# Default: 60 seconds
#elasticsearch_socket_timeout = 60s

# Maximum idle time for an Elasticsearch connection. If this is exceeded, this connection will
# be tore down.
#
# Default: inf
#elasticsearch_idle_timeout = -1s

# Maximum number of total connections to Elasticsearch.
#
# Default: 20
#elasticsearch_max_total_connections = 20

# Maximum number of total connections per Elasticsearch route (normally this means per
# elasticsearch server).
#
# Default: 2
#elasticsearch_max_total_connections_per_route = 2

# Maximum number of times Graylog will retry failed requests to Elasticsearch.
#
# Default: 2
#elasticsearch_max_retries = 2

# Enable automatic Elasticsearch node discovery through Nodes Info,
# see https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cluster-nodes-info.html
#
# WARNING: Automatic node discovery does not work if Elasticsearch requires authentication, e. g. with Shield.
#
# Default: false
#elasticsearch_discovery_enabled = true

# Filter for including/excluding Elasticsearch nodes in discovery according to their custom attributes,
# see https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cluster.html    #cluster-nodes
#
# Default: empty
#elasticsearch_discovery_filter = rack:42

# Frequency of the Elasticsearch node discovery.
#
# Default: 30s
# elasticsearch_discovery_frequency = 30s

# Enable payload compression for Elasticsearch requests.
#
# Default: false
#elasticsearch_compression_enabled = true

# Disable checking the version of Elasticsearch for being compatible with this Graylog release.
# WARNING: Using Graylog with unsupported and untested versions of Elasticsearch may lead to data loss!
#elasticsearch_disable_version_check = true

# Disable message retention on this node, i. e. disable Elasticsearch index rotation.
#no_retention = false

# Do you want to allow searches with leading wildcards? This can be extremely resource hungry and should only
# be enabled with care. See also: http://docs.graylog.org/en/2.1/pages/queries.html
allow_leading_wildcard_searches = false

# Do you want to allow searches to be highlighted? Depending on the size of your messages this can be memory hungry and
# should only be enabled after making sure your Elasticsearch cluster has enough memory.
allow_highlighting = false

# Global request timeout for Elasticsearch requests (e. g. during search, index creation, or index time-range
# calculations) based on a best-effort to restrict the runtime of Elasticsearch operations.
# Default: 1m
#elasticsearch_request_timeout = 1m

# Global timeout for index optimization (force merge) requests.
# Default: 1h
#elasticsearch_index_optimization_timeout = 1h

# Maximum number of concurrently running index optimization (force merge) jobs.
# If you are using lots of different index sets, you might want to increase that number.
# Default: 20
#elasticsearch_index_optimization_jobs = 20

# Time interval for index range information cleanups. This setting defines how often stale index range information
# is being purged from the database.
# Default: 1h
#index_ranges_cleanup_interval = 1h

# Batch size for the Elasticsearch output. This is the maximum (!) number of messages the Elasticsearch output
# module will get at once and write to Elasticsearch in a batch call. If the configured batch size has not been
# reached within output_flush_interval seconds, everything that is available will be flushed at once. Remember
# that every outputbuffer processor manages its own batch and performs its own batch write calls.
# ("outputbuffer_processors" variable)
output_batch_size = 500

# Flush interval (in seconds) for the Elasticsearch output. This is the maximum amount of time between two
# batches of messages written to Elasticsearch. It is only effective at all if your minimum number of messages
# for this time period is less than output_batch_size * outputbuffer_processors.
output_flush_interval = 1

# As stream outputs are loaded only on demand, an output which is failing to initialize will be tried over and
# over again. To prevent this, the following configuration options define after how many faults an output will
# not be tried again for an also configurable amount of seconds.
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30

# The number of parallel running processors.
# Raise this number if your buffers are filling up.
processbuffer_processors = 5
outputbuffer_processors = 3

# The following settings (outputbuffer_processor_*) configure the thread pools backing each output buffer processor.
# See https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html for technical details

# When the number of threads is greater than the core (see outputbuffer_processor_threads_core_pool_size),
# this is the maximum time in milliseconds that excess idle threads will wait for new tasks before terminating.
# Default: 5000
#outputbuffer_processor_keep_alive_time = 5000

# The number of threads to keep in the pool, even if they are idle, unless allowCoreThreadTimeOut is set
# Default: 3
#outputbuffer_processor_threads_core_pool_size = 3

# The maximum number of threads to allow in the pool
# Default: 30
#outputbuffer_processor_threads_max_pool_size = 30

# UDP receive buffer size for all message inputs (e. g. SyslogUDPInput).
#udp_recvbuffer_sizes = 1048576

# Wait strategy describing how buffer processors wait on a cursor sequence. (default: sleeping)
# Possible types:
#  - yielding
#     Compromise between performance and CPU usage.
#  - sleeping
#     Compromise between performance and CPU usage. Latency spikes can occur after quiet periods.
#  - blocking
#     High throughput, low latency, higher CPU usage.
#  - busy_spinning
#     Avoids syscalls which could introduce latency jitter. Best when threads can be bound to specific CPU cores.
processor_wait_strategy = blocking

# Size of internal ring buffers. Raise this if raising outputbuffer_processors does not help anymore.
# For optimum performance your LogMessage objects in the ring buffer should fit in your CPU L3 cache.
# Must be a power of 2. (512, 1024, 2048, ...)
ring_size = 65536

inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking

# Enable the disk based message journal.
message_journal_enabled = true

# The directory which will be used to store the message journal. The directory must me exclusively used by Graylog and
# must not contain any other files than the ones created by Graylog itself.
#
# ATTENTION:
#   If you create a seperate partition for the journal files and use a file system creating directories like 'lost+found'
#   in the root directory, you need to create a sub directory for your journal.
#   Otherwise Graylog will log an error message that the journal is corrupt and Graylog will not start.
message_journal_dir = /usr/share/graylog/data/journal

# Journal hold messages before they could be written to Elasticsearch.
# For a maximum of 12 hours or 5 GB whichever happens first.
# During normal operation the journal will be smaller.
#message_journal_max_age = 12h
#message_journal_max_size = 5gb

#message_journal_flush_age = 1m
#message_journal_flush_interval = 1000000
#message_journal_segment_age = 1h
#message_journal_segment_size = 100mb

# Number of threads used exclusively for dispatching internal events. Default is 2.
#async_eventbus_processors = 2

# How many seconds to wait between marking node as DEAD for possible load balancers and starting the actual
# shutdown process. Set to 0 if you have no status checking load balancers in front.
lb_recognition_period_seconds = 3

# Journal usage percentage that triggers requesting throttling for this server node from load balancers. The feature is
# disabled if not set.
#lb_throttle_threshold_percentage = 95

# Every message is matched against the configured streams and it can happen that a stream contains rules which
# take an unusual amount of time to run, for example if its using regular expressions that perform excessive backtracking.
# This will impact the processing of the entire server. To keep such misbehaving stream rules from impacting other
# streams, Graylog limits the execution time for each stream.
# The default values are noted below, the timeout is in milliseconds.
# If the stream matching for one stream took longer than the timeout value, and this happened more than "max_faults" times
# that stream is disabled and a notification is shown in the web interface.
#stream_processing_timeout = 2000
#stream_processing_max_faults = 3

# Length of the interval in seconds in which the alert conditions for all streams should be checked
# and alarms are being sent.
#alert_check_interval = 60

# Since 0.21 the Graylog server supports pluggable output modules. This means a single message can be written to multiple
# outputs. The next setting defines the timeout for a single output module, including the default output module where all
# messages end up.
#
# Time in milliseconds to wait for all message outputs to finish writing a single message.
#output_module_timeout = 10000

# Time in milliseconds after which a detected stale master node is being rechecked on startup.
#stale_master_timeout = 2000

# Time in milliseconds which Graylog is waiting for all threads to stop on shutdown.
#shutdown_timeout = 30000

# MongoDB connection string
# See https://docs.mongodb.com/manual/reference/connection-string/ for details
mongodb_uri = mongodb://mongo/graylog

# Authenticate against the MongoDB server
#mongodb_uri = mongodb://grayloguser:secret@mongo:27017/graylog

# Use a replica set instead of a single host
#mongodb_uri = mongodb://grayloguser:secret@mongo:27017,mongo:27018,mongo:27019/graylog

# Increase this value according to the maximum connections your MongoDB server can handle from a single client
# if you encounter MongoDB connection problems.
mongodb_max_connections = 100

# Number of threads allowed to be blocked by MongoDB connections multiplier. Default: 5
# If mongodb_max_connections is 100, and mongodb_threads_allowed_to_block_multiplier is 5,
# then 500 threads can block. More than that and an exception will be thrown.
# http://api.mongodb.com/java/current/com/mongodb/MongoOptions.html    #threadsAllowedToBlockForConnectionMultiplier
mongodb_threads_allowed_to_block_multiplier = 5

# Drools Rule File (Use to rewrite incoming log messages)
# See: http://docs.graylog.org/en/2.1/pages/drools.html
#rules_file = /etc/graylog/server/rules.drl

# Email transport
#transport_email_enabled = false
#transport_email_hostname = mail.example.com
#transport_email_port = 587
#transport_email_use_auth = true
#transport_email_use_tls = true
#transport_email_use_ssl = true
#transport_email_auth_username = you@example.com
#transport_email_auth_password = secret
#transport_email_subject_prefix = [graylog]
#transport_email_from_email = graylog@example.com

# Specify and uncomment this if you want to include links to the stream in your stream alert mails.
# This should define the fully qualified base url to your web interface exactly the same way as it is accessed by your users.
#transport_email_web_interface_url = https://graylog.example.com

# The default connect timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 5s
#http_connect_timeout = 5s

# The default read timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 10s
#http_read_timeout = 10s

# The default write timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 10s
#http_write_timeout = 10s

# HTTP proxy for outgoing HTTP connections
#http_proxy_uri = http://127.0.0.1:443

# The threshold of the garbage collection runs. If GC runs take longer than this threshold, a system notification
# will be generated to warn the administrator about possible problems with the system. Default is 1 second.
#gc_warning_threshold = 1s

# Connection timeout for a configured LDAP server (e. g. ActiveDirectory) in milliseconds.
#ldap_connection_timeout = 2000

# Disable the use of SIGAR for collecting system stats
#disable_sigar = false

# The default cache time for dashboard widgets. (Default: 10 seconds, minimum: 1 second)
#dashboard_widget_default_cache_time = 10s

# Automatically load content packs in "content_packs_dir" on the first start of Graylog.
content_packs_loader_enabled = true

# The directory which contains content packs which should be loaded on the first start of Graylog.
content_packs_dir = /usr/share/graylog/data/contentpacks

# A comma-separated list of content packs (files in "content_packs_dir") which should be applied on
# the first start of Graylog.
# Default: empty
content_packs_auto_load = grok-patterns.json

# For some cluster-related REST requests, the node must query all other nodes in the cluster. This is the maximum number
# of threads available for this. Increase it, if '/cluster/*' requests take long to complete.
# Should be rest_thread_pool_size * average_cluster_size if you have a high number of concurrent users.
proxied_requests_thread_pool_size = 32

#elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 127.0.0.1

#nginx.conf

events { }
http {

  upstream backend {
    server test-graylog.mydomain.com:9000;
    keepalive 32;
  }

  server {
    listen              80;
    listen              443 ssl;
    server_name         test-graylog.mydomain.com;
    ssl_certificate     mychain.pem;
    ssl_certificate_key mykey.pem;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

  location / {
    proxy_pass http://backend;
    }
   }
 }

Thanks folks!

0 Likes

(G) #2

Here is the tail end of my docker-compose logs.

elasticsearch  | [2019-04-11T16:51:46,854][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded module [x-pack-deprecation]
elasticsearch  | [2019-04-11T16:51:46,854][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded module [x-pack-graph]
elasticsearch  | [2019-04-11T16:51:46,854][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded module [x-pack-ilm]
graylog        | 2019-04-11 16:51:51,630 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:1}] to mongo:27017
graylog        | 2019-04-11 16:51:51,638 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongo:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 10]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=6582942}
nginx | 2019/04/11 16:52:44 [error] 6#6: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.xxx.xxx.100, server: test-graylog.mydomain.com, request: "GET / HTTP/1.1", upstream: "http://10.xxx.xxx.27:9000/", host: "test-graylog.mydomain.com"
graylog        | 2019-04-11 16:51:51,677 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:2}] to mongo:27017
elasticsearch  | [2019-04-11T16:51:46,854][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded module [x-pack-logstash]
nginx | 10.xxx.xxx.100 - - [11/Apr/2019:16:52:44 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "https://test-graylog.mydomain.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
elasticsearch  | [2019-04-11T16:51:46,854][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded module [x-pack-ml]
nginx | 2019/04/11 16:52:44 [error] 6#6: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.xxx.xxx.100, server: test-graylog.mydomain.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.xxx.xxx.27:9000/favicon.ico", host: "test-graylog.mydomain.com", referrer: "https://test-graylog.mydomain.com/"
elasticsearch  | [2019-04-11T16:51:46,854][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded module [x-pack-monitoring]
graylog        | 2019-04-11 16:51:52,295 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 1 servers: [https://0.0.0.0:9200]
nginx | 10.xxx.xxx.100 - - [11/Apr/2019:16:52:44 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "https://test-graylog.mydomain.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
elasticsearch  | [2019-04-11T16:51:46,854][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded module [x-pack-rollup]
nginx | 2019/04/11 16:52:44 [error] 6#6: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.xxx.xxx.100, server: test-graylog.mydomain.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.xxx.xxx.27:9000/favicon.ico", host: "test-graylog.mydomain.com", referrer: "https://test-graylog.mydomain.com/"
elasticsearch  | [2019-04-11T16:51:46,854][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded module [x-pack-security]
graylog        | 2019-04-11 16:51:52,297 INFO : io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager
nginx | 2019/04/11 16:52:45 [error] 6#6: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.xxx.xxx.100, server: test-graylog.mydomain.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.xxx.xxx.27:9000/favicon.ico", host: "test-graylog.mydomain.com", referrer: "https://test-graylog.mydomain.com/"
elasticsearch  | [2019-04-11T16:51:46,855][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded module [x-pack-sql]
elasticsearch  | [2019-04-11T16:51:46,855][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded module [x-pack-upgrade]
nginx | 10.xxx.xxx.100 - - [11/Apr/2019:16:52:45 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "https://test-graylog.mydomain.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
graylog        | 2019-04-11 16:51:52,496 INFO : io.searchbox.client.JestClientFactory - Using custom ObjectMapper instance
graylog        | 2019-04-11 16:51:52,502 INFO : io.searchbox.client.JestClientFactory - Node Discovery disabled...
graylog        | 2019-04-11 16:51:52,503 INFO : io.searchbox.client.JestClientFactory - Idle connection reaping disabled...
elasticsearch  | [2019-04-11T16:51:46,855][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded module [x-pack-watcher]
graylog        | 2019-04-11 16:51:52,903 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
elasticsearch  | [2019-04-11T16:51:46,855][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded plugin [ingest-geoip]
graylog        | 2019-04-11 16:51:53,543 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
elasticsearch  | [2019-04-11T16:51:46,855][INFO ][o.e.p.PluginsService     ] [ThfhcRO] loaded plugin [ingest-user-agent]
graylog        | 2019-04-11 16:51:53,616 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
elasticsearch  | [2019-04-11T16:51:55,938][INFO ][o.e.d.DiscoveryModule    ] [ThfhcRO] using discovery type [zen] and host providers [settings]
elasticsearch  | [2019-04-11T16:51:57,199][INFO ][o.e.n.Node               ] [ThfhcRO] initialized
graylog        | 2019-04-11 16:51:53,687 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:3}] to mongo:27017
elasticsearch  | [2019-04-11T16:51:57,199][INFO ][o.e.n.Node               ] [ThfhcRO] starting ...
graylog        | 2019-04-11 16:51:53,700 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
elasticsearch  | [2019-04-11T16:51:57,490][INFO ][o.e.t.TransportService   ] [ThfhcRO] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
elasticsearch  | [2019-04-11T16:52:00,640][INFO ][o.e.c.s.MasterService    ] [ThfhcRO] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {ThfhcRO}{ThfhcROrS92b02eX_BdnQQ}{MPA0kp4OSBGShPQz4Ya2yw}{localhost}{127.0.0.1:9300}{xpack.installed=true}
graylog        | 2019-04-11 16:51:53,776 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
elasticsearch  | [2019-04-11T16:52:00,660][INFO ][o.e.c.s.ClusterApplierService] [ThfhcRO] new_master {ThfhcRO}{ThfhcROrS92b02eX_BdnQQ}{MPA0kp4OSBGShPQz4Ya2yw}{localhost}{127.0.0.1:9300}{xpack.installed=true}, reason: apply cluster state (from master [master {ThfhcRO}{ThfhcROrS92b02eX_BdnQQ}{MPA0kp4OSBGShPQz4Ya2yw}{localhost}{127.0.0.1:9300}{xpack.installed=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
elasticsearch  | [2019-04-11T16:52:00,735][INFO ][o.e.h.n.Netty4HttpServerTransport] [ThfhcRO] publish_address {172.21.0.2:9200}, bound_addresses {0.0.0.0:9200}
elasticsearch  | [2019-04-11T16:52:00,735][INFO ][o.e.n.Node               ] [ThfhcRO] started
elasticsearch  | [2019-04-11T16:52:01,340][INFO ][o.e.l.LicenseService     ] [ThfhcRO] license [af612e28-a9e5-4a92-a390-ff369f20d30b] mode [basic] - valid
elasticsearch  | [2019-04-11T16:52:01,345][INFO ][o.e.g.GatewayService     ] [ThfhcRO] recovered [1] indices into cluster_state
elasticsearch  | [2019-04-11T16:52:02,364][INFO ][o.e.c.r.a.AllocationService] [ThfhcRO] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0], [graylog_0][1], [graylog_0][2]] ...]).
graylog        | 2019-04-11 16:51:53,841 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
graylog        | 2019-04-11 16:51:53,882 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
graylog        | 2019-04-11 16:51:54,999 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 3.0.1+de74b68 starting up
graylog        | 2019-04-11 16:51:55,002 INFO : org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_212 on Linux 4.15.0-33-generic
graylog        | 2019-04-11 16:51:55,004 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: docker
graylog        | 2019-04-11 16:51:55,005 INFO : org.graylog2.bootstrap.ServerBootstrap - OS: Debian GNU/Linux 9 (stretch) (debian)
graylog        | 2019-04-11 16:51:55,007 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: amd64
graylog        | 2019-04-11 16:51:55,088 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 27 periodicals ...
graylog        | 2019-04-11 16:51:55,090 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
graylog        | 2019-04-11 16:51:55,151 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
graylog        | 2019-04-11 16:51:55,154 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
graylog        | 2019-04-11 16:51:55,156 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
graylog        | 2019-04-11 16:51:55,173 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
graylog        | 2019-04-11 16:51:55,175 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
graylog        | 2019-04-11 16:51:55,177 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
graylog        | 2019-04-11 16:51:55,183 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:4}] to mongo:27017
graylog        | 2019-04-11 16:51:55,190 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
graylog        | 2019-04-11 16:51:55,192 INFO : org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration - Legacy default stream has no connections, no migration needed.
graylog        | 2019-04-11 16:51:55,194 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
graylog        | 2019-04-11 16:51:55,280 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
graylog        | 2019-04-11 16:51:55,293 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
graylog        | 2019-04-11 16:51:55,293 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
graylog        | 2019-04-11 16:51:55,294 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
graylog        | 2019-04-11 16:51:55,297 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
graylog        | 2019-04-11 16:51:55,300 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
graylog        | 2019-04-11 16:51:55,307 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
graylog        | 2019-04-11 16:51:55,322 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
graylog        | 2019-04-11 16:51:55,375 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:7}] to mongo:27017
graylog        | 2019-04-11 16:51:55,383 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:7, serverValue:5}] to mongo:27017
graylog        | 2019-04-11 16:51:55,403 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:6}] to mongo:27017
graylog        | 2019-04-11 16:51:55,448 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
graylog        | 2019-04-11 16:51:55,450 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
graylog        | 2019-04-11 16:51:55,452 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
graylog        | 2019-04-11 16:51:55,492 INFO : org.graylog2.periodical.IndexRetentionThread - Elasticsearch cluster not available, skipping index retention checks.
graylog        | 2019-04-11 16:51:55,495 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
graylog        | 2019-04-11 16:51:55,511 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to https://0.0.0.0:9200)
graylog        | 2019-04-11 16:51:55,514 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
graylog        | 2019-04-11 16:51:55,562 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
graylog        | 2019-04-11 16:51:55,586 INFO : org.graylog2.lookup.LookupTableService - Data Adapter abuse-ch-ransomware-ip/5c5df240c41479000a83dec4 [@317b3b0f] STARTING
graylog        | 2019-04-11 16:51:55,589 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
graylog        | 2019-04-11 16:51:55,592 ERROR: org.graylog2.plugin.lookup.LookupDataAdapter - Couldn't start data adapter <abuse-ch-ransomware-ip/5c5df240c41479000a83dec4/@317b3b0f>
graylog        | org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
graylog        |      at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:96) ~[?:?]
graylog        |      at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
graylog        |      at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
graylog        |      at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) [graylog.jar:?]
graylog        |      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
graylog        | 2019-04-11 16:51:55,654 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
graylog        | 2019-04-11 16:51:55,659 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
graylog        | 2019-04-11 16:51:55,660 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
graylog        | 2019-04-11 16:51:55,662 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
graylog        | 2019-04-11 16:51:55,657 INFO : org.graylog2.lookup.LookupTableService - Data Adapter whois/5c5df240c41479000a83dec3 [@5cbb5b64] STARTING
graylog        | 2019-04-11 16:51:55,658 INFO : org.graylog2.lookup.LookupTableService - Data Adapter abuse-ch-ransomware-ip/5c5df240c41479000a83dec4 [@317b3b0f] RUNNING
graylog        | 2019-04-11 16:51:55,658 WARN : org.graylog.plugins.threatintel.adapters.otx.OTXDataAdapter - OTX API key is missing. Make sure to add the key to allow higher request limits.
graylog        | 2019-04-11 16:51:55,657 INFO : org.graylog2.lookup.LookupTableService - Data Adapter otx-api-domain/5c5df240c41479000a83dec1 [@18644da9] STARTING
graylog        | 2019-04-11 16:51:55,655 INFO : org.graylog2.lookup.LookupTableService - Data Adapter abuse-ch-ransomware-domains/5c5df240c41479000a83dec2 [@4712eba] STARTING
graylog        | 2019-04-11 16:51:55,655 INFO : org.graylog2.lookup.LookupTableService - Data Adapter otx-api-ip/5c5df240c41479000a83debe [@37a5ef8d] STARTING
graylog        | 2019-04-11 16:51:55,655 INFO : org.graylog2.lookup.LookupTableService - Data Adapter spamhaus-drop/5c5df240c41479000a83debf [@4c6e6b38] STARTING
graylog        | 2019-04-11 16:51:55,693 ERROR: org.graylog2.plugin.lookup.LookupDataAdapter - Couldn't start data adapter <tor-exit-node/5c5df240c41479000a83dec0/@57b07e0c>
graylog        | org.graylog.plugins.threatintel.tools.AdapterDisabledException: TOR service is disabled, not starting TOR exit addresses adapter. To enable it please go to System / Configurations.
graylog        |      at org.graylog.plugins.threatintel.adapters.tor.TorExitNodeDataAdapter.doStart(TorExitNodeDataAdapter.java:89) ~[?:?]
graylog        |      at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
graylog        |      at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
graylog        |      at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) [graylog.jar:?]
graylog        |      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
graylog        | 2019-04-11 16:51:55,672 INFO : org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical - Cluster not connected yet, delaying index field type initialization until it is reachable.
graylog        | 2019-04-11 16:51:55,655 ERROR: org.graylog2.plugin.lookup.LookupDataAdapter - Couldn't start data adapter <spamhaus-drop/5c5df240c41479000a83debf/@4c6e6b38>
graylog        | org.graylog.plugins.threatintel.tools.AdapterDisabledException: Spamhaus service is disabled, not starting (E)DROP adapter. To enable it please go to System / Configurations.
graylog        |      at org.graylog.plugins.threatintel.adapters.spamhaus.SpamhausEDROPDataAdapter.doStart(SpamhausEDROPDataAdapter.java:85) ~[?:?]
graylog        |      at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
graylog        |      at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
graylog        |      at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) [graylog.jar:?]
graylog        |      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
graylog        | 2019-04-11 16:51:55,656 INFO : org.graylog2.lookup.LookupTableService - Data Adapter tor-exit-node/5c5df240c41479000a83dec0 [@57b07e0c] STARTING
graylog        | 2019-04-11 16:51:55,705 WARN : org.graylog.plugins.threatintel.adapters.otx.OTXDataAdapter - OTX API key is missing. Make sure to add the key to allow higher request limits.
graylog        | 2019-04-11 16:51:55,658 ERROR: org.graylog2.plugin.lookup.LookupDataAdapter - Couldn't start data adapter <abuse-ch-ransomware-domains/5c5df240c41479000a83dec2/@4712eba>
graylog        | org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
graylog        |      at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:96) ~[?:?]
graylog        |      at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
graylog        |      at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
graylog        |      at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) [graylog.jar:?]
graylog        |      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
graylog        | 2019-04-11 16:51:55,736 INFO : org.graylog2.lookup.LookupTableService - Data Adapter spamhaus-drop/5c5df240c41479000a83debf [@4c6e6b38] RUNNING
graylog        | 2019-04-11 16:51:55,748 INFO : org.graylog2.lookup.LookupTableService - Data Adapter tor-exit-node/5c5df240c41479000a83dec0 [@57b07e0c] RUNNING
graylog        | 2019-04-11 16:51:55,749 INFO : org.graylog2.lookup.LookupTableService - Data Adapter otx-api-ip/5c5df240c41479000a83debe [@37a5ef8d] RUNNING
graylog        | 2019-04-11 16:51:55,749 INFO : org.graylog2.lookup.LookupTableService - Data Adapter abuse-ch-ransomware-domains/5c5df240c41479000a83dec2 [@4712eba] RUNNING
graylog        | 2019-04-11 16:51:55,750 INFO : org.graylog2.lookup.LookupTableService - Data Adapter whois/5c5df240c41479000a83dec3 [@5cbb5b64] RUNNING
graylog        | 2019-04-11 16:51:55,764 INFO : org.graylog2.lookup.LookupTableService - Data Adapter otx-api-domain/5c5df240c41479000a83dec1 [@18644da9] RUNNING
graylog        | 2019-04-11 16:51:55,817 INFO : org.graylog2.lookup.LookupTableService - Cache otx-api-ip-cache/5c5df240c41479000a83deb8 [@58134577] STARTING
graylog        | 2019-04-11 16:51:55,841 INFO : org.graylog2.lookup.LookupTableService - Cache threat-intel-uncached-adapters/5c5df240c41479000a83debb [@6c3e31ca] STARTING
graylog        | 2019-04-11 16:51:55,844 INFO : org.graylog2.lookup.LookupTableService - Cache whois-cache/5c5df240c41479000a83deba [@2209771c] STARTING
graylog        | 2019-04-11 16:51:55,843 INFO : org.graylog2.lookup.LookupTableService - Cache otx-api-ip-cache/5c5df240c41479000a83deb8 [@58134577] RUNNING
graylog        | 2019-04-11 16:51:55,849 INFO : org.graylog2.lookup.LookupTableService - Cache otx-api-domain-cache/5c5df240c41479000a83deb9 [@5c488375] STARTING
graylog        | 2019-04-11 16:51:55,849 INFO : org.graylog2.lookup.LookupTableService - Cache spamhaus-e-drop-cache/5c5df240c41479000a83debc [@63949ddf] STARTING
graylog        | 2019-04-11 16:51:55,885 INFO : org.graylog2.lookup.LookupTableService - Cache whois-cache/5c5df240c41479000a83deba [@2209771c] RUNNING
graylog        | 2019-04-11 16:51:55,888 INFO : org.graylog2.lookup.LookupTableService - Cache spamhaus-e-drop-cache/5c5df240c41479000a83debc [@63949ddf] RUNNING
graylog        | 2019-04-11 16:51:55,891 INFO : org.graylog2.lookup.LookupTableService - Cache threat-intel-uncached-adapters/5c5df240c41479000a83debb [@6c3e31ca] RUNNING
graylog        | 2019-04-11 16:51:55,891 INFO : org.graylog2.lookup.LookupTableService - Cache otx-api-domain-cache/5c5df240c41479000a83deb9 [@5c488375] RUNNING
graylog        | 2019-04-11 16:51:55,909 INFO : org.graylog2.lookup.LookupTableService - Starting lookup table whois/5c5df240c41479000a83dec6 [@64b7420d] using cache whois-cache/5c5df240c41479000a83deba [@2209771c], data adapter whois/5c5df240c41479000a83dec3 [@5cbb5b64]
graylog        | 2019-04-11 16:51:55,912 INFO : org.graylog2.lookup.LookupTableService - Starting lookup table otx-api-ip/5c5df240c41479000a83dec7 [@31bfb62f] using cache otx-api-ip-cache/5c5df240c41479000a83deb8 [@58134577], data adapter otx-api-ip/5c5df240c41479000a83debe [@37a5ef8d]
graylog        | 2019-04-11 16:51:55,919 INFO : org.graylog2.lookup.LookupTableService - Starting lookup table spamhaus-drop/5c5df240c41479000a83dec8 [@1f777e44] using cache spamhaus-e-drop-cache/5c5df240c41479000a83debc [@63949ddf], data adapter spamhaus-drop/5c5df240c41479000a83debf [@4c6e6b38]
graylog        | 2019-04-11 16:51:55,920 INFO : org.graylog2.lookup.LookupTableService - Starting lookup table abuse-ch-ransomware-domains/5c5df240c41479000a83dec9 [@16499ef8] using cache threat-intel-uncached-adapters/5c5df240c41479000a83debb [@6c3e31ca], data adapter abuse-ch-ransomware-domains/5c5df240c41479000a83dec2 [@4712eba]
graylog        | 2019-04-11 16:51:55,920 INFO : org.graylog2.lookup.LookupTableService - Starting lookup table otx-api-domain/5c5df240c41479000a83deca [@48300116] using cache otx-api-domain-cache/5c5df240c41479000a83deb9 [@5c488375], data adapter otx-api-domain/5c5df240c41479000a83dec1 [@18644da9]
graylog        | 2019-04-11 16:51:55,921 INFO : org.graylog2.lookup.LookupTableService - Starting lookup table tor-exit-node-list/5c5df240c41479000a83decb [@472c8f1] using cache threat-intel-uncached-adapters/5c5df240c41479000a83debb [@6c3e31ca], data adapter tor-exit-node/5c5df240c41479000a83dec0 [@57b07e0c]
graylog        | 2019-04-11 16:51:55,921 INFO : org.graylog2.lookup.LookupTableService - Starting lookup table abuse-ch-ransomware-ip/5c5df240c41479000a83decc [@20531365] using cache threat-intel-uncached-adapters/5c5df240c41479000a83debb [@6c3e31ca], data adapter abuse-ch-ransomware-ip/5c5df240c41479000a83dec4 [@317b3b0f]
graylog        | 2019-04-11 16:51:55,957 INFO : org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges - Cluster not connected yet, delaying migration until it is reachable.
graylog        | 2019-04-11 16:51:56,346 INFO : org.graylog2.shared.initializers.JerseyService - Enabling CORS for HTTP endpoint
graylog        | 2019-04-11 16:52:10,339 INFO : org.graylog2.periodical.IndexRangesCleanupPeriodical - Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy
graylog        | 2019-04-11 16:52:16,151 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [127.0.0.1:9000]
graylog        | 2019-04-11 16:52:16,152 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
graylog        | 2019-04-11 16:52:16,152 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <127.0.0.1:9000>
graylog        | 2019-04-11 16:52:16,152 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
graylog        | 2019-04-11 16:52:16,154 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {OutputSetupService [RUNNING]=71, BufferSynchronizerService [RUNNING]=72, GracefulShutdownService [RUNNING]=78, KafkaJournal [RUNNING]=79, JournalReader [RUNNING]=177, EtagService [RUNNING]=190, InputSetupService [RUNNING]=223, ConfigurationEtagService [RUNNING]=233, StreamCacheService [RUNNING]=294, PeriodicalsService [RUNNING]=654, LookupTableService [RUNNING]=851, JerseyService [RUNNING]=21086}
graylog        | 2019-04-11 16:52:16,163 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
graylog        | 2019-04-11 16:52:16,163 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
graylog        | 2019-04-11 16:52:25,182 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to https://0.0.0.0:9200)
graylog        | 2019-04-11 16:52:25,182 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
graylog        | 2019-04-11 16:52:55,183 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to https://0.0.0.0:9200)
graylog        | 2019-04-11 16:52:55,183 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
graylog        | 2019-04-11 16:52:55,701 WARN : org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical - Interrupted or timed out waiting for Elasticsearch cluster, checking again.
graylog        | 2019-04-11 16:52:55,972 WARN : org.graylog2.migrations.V20161130141500_DefaultStreamRecalcIndexRanges - Interrupted or timed out waiting for Elasticsearch cluster, checking again.
graylog        | 2019-04-11 16:53:25,184 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to https://0.0.0.0:9200)
graylog        | 2019-04-11 16:53:25,184 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
graylog        | 2019-04-11 16:53:55,183 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to https://0.0.0.0:9200)
graylog        | 2019-04-11 16:53:55,184 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.
graylog        | 2019-04-11 16:53:55,704 WARN : org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical - Interrupted or timed out waiting for Elasticsearch cluster, checking again.
0 Likes

(Michael Ehrenreich) #3

Hi,

maybe I missed it. Where did you configure Elasticsearch to listen to https?

And this seems wrong (graylog.conf):

These are the hosts you’re trying to connect to. You don’t start them here.

0 Likes

(G) #4

Hello, thanks for the response!

Maybe that’s my problem. I haven’t changed much of the config for elasticsearch besides that directive. Look for elasticsearch.yml and the Elasticsearch section in my docker-compose file in my original post.

As far as

elasticsearch_hosts = https://0.0.0.0:9200

I gave that a shot since it wasn’t working and the container is listening on that address. I set it back to

 https://127.0.0.1:9200

I also tried turning it off and got the same result just to the default non https address.

ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)	

That needs to be set for https to work correct?

0 Likes

(Michael Ehrenreich) #5

Please try connecting to the host name:

http://elasticsearch:9200

That should be the known name within the network that Docker-compose creates.

0 Likes

(G) #6

Thank you so much! That fixed that error! Those elasticsearch errors are gone.

I’m still getting a 502 Bad Gateway on my proxy when I try to access the graylog site. Can you check and see if my nginx.conf looks ok and if there is anything correlated that could be wrong in my graylog.conf? After that I should be good to go. I don’t see any other errors. I really appreciate the help.

nginx | 2019/04/12 21:18:30 [error] 6#6: *2 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.201.68.100, server: test-graylog.mydomain.com, request: "GET / HTTP/1.1", upstream: "http://10.201.68.27:9000/", host: "test-graylog.mydomain.com"
nginx | 2019/04/12 21:18:30 [error] 6#6: *2 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.201.68.100, server: test-graylog.mydomain.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.201.68.27:9000/favicon.ico", host: "test-graylog.mydomain.com", referrer: "http://test-graylog.mydomain.com/"
0 Likes

(Michael Ehrenreich) #7

Can you show me your actual nginx.conf?

0 Likes

(G) #8

I posted it in my original post. No worries, I’ve posted it again below.

#nginx.conf

events { }
http {

  upstream backend {
    server test-graylog.mydomain.com:9000;
    keepalive 32;
  }

  server {
    listen              80;
    listen              443 ssl;
    server_name         test-graylog.mydomain.com;
    ssl_certificate     mychain.pem;
    ssl_certificate_key mykey.pem;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

  location / {
    proxy_pass http://backend;
     }
  }
 }
0 Likes

(Jan Doberstein) #9
  upstream backend {
    server test-graylog.mydomain.com:9000;
    keepalive 32;
  }

How should test-graylog.mydomain.com:9000 work within the docker env? You would need graylog but nothing else.

      - GRAYLOG_WEB_ENDPOINT_URI=https://10.201.68.27:9000/api

In addition you should check that you use valid configuration parameters for the Graylog Version you run in Docker …

0 Likes

(G) #10

I see what your saying. I think part of my problem was configuring the containers to properly work with the internal docker network. I made some changes as you suggested:

I changed my nginx.conf to:

 upstream backend {
    server graylog;
    keepalive 32;
  }

I checked the graylog 3.0 params and I ‘was’ going to change the graylog_web_endpoint to:

- GRAYLOG_WEB_ENDPOINT_URI=https://graylog:9000/api

but found since I’m declaring that in the graylog.conf file I don’t really need to pass that. So I just commented it out for now. It did not change my result.

I did find something I overlooked in the logs. It struck me a bit odd as previously the error was spamming the elasticsearch container logs. This time the elasticsearch container logs were clean; this came from the graylog container logs and occurred only once. I didn’t see a successful connection afterwards so I’m not sure if it connected later? The containers seem fine and show as being up in the logs.

2019-04-15 21:03:57,348 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to http://elasticsearch:9200) 

I’m still stuck with the same 502 bad gateway error when I try to access the graylog webpage or do a local curl. Unfortunately, nothing I changed made any difference.

 nginx | 2019/04/15 22:45:49 [error] 6#6: *80 connect() failed (111: Connection refused) while connecting to upstream, client: *myip*, server: test-graylog.twec.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://GraylogServerIP:80/favicon.ico", host: "test-graylog.twec.com", referrer: "https://test-graylog.twec.com/"

I greatly appreciate the help.

0 Likes

(Jan Doberstein) #11

I checked the graylog 3.0 params and I ‘was’ going to change the graylog_web_endpoint to:

you did not get my point completely:

Please re-read: http://docs.graylog.org/en/3.0/pages/upgrade/graylog-3.0.html#simplified-http-interface-configuration

0 Likes

(G) #12

Thanks, I see now. These parameters are completely different than the previous version. /api is now hardcoded so I removed that where I found it. I then went down the list.

I changed:

rest_listen_uri = https://graylog:9000/api 

to

http_bind_address = https://graylog

and

rest_enable_tls = true

to

http_enable_tls = true

web and rest references to:

http_tls_cert_file = /etc/nginx/mychain.pem
http_tls_key_file = /etc/nginx/mykey.pem

commented these since https://127.0.0.1:9000 is default

web_listen_uri = https://127.0.0.1:9000/
http_bind_address = https://127.0.0.1:9000/

and

web_enable_tls = true

to

http_enable_tls = true

after those changes, I did my usual docker-compose down and docker-compose up -d and I still got the same one line error.

graylog | 2019-04-17 01:15:28,145 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn't read cluster health for indices [graylog_*] (Could not connect to http://elasticsearch:9200)

However, on the bright side, it seems my graylog container shows ‘healthy’ now.

CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                   PORTS                                                                                                                    NAMES
53ccb7c6bf45        graylog/graylog:3.0.1                                 "/docker-entrypoint.…"   2 minutes ago       Up 2 minutes (healthy)   0.0.0.0:514->514/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:514->514/udp, 0.0.0.0:12201->12201/tcp, 0.0.0.0:12201->12201/udp   graylog
495a342c8d57        mongo:3                                               "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes             27017/tcp                                                                                                                mongodb
2e6b6d7b179e        docker.elastic.co/elasticsearch/elasticsearch:6.6.0   "/usr/local/bin/dock…"   2 minutes ago       Up 2 minutes             0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp                                                                           elasticsearch
fb1cb67bc525        nginx:latest                                          "nginx -g 'daemon of…"   2 minutes ago       Up 2 minutes             0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                                                 nginx

So, I ‘think’ that means its working? However, I still get a 502 and can’t access the website! I did notice, a few things in my nginx config. I found the directive ‘server graylog’ like below, made it resolve to my production server graylog.mydomain.com’s IP.

  upstream backend {
    server graylog;
    keepalive 32;
  }

So, I changed it back to:

upstream backend {
    server test-graylog.mydomain.com:9000;
    keepalive 32;
  }

and added https for the proxy_pass:

   location / {
      proxy_pass https://backend;
   }

and then it tries to connect for 30 seconds but times out with an SSL handshake error and another 502.

2019/04/17 02:48:17 [error] 6#6: *7 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 10.x.x.100, server: test-graylog.twec.com, request: "GET /favicon.ico HTTP/1.1", upstream: "https://10.x.x.27:9000/favicon.ico", host: "test-graylog.twec.com", referrer: "https://test-graylog.twec.com/"
10.x.x.100 - - [17/Apr/2019:02:48:17 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "https://test-graylog.twec.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"

I feel like I’m going in the right direction here, although I’m not sure. Is there something else I need to do to get the handshake to work? Do I have to pass an upstream header in my nginx.conf?

Thanks again guys.

0 Likes

(Ben van Staveren) #13

Either way, that is the wrong IP address. If you’re running your container normally, you’ll never bind to the actual container IP, and if it runs in host mode, you’re binding to the localhost.

Try using 0.0.0.0:9000 here, and then restore your nginx to server graylog; in the upstream block.

0 Likes

(Jan Doberstein) #14

I want to add in addition:

http_bind_address = 127.0.0.1:9000

        The network interface used by the Graylog HTTP interface.
        This network interface must be accessible by all Graylog nodes in the cluster and by all clients using the Graylog web interface.
        If the port is omitted, Graylog will use port 9000 by default.

no protocol - only the IP and port …

You might want to re-read the docs: http://docs.graylog.org/en/3.0/pages/configuration/server.conf.html#web-rest-api

0 Likes