-
Graylog UI works ~3mins after start/restart the container then time out with error message:
“Server currently unavailable
We are experiencing problems connecting to the Graylog server running on https://<<HOST_DNS_NAME>>:8917/api/. Please verify that the server is healthy and working correctly.
You will be automatically redirected to the previous page once we can connect to the server.
Do you need a hand? We can help you.” -
Describe your environment:
-
OS Information:
docker host: CentOS Linux release 7.9.2009 (Core) -
Docker image contains a bundle of Graylog, Elasticsearch and Mongo:
*Container envs:
“Env”: [
“GRAYLOG_NODE_ID=01”,
“ES_HEAP_SIZE=4g”,
“GRAYLOG_WEB_PORT=8917”,
“GRAYLOG_SERVER_URL=<<HOST_DNS_NAME>>”,
“GRAYLOG_HEAP_SIZE=-Xms3g -Xmx4g”,
“PATH=/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin”,
“LANG=C.UTF-8”,
“JAVA_HOME=/usr/local/openjdk-8”,
“JAVA_VERSION=8u272”,
“LS_HEAP_SIZE=512m”,
“LOGSTASH_VERSION=7.9.3”,
“JMX_PROM_JAVAAGENT_VERSION=0.12.0”,
“GRAYLOG_VERSION=4.2.5”,
“ELASTICSEARCH_VERSION=6.8.23”,
“MONGO_VERSION=4.0.5”
], -
Service logs, configurations, and environment variables:
nginx.conf
server
{
listen 80 default_server;
server_name localhost;
location /api/
{
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:12900/;
}
location /
{
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL http://127.0.0.1:80/api;
proxy_pass http://127.0.0.1:9000;
}
}
graylog.conf
is_master = true
root_username = admin
password_secret = <<SECRET>>
root_password_sha2 = <<SECRET>>
http_bind_address = 0.0.0.0:9000
http_publish_uri = https://<<HOST_DNS_NAME>>:8917/
http_external_uri = https://<<HOST_DNS_NAME>>:8917/
node_id_file = /opt/graylog/node-id
plugin_dir = plugin
http_enable_cors = true
rotation_strategy = count
retention_strategy = delete
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
processor_wait_strategy = blocking
ring_size = 65536
message_journal_enabled = true
message_journal_dir = data/journal
lb_recognition_period_seconds = 3
content_packs_auto_load = grok-patterns.json
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 8
elasticsearch_cluster_name = graylog2
elasticsearch_shards = 2
elasticsearch_replicas = 0
elasticsearch_discovery_zen_ping_unicast_hosts = http://localhost:9300
elasticsearch_hosts = http://localhost:9200
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
http_enable_tls = true
http_tls_key_password = <<SECRET>>
http_tls_key_file = /opt/graylog/config/pkcs8-encrypted.pem
http_tls_cert_file = /opt/graylog/config/cert.pem
elastic search has a default config
logs from container CRIT WARN and ERROR only:
2023-06-15 13:34:15,078 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2023-06-15 13:34:16,813 WARN received SIGTERM indicating exit request
2023-06-15 13:34:54,604 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2023-06-15 13:34:57,292 ERROR: org.graylog.plugins.sidecar.migrations.V20180212165000_AddDefaultCollectors - Couldn't find collector 'filebeat on linux' fixing it.
2023-06-15 13:34:57,331 ERROR: org.graylog.plugins.sidecar.migrations.V20180212165000_AddDefaultCollectors - Couldn't find collector 'winlogbeat on windows' fixing it.
2023-06-15 13:34:57,342 ERROR: org.graylog.plugins.sidecar.migrations.V20180212165000_AddDefaultCollectors - Couldn't find collector 'nxlog on linux' fixing it.
2023-06-15 13:34:57,345 ERROR: org.graylog.plugins.sidecar.migrations.V20180212165000_AddDefaultCollectors - Couldn't find collector 'nxlog on windows' fixing it.
2023-06-15 13:34:57,348 ERROR: org.graylog.plugins.sidecar.migrations.V20180212165000_AddDefaultCollectors - Couldn't find collector 'filebeat on windows' fixing it.
2023-06-15 13:34:57,373 ERROR: org.graylog2.migrations.MigrationHelpers - Invalid user 'graylog-sidecar', fixing it.
2023-06-15 13:35:02,225 WARN : org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical - Active write index for index set "Default index set" (648b1380ee92dc4a0a7f9059) doesn't exist yet
elasticsearch.log
[2023-06-15T13:34:24,080][INFO ][o.e.e.NodeEnvironment ] [gZYwh2E] using [1] data paths, mounts [[/opt/elasticsearch/data (/dev/mapper/system-tmp)]], net usable_space [681.1mb], net total_space [1014mb], types [xfs]
[2023-06-15T13:34:24,083][INFO ][o.e.e.NodeEnvironment ] [gZYwh2E] heap size [989.8mb], compressed ordinary object pointers [true]
[2023-06-15T13:34:24,085][INFO ][o.e.n.Node ] [gZYwh2E] node name derived from node ID [gZYwh2EIRhKFRVdCVPmrig]; set [node.name] to override
[2023-06-15T13:34:24,085][INFO ][o.e.n.Node ] [gZYwh2E] version[6.8.23], pid[13], build[default/tar/4f67856/2022-01-06T21:30:50.087716Z], OS[Linux/3.10.0-1160.42.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_272/25.272-b10]
[2023-06-15T13:34:24,086][INFO ][o.e.n.Node ] [gZYwh2E] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.io.tmpdir=/tmp/elasticsearch-8553776622003653843, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/opt/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2023-06-15T13:34:26,113][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [aggs-matrix-stats]
[2023-06-15T13:34:26,113][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [analysis-common]
[2023-06-15T13:34:26,113][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [ingest-common]
[2023-06-15T13:34:26,114][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [ingest-geoip]
[2023-06-15T13:34:26,114][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [ingest-user-agent]
[2023-06-15T13:34:26,114][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [lang-expression]
[2023-06-15T13:34:26,114][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [lang-mustache]
[2023-06-15T13:34:26,114][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [lang-painless]
[2023-06-15T13:34:26,114][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [mapper-extras]
[2023-06-15T13:34:26,114][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [parent-join]
[2023-06-15T13:34:26,114][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [percolator]
[2023-06-15T13:34:26,114][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [rank-eval]
[2023-06-15T13:34:26,115][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [reindex]
[2023-06-15T13:34:26,115][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [repository-url]
[2023-06-15T13:34:26,115][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [transport-netty4]
[2023-06-15T13:34:26,115][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [tribe]
[2023-06-15T13:34:26,115][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-ccr]
[2023-06-15T13:34:26,115][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-core]
[2023-06-15T13:34:26,115][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-deprecation]
[2023-06-15T13:34:26,115][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-graph]
[2023-06-15T13:34:26,115][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-ilm]
[2023-06-15T13:34:26,115][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-logstash]
[2023-06-15T13:34:26,115][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-ml]
[2023-06-15T13:34:26,116][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-monitoring]
[2023-06-15T13:34:26,116][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-rollup]
[2023-06-15T13:34:26,116][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-security]
[2023-06-15T13:34:26,116][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-sql]
[2023-06-15T13:34:26,116][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-upgrade]
[2023-06-15T13:34:26,116][INFO ][o.e.p.PluginsService ] [gZYwh2E] loaded module [x-pack-watcher]
[2023-06-15T13:34:26,117][INFO ][o.e.p.PluginsService ] [gZYwh2E] no plugins loaded
[2023-06-15T13:34:30,918][INFO ][o.e.x.s.a.s.FileRolesStore] [gZYwh2E] parsed [0] roles from file [/opt/elasticsearch/config/roles.yml]
[2023-06-15T13:34:31,625][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [gZYwh2E] [controller/139] [Main.cc@114] controller (64 bit): Version 6.8.23 (Build 31256deab94add) Copyright (c) 2022 Elasticsearch BV
[2023-06-15T13:34:32,211][DEBUG][o.e.a.ActionModule ] [gZYwh2E] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2023-06-15T13:34:32,447][INFO ][o.e.d.DiscoveryModule ] [gZYwh2E] using discovery type [zen] and host providers [settings]
[2023-06-15T13:34:33,393][INFO ][o.e.n.Node ] [gZYwh2E] initialized
[2023-06-15T13:34:33,394][INFO ][o.e.n.Node ] [gZYwh2E] starting ...
[2023-06-15T13:34:33,573][INFO ][o.e.t.TransportService ] [gZYwh2E] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2023-06-15T13:34:33,595][WARN ][o.e.b.BootstrapChecks ] [gZYwh2E] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2023-06-15T13:34:36,669][INFO ][o.e.c.s.MasterService ] [gZYwh2E] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {gZYwh2E}{gZYwh2EIRhKFRVdCVPmrig}{M7cSSX2iREKcTcEPGeIPHQ}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=16106127360, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2023-06-15T13:34:36,675][INFO ][o.e.c.s.ClusterApplierService] [gZYwh2E] new_master {gZYwh2E}{gZYwh2EIRhKFRVdCVPmrig}{M7cSSX2iREKcTcEPGeIPHQ}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=16106127360, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {gZYwh2E}{gZYwh2EIRhKFRVdCVPmrig}{M7cSSX2iREKcTcEPGeIPHQ}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=16106127360, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2023-06-15T13:34:36,736][INFO ][o.e.h.n.Netty4HttpServerTransport] [gZYwh2E] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2023-06-15T13:34:36,737][INFO ][o.e.n.Node ] [gZYwh2E] started
[2023-06-15T13:34:36,751][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [gZYwh2E] Failed to clear cache for realms [[]]
[2023-06-15T13:34:36,794][INFO ][o.e.g.GatewayService ] [gZYwh2E] recovered [0] indices into cluster_state
[2023-06-15T13:34:36,971][INFO ][o.e.c.m.MetaDataIndexTemplateService] [gZYwh2E] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[2023-06-15T13:34:37,001][INFO ][o.e.c.m.MetaDataIndexTemplateService] [gZYwh2E] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2023-06-15T13:34:37,032][INFO ][o.e.c.m.MetaDataIndexTemplateService] [gZYwh2E] adding template [.watches] for index patterns [.watches*]
[2023-06-15T13:34:37,060][INFO ][o.e.c.m.MetaDataIndexTemplateService] [gZYwh2E] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[2023-06-15T13:34:37,109][INFO ][o.e.c.m.MetaDataIndexTemplateService] [gZYwh2E] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[2023-06-15T13:34:37,142][INFO ][o.e.c.m.MetaDataIndexTemplateService] [gZYwh2E] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[2023-06-15T13:34:37,165][INFO ][o.e.c.m.MetaDataIndexTemplateService] [gZYwh2E] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[2023-06-15T13:34:37,192][INFO ][o.e.c.m.MetaDataIndexTemplateService] [gZYwh2E] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[2023-06-15T13:34:37,307][INFO ][o.e.l.LicenseService ] [gZYwh2E] license [676fb1e4-97cd-46d0-83ec-cca17c8edbb4] mode [basic] - valid
[2023-06-15T13:35:06,260][INFO ][o.e.c.m.MetaDataIndexTemplateService] [gZYwh2E] adding template [graylog-internal] for index patterns [graylog_*]
[2023-06-15T13:35:06,316][INFO ][o.e.c.m.MetaDataCreateIndexService] [gZYwh2E] [graylog_0] creating index, cause [api], templates [graylog-internal], shards [2]/[0], mappings [message]
[2023-06-15T13:35:06,603][INFO ][o.e.c.r.a.AllocationService] [gZYwh2E] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[graylog_0][1]] ...]).
[2023-06-15T13:35:06,762][INFO ][o.e.c.m.MetaDataIndexTemplateService] [gZYwh2E] adding template [gl-events-template] for index patterns [gl-events_*]
[2023-06-15T13:35:06,799][INFO ][o.e.c.m.MetaDataCreateIndexService] [gZYwh2E] [gl-events_0] creating index, cause [api], templates [gl-events-template], shards [2]/[0], mappings [message]
[2023-06-15T13:35:06,904][INFO ][o.e.c.r.a.AllocationService] [gZYwh2E] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[gl-events_0][0]] ...]).
[2023-06-15T13:35:07,008][INFO ][o.e.c.m.MetaDataIndexTemplateService] [gZYwh2E] adding template [gl-system-events-template] for index patterns [gl-system-events_*]
[2023-06-15T13:35:07,038][INFO ][o.e.c.m.MetaDataCreateIndexService] [gZYwh2E] [gl-system-events_0] creating index, cause [api], templates [gl-system-events-template], shards [2]/[0], mappings [message]
[2023-06-15T13:35:07,133][INFO ][o.e.c.r.a.AllocationService] [gZYwh2E] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[gl-system-events_0][1]] ...]).
elasticsearch_deprecation.log
[2023-06-15T13:35:06,244][WARN ][o.e.d.r.a.a.i.RestPutIndexTemplateAction] [gZYwh2E] [types removal] The parameter include_type_name should be explicitly specified in put template requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions.
[2023-06-15T13:35:06,246][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] [gZYwh2E] Deprecated field [template] used, replaced by [index_patterns]
[2023-06-15T13:35:07,222][WARN ][o.e.d.r.a.a.i.RestGetMappingAction] [gZYwh2E] [types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions.
I heave spent 3 days on figuring it out. In the first 2-3 mins Graylog UI is working normally I could log in. Adding inputs to /api/system/inputs working fine. Then it stucks and I have a time out. The same is with “curl -k https://localhost:8917”. At the beginning I have response and after some time no response at all.