Server currently unavailable [Ubuntu 16.04]


(Patrick Hugh Dayton) #1

Graylog will start upon an initial installation on a fresh EC2 instance, but will eventually stop working for unknown reasons. I also get a large amount of 500 error codes whenever navigating the web UI while the Graylog service is functioning.

Here are my log and configuration files:
/var/log/graylog-server/server.log

2018-01-23T16:10:48.127Z INFO  [CmdLineTool] Loaded plugin: AWS plugins 2.4.1 [org.graylog.aws.plugin.AWSPlugin]
2018-01-23T16:10:48.129Z INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.4.1 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-01-23T16:10:48.130Z INFO  [CmdLineTool] Loaded plugin: CEF Input 2.4.1 [org.graylog.plugins.cef.CEFInputPlugin]
2018-01-23T16:10:48.131Z INFO  [CmdLineTool] Loaded plugin: Collector 2.4.1 [org.graylog.plugins.collector.CollectorPlugin]
2018-01-23T16:10:48.131Z INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.4.1 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2018-01-23T16:10:48.132Z INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.4.1 [org.graylog.plugins.map.MapWidgetPlugin]
2018-01-23T16:10:48.133Z INFO  [CmdLineTool] Loaded plugin: NetFlow Plugin 2.4.1 [org.graylog.plugins.netflow.NetFlowPlugin]
2018-01-23T16:10:48.139Z INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.4.1 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-01-23T16:10:48.139Z INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.4.1 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2018-01-23T16:10:48.509Z INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2018-01-23T16:10:48.781Z INFO  [Version] HV000001: Hibernate Validator 5.1.3.Final
2018-01-23T16:10:51.698Z INFO  [InputBufferImpl] Message journal is enabled.
2018-01-23T16:10:51.747Z INFO  [NodeId] Node ID: 02bdc1df-4cf8-4e32-a9cb-ee6a49086655
2018-01-23T16:10:52.101Z INFO  [LogManager] Loading logs.
2018-01-23T16:10:52.112Z INFO  [LogManager] Logs loading complete.
2018-01-23T16:10:52.190Z INFO  [LogManager] Created log for partition [messagejournal,0] in /var/lib/graylog-server/journal with properties {file.delete.delay.ms -> 60000, compact -> false, max.message.bytes -> 104857600, min.insync.replicas -> 1, segment.jitter.ms -> 0, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> true, retention.bytes -> 5368709120, delete.retention.ms -> 86400000, flush.ms -> 60000, segment.bytes -> 104857600, segment.ms -> 3600000, retention.ms -> 43200000, flush.messages -> 1000000, segment.index.bytes -> 1048576}.
2018-01-23T16:10:52.190Z INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2018-01-23T16:10:52.204Z INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2018-01-23T16:10:52.226Z INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2018-01-23T16:10:52.332Z INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2018-01-23T16:10:52.403Z INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:1}] to localhost:27017
2018-01-23T16:10:52.407Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 6, 10]}, minWireVersion=0, maxWireVersion=2, maxDocumentSize=16777216, roundTripTimeNanos=413906}
2018-01-23T16:10:52.424Z INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:2}] to localhost:27017
2018-01-23T16:10:53.276Z INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2018-01-23T16:10:53.277Z INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2018-01-23T16:10:53.420Z INFO  [JestClientFactory] Using custom ObjectMapper instance
2018-01-23T16:10:53.420Z INFO  [JestClientFactory] Node Discovery enabled...
2018-01-23T16:10:53.441Z INFO  [JestClientFactory] Idle connection reaping disabled...
2018-01-23T16:10:53.892Z INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://172.31.3.188:9200]
2018-01-23T16:10:54.004Z INFO  [RoleServiceImpl] Admin role is missing or invalid, re-adding it as a built-in role.
2018-01-23T16:10:54.199Z INFO  [RoleServiceImpl] Reader role is missing or invalid, re-adding it as a built-in role.
2018-01-23T16:10:54.402Z INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-01-23T16:10:57.207Z INFO  [RulesEngineProvider] No static rules file loaded.
2018-01-23T16:10:57.229Z INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:3}] to localhost:27017
2018-01-23T16:10:57.405Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-01-23T16:10:57.430Z INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-01-23T16:10:57.465Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-01-23T16:10:57.487Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-01-23T16:10:57.510Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-01-23T16:10:57.540Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-01-23T16:10:58.059Z INFO  [ServerBootstrap] Graylog server 2.4.1+0f97411 starting up
2018-01-23T16:10:58.060Z INFO  [ServerBootstrap] JRE: Oracle Corporation 1.8.0_151 on Linux 4.4.0-1047-aws
2018-01-23T16:10:58.060Z INFO  [ServerBootstrap] Deployment: deb
2018-01-23T16:10:58.060Z INFO  [ServerBootstrap] OS: Ubuntu 16.04.3 LTS (xenial)
2018-01-23T16:10:58.060Z INFO  [ServerBootstrap] Arch: amd64
2018-01-23T16:10:58.068Z WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>

/var/log/elasticsearch/graylog.log:

[2018-01-23T16:09:53,853][INFO ][o.e.n.Node               ] [graylog-test] initializing ...
[2018-01-23T16:09:53,929][INFO ][o.e.e.NodeEnvironment    ] [graylog-test] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [5.4gb], net total_space [7.6gb], spins? [no], types [ext4]
[2018-01-23T16:09:53,929][INFO ][o.e.e.NodeEnvironment    ] [graylog-test] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-01-23T16:09:53,931][INFO ][o.e.n.Node               ] [graylog-test] node name [graylog-test], node ID [IFEoIUslR5mf5ULYJesdQw]
[2018-01-23T16:09:53,931][INFO ][o.e.n.Node               ] [graylog-test] version[5.6.2], pid[18453], build[57e20f3/2017-09-23T13:16:45.703Z], OS[Linux/4.4.0-1047-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-01-23T16:09:53,931][INFO ][o.e.n.Node               ] [graylog-test] JVM arguments [-Xms1975m, -Xmx1975m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [aggs-matrix-stats]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [ingest-common]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-expression]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-groovy]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-mustache]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-painless]
[2018-01-23T16:09:54,961][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [parent-join]
[2018-01-23T16:09:54,961][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [percolator]
[2018-01-23T16:09:54,961][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [reindex]
[2018-01-23T16:09:54,961][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [transport-netty3]
[2018-01-23T16:09:54,961][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [transport-netty4]
[2018-01-23T16:09:54,967][INFO ][o.e.p.PluginsService     ] [graylog-test] no plugins loaded
[2018-01-23T16:09:57,888][INFO ][o.e.d.DiscoveryModule    ] [graylog-test] using discovery type [zen]
[2018-01-23T16:09:58,469][INFO ][o.e.n.Node               ] [graylog-test] initialized
[2018-01-23T16:09:58,469][INFO ][o.e.n.Node               ] [graylog-test] starting ...
[2018-01-23T16:09:58,627][INFO ][o.e.t.TransportService   ] [graylog-test] publish_address {172.31.3.188:9300}, bound_addresses {[::]:9300}
[2018-01-23T16:09:58,638][INFO ][o.e.b.BootstrapChecks    ] [graylog-test] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-01-23T16:10:01,687][INFO ][o.e.c.s.ClusterService   ] [graylog-test] new_master {graylog-test}{IFEoIUslR5mf5ULYJesdQw}{VxkYZf7OSy23TRgIFij7QA}{172.31.3.188}{172.31.3.188:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-01-23T16:10:01,712][INFO ][o.e.h.n.Netty4HttpServerTransport] [graylog-test] publish_address {172.31.3.188:9200}, bound_addresses {[::]:9200}
[2018-01-23T16:10:01,712][INFO ][o.e.n.Node               ] [graylog-test] started
[2018-01-23T16:10:01,720][INFO ][o.e.g.GatewayService     ] [graylog-test] recovered [0] indices into cluster_state
[2018-01-23T16:14:29,103][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [graylog_0] creating index, cause [api], templates [graylog-internal], shards [1]/[0], mappings [message]
[2018-01-23T16:14:29,358][INFO ][o.e.c.r.a.AllocationService] [graylog-test] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2018-01-23T17:59:29,365][INFO ][o.e.n.Node               ] [graylog-test] stopping ...
[2018-01-23T17:59:29,388][INFO ][o.e.n.Node               ] [graylog-test] stopped
[2018-01-23T17:59:29,388][INFO ][o.e.n.Node               ] [graylog-test] closing ...
[2018-01-23T17:59:29,397][INFO ][o.e.n.Node               ] [graylog-test] closed
[2018-01-23T17:59:52,198][INFO ][o.e.n.Node               ] [graylog-test] initializing ...
[2018-01-23T17:59:52,379][INFO ][o.e.e.NodeEnvironment    ] [graylog-test] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [4.9gb], net total_space [7.6gb], spins? [no], types [ext4]
[2018-01-23T17:59:52,379][INFO ][o.e.e.NodeEnvironment    ] [graylog-test] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-01-23T17:59:52,403][INFO ][o.e.n.Node               ] [graylog-test] node name [graylog-test], node ID [IFEoIUslR5mf5ULYJesdQw]
[2018-01-23T17:59:52,403][INFO ][o.e.n.Node               ] [graylog-test] version[5.6.2], pid[27845], build[57e20f3/2017-09-23T13:16:45.703Z], OS[Linux/4.4.0-1047-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-01-23T17:59:52,403][INFO ][o.e.n.Node               ] [graylog-test] JVM arguments [-Xms1975m, -Xmx1975m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [aggs-matrix-stats]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [ingest-common]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-expression]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-groovy]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-mustache]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-painless]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [parent-join]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [percolator]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [reindex]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [transport-netty3]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [transport-netty4]
[2018-01-23T17:59:55,795][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded plugin [discovery-ec2]
[2018-01-23T17:59:55,795][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded plugin [x-pack]
[2018-01-23T17:59:59,979][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin org.elasticsearch.xpack.XPackPlugin
[2018-01-23T18:00:04,043][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/27989] [Main.cc@128] controller (64 bit): Version 5.6.2 (Build 228329870d1c63) Copyright (c) 2017 Elasticsearch BV
[2018-01-23T18:00:04,097][INFO ][o.e.d.DiscoveryModule    ] [graylog-test] using discovery type [zen]
[2018-01-23T18:00:05,626][INFO ][o.e.n.Node               ] [graylog-test] initialized
[2018-01-23T18:00:05,631][INFO ][o.e.n.Node               ] [graylog-test] starting ...
[2018-01-23T18:00:06,035][INFO ][o.e.t.TransportService   ] [graylog-test] publish_address {172.31.3.188:9300}, bound_addresses {172.31.3.188:9300}
[2018-01-23T18:00:06,048][INFO ][o.e.b.BootstrapChecks    ] [graylog-test] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-01-23T18:00:09,136][INFO ][o.e.c.s.ClusterService   ] [graylog-test] new_master {graylog-test}{IFEoIUslR5mf5ULYJesdQw}{AEm6nIOjQ4uRGglvyW0lRQ}{172.31.3.188}{172.31.3.188:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-01-23T18:00:09,208][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [graylog-test] publish_address {172.31.3.188:9200}, bound_addresses {172.31.3.188:9200}
[2018-01-23T18:00:09,208][INFO ][o.e.n.Node               ] [graylog-test] started
[2018-01-23T18:00:09,799][INFO ][o.e.g.GatewayService     ] [graylog-test] recovered [1] indices into cluster_state
[2018-01-23T18:00:09,871][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [graylog-test] successfully created .ml-meta index template
[2018-01-23T18:00:09,973][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [graylog-test] successfully created .ml-state index template
[2018-01-23T18:00:10,043][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [graylog-test] successfully created .ml-notifications index template
[2018-01-23T18:00:10,289][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [graylog-test] successfully created .ml-anomalies- index template
[2018-01-23T18:00:11,130][INFO ][o.e.l.LicenseService     ] [graylog-test] license [ccae04a5-899a-4e5c-b49b-4b7366f83ca3] mode [trial] - valid
[2018-01-23T18:00:11,137][INFO ][o.e.c.r.a.AllocationService] [graylog-test] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2018-01-23T18:00:16,217][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [.monitoring-es-6-2018.01.23] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc]
[2018-01-23T18:00:16,298][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [.watches] creating index, cause [auto(bulk api)], templates [watches], shards [1]/[1], mappings [watch]
[2018-01-23T18:00:16,494][INFO ][o.e.c.m.MetaDataMappingService] [graylog-test] [.watches/C6bAXWo6TOabt-rdzRKizw] update_mapping [watch]
[2018-01-23T18:01:17,017][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [.triggered_watches] creating index, cause [auto(bulk api)], templates [triggered_watches], shards [1]/[1], mappings [triggered_watch]
[2018-01-23T18:01:17,237][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [.watcher-history-6-2018.01.23] creating index, cause [auto(bulk api)], templates [.watch-history-6], shards [1]/[1], mappings [doc]
[2018-01-23T18:01:17,350][INFO ][o.e.c.m.MetaDataMappingService] [graylog-test] [.watcher-history-6-2018.01.23/QhZscqciR5mK6eLqhVpXrA] update_mapping [doc]
[2018-01-23T18:01:17,396][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [.monitoring-alerts-6] creating index, cause [auto(bulk api)], templates [.monitoring-alerts], shards [1]/[1], mappings [doc]
[2018-01-23T18:01:17,476][INFO ][o.e.c.m.MetaDataMappingService] [graylog-test] [.watcher-history-6-2018.01.23/QhZscqciR5mK6eLqhVpXrA] update_mapping [doc]
[2018-01-23T18:13:56,869][INFO ][o.e.n.Node               ] [graylog-test] stopping ...
[2018-01-23T18:13:56,899][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/27989] [Main.cc@168] Ml controller exiting
[2018-01-23T18:13:56,901][INFO ][o.e.x.w.w.WatchStore     ] [graylog-test] stopped watch store
[2018-01-23T18:13:56,901][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
[2018-01-23T18:13:57,411][INFO ][o.e.n.Node               ] [graylog-test] stopped
[2018-01-23T18:13:57,412][INFO ][o.e.n.Node               ] [graylog-test] closing ...
[2018-01-23T18:13:57,423][INFO ][o.e.n.Node               ] [graylog-test] closed
[2018-01-23T18:13:59,607][INFO ][o.e.n.Node               ] [graylog-test] initializing ...
[2018-01-23T18:13:59,793][INFO ][o.e.e.NodeEnvironment    ] [graylog-test] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [4.9gb], net total_space [7.6gb], spins? [no], types [ext4]
[2018-01-23T18:13:59,794][INFO ][o.e.e.NodeEnvironment    ] [graylog-test] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-01-23T18:13:59,830][INFO ][o.e.n.Node               ] [graylog-test] node name [graylog-test], node ID [IFEoIUslR5mf5ULYJesdQw]
[2018-01-23T18:13:59,832][INFO ][o.e.n.Node               ] [graylog-test] version[5.6.2], pid[28394], build[57e20f3/2017-09-23T13:16:45.703Z], OS[Linux/4.4.0-1047-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-01-23T18:13:59,833][INFO ][o.e.n.Node               ] [graylog-test] JVM arguments [-Xms1975m, -Xmx1975m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-01-23T18:14:02,834][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [aggs-matrix-stats]
[2018-01-23T18:14:02,834][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [ingest-common]
[2018-01-23T18:14:02,834][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-expression]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-groovy]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-mustache]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [lang-painless]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [parent-join]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [percolator]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [reindex]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [transport-netty3]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded module [transport-netty4]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded plugin [discovery-ec2]
[2018-01-23T18:14:02,841][INFO ][o.e.p.PluginsService     ] [graylog-test] loaded plugin [x-pack]
[2018-01-23T18:14:06,766][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin org.elasticsearch.xpack.XPackPlugin
[2018-01-23T18:14:08,533][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/28524] [Main.cc@128] controller (64 bit): Version 5.6.2 (Build 228329870d1c63) Copyright (c) 2017 Elasticsearch BV
[2018-01-23T18:14:08,617][INFO ][o.e.d.DiscoveryModule    ] [graylog-test] using discovery type [zen]
[2018-01-23T18:14:10,200][INFO ][o.e.n.Node               ] [graylog-test] initialized
[2018-01-23T18:14:10,200][INFO ][o.e.n.Node               ] [graylog-test] starting ...
[2018-01-23T18:14:10,561][INFO ][o.e.t.TransportService   ] [graylog-test] publish_address {172.31.3.188:9300}, bound_addresses {172.31.3.188:9300}
[2018-01-23T18:14:10,574][INFO ][o.e.b.BootstrapChecks    ] [graylog-test] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-01-23T18:14:13,674][INFO ][o.e.c.s.ClusterService   ] [graylog-test] new_master {graylog-test}{IFEoIUslR5mf5ULYJesdQw}{o8seVW0XSxSTxmYbKGt4eA}{172.31.3.188}{172.31.3.188:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-01-23T18:14:13,767][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [graylog-test] publish_address {172.31.3.188:9200}, bound_addresses {172.31.3.188:9200}
[2018-01-23T18:14:13,767][INFO ][o.e.n.Node               ] [graylog-test] started
[2018-01-23T18:14:14,684][INFO ][o.e.l.LicenseService     ] [graylog-test] license [ccae04a5-899a-4e5c-b49b-4b7366f83ca3] mode [trial] - valid
[2018-01-23T18:14:14,714][INFO ][o.e.g.GatewayService     ] [graylog-test] recovered [6] indices into cluster_state
[2018-01-23T18:14:15,514][INFO ][o.e.c.r.a.AllocationService] [graylog-test] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[graylog_0][0]] ...]).
[2018-01-23T19:26:45,799][INFO ][o.e.x.s.a.f.FileUserPasswdStore] [graylog-test] users file [/etc/elasticsearch/x-pack/users] changed. updating users... )
[2018-01-23T19:26:45,800][INFO ][o.e.x.s.a.f.FileUserRolesStore] [graylog-test] users roles file [/etc/elasticsearch/x-pack/users_roles] changed. updating users roles...
[2018-01-23T19:26:45,801][INFO ][o.e.x.s.a.s.FileRolesStore] [graylog-test] updated roles (roles file [/etc/elasticsearch/x-pack/roles.yml] changed)

/etc/graylog/server/server.conf:

############################
# GRAYLOG CONFIGURATION FILE
############################

is_master = true

node_id_file = /etc/graylog/server/node-id
password_secret = secret
root_username = custom
root_password_sha2 = secret
root_email = "my@email.com"
root_timezone = America/Detroit

### Plugins Directory
plugin_dir = /usr/share/graylog-server/plugin

### REST Interface
rest_listen_uri = http://0.0.0.0:12900/api/
rest_transport_uri = http://34.238.121.38:12900
rest_enable_cors = true
rest_enable_gzip = true
rest_enable_tls = false

### Web Interface
web_enable = true
web_listen_uri = http://0.0.0.0:9000
web_endpoint_uri = http://127.0.0.1:9000/api

web_enable_cors = true
web_enable_gzip = true
web_enable_tls = false

### Elasticearch
elasticsearch_discovery_enabled = true
elasticsearch_hosts = http://127.0.0.1:9200
elasticsearch_connect_timeout = 10s
elasticsearch_socket_timeout = 60s
# elasticsearch_idle_timeout = -1s
elasticsearch_max_total_connections = 20
elasticsearch_max_total_connections_per_route = 2
elasticsearch_max_retries = 2

rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000

elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 1
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog

### Search Options
allow_leading_wildcard_searches = false
allow_highlighting = false

elasticsearch_analyzer = standard

elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1
elasticsearch_transport_tcp_port = 9300

output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3

processor_wait_strategy = blocking

ring_size = 65536

inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking

message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3

mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5

content_packs_dir = /usr/share/graylog-server/contentpacks

content_packs_auto_load = grok-patterns.json

proxied_requests_thread_pool_size = 32

/etc/elasticsearch/elasticsearch.yml:

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# THIS FILE IS MANAGED BY CHEF, DO NOT EDIT MANUALLY, YOUR CHANGES WILL BE OVERWRITTEN!
#
# Please see the documentation for further information on configuration options:
# <https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html>
#
---
cluster.name: graylog
node.name: graylog-test
path.conf: "/etc/elasticsearch"
path.data: "/var/lib/elasticsearch"
path.logs: "/var/log/elasticsearch"
network.host: 172.31.3.188
http.port: 9200
transport.tcp.port: 9300

I am installing everything with a custom wrapper cookbook on a t2.medium. I believe 4 Gigs of RAM is enough. If not, that’s an easy fix, but I still dont know how to resolve the 500 level errors whenever navigating the web UI.


(Jochen) #2

These don’t match up.

These also don’t match.


(Patrick Hugh Dayton) #3

Could you please elaborate? I don’t know what you mean by “these”. I know the rest_transport_uri needs to be the public IP of my instance, and I appreciate you pointing out that the addresses for elasticsearch mismatching. That make sense. What would you set these addresses to?


(Jochen) #4

Please read http://docs.graylog.org/en/2.4/pages/configuration/web_interface.html and configure the rest_listen_uri, rest_transport_uri, web_listen_uri, and web_endpoint_uri settings accordingly.

Whatever is correct for your individual environment.

If Elasticsearch and Graylog are running on the same machine and you don’t intend to run more than a single-node Elasticsearch cluster, using 127.0.0.1 should be fine.


(system) #5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.