Graylog will start upon an initial installation on a fresh EC2 instance, but will eventually stop working for unknown reasons. I also get a large amount of 500 error codes whenever navigating the web UI while the Graylog service is functioning.
Here are my log and configuration files:
/var/log/graylog-server/server.log
2018-01-23T16:10:48.127Z INFO [CmdLineTool] Loaded plugin: AWS plugins 2.4.1 [org.graylog.aws.plugin.AWSPlugin]
2018-01-23T16:10:48.129Z INFO [CmdLineTool] Loaded plugin: Elastic Beats Input 2.4.1 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-01-23T16:10:48.130Z INFO [CmdLineTool] Loaded plugin: CEF Input 2.4.1 [org.graylog.plugins.cef.CEFInputPlugin]
2018-01-23T16:10:48.131Z INFO [CmdLineTool] Loaded plugin: Collector 2.4.1 [org.graylog.plugins.collector.CollectorPlugin]
2018-01-23T16:10:48.131Z INFO [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.4.1 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2018-01-23T16:10:48.132Z INFO [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.4.1 [org.graylog.plugins.map.MapWidgetPlugin]
2018-01-23T16:10:48.133Z INFO [CmdLineTool] Loaded plugin: NetFlow Plugin 2.4.1 [org.graylog.plugins.netflow.NetFlowPlugin]
2018-01-23T16:10:48.139Z INFO [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.4.1 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-01-23T16:10:48.139Z INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.4.1 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2018-01-23T16:10:48.509Z INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2018-01-23T16:10:48.781Z INFO [Version] HV000001: Hibernate Validator 5.1.3.Final
2018-01-23T16:10:51.698Z INFO [InputBufferImpl] Message journal is enabled.
2018-01-23T16:10:51.747Z INFO [NodeId] Node ID: 02bdc1df-4cf8-4e32-a9cb-ee6a49086655
2018-01-23T16:10:52.101Z INFO [LogManager] Loading logs.
2018-01-23T16:10:52.112Z INFO [LogManager] Logs loading complete.
2018-01-23T16:10:52.190Z INFO [LogManager] Created log for partition [messagejournal,0] in /var/lib/graylog-server/journal with properties {file.delete.delay.ms -> 60000, compact -> false, max.message.bytes -> 104857600, min.insync.replicas -> 1, segment.jitter.ms -> 0, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> true, retention.bytes -> 5368709120, delete.retention.ms -> 86400000, flush.ms -> 60000, segment.bytes -> 104857600, segment.ms -> 3600000, retention.ms -> 43200000, flush.messages -> 1000000, segment.index.bytes -> 1048576}.
2018-01-23T16:10:52.190Z INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2018-01-23T16:10:52.204Z INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2018-01-23T16:10:52.226Z INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2018-01-23T16:10:52.332Z INFO [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2018-01-23T16:10:52.403Z INFO [connection] Opened connection [connectionId{localValue:1, serverValue:1}] to localhost:27017
2018-01-23T16:10:52.407Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 6, 10]}, minWireVersion=0, maxWireVersion=2, maxDocumentSize=16777216, roundTripTimeNanos=413906}
2018-01-23T16:10:52.424Z INFO [connection] Opened connection [connectionId{localValue:2, serverValue:2}] to localhost:27017
2018-01-23T16:10:53.276Z INFO [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2018-01-23T16:10:53.277Z INFO [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2018-01-23T16:10:53.420Z INFO [JestClientFactory] Using custom ObjectMapper instance
2018-01-23T16:10:53.420Z INFO [JestClientFactory] Node Discovery enabled...
2018-01-23T16:10:53.441Z INFO [JestClientFactory] Idle connection reaping disabled...
2018-01-23T16:10:53.892Z INFO [AbstractJestClient] Setting server pool to a list of 1 servers: [http://172.31.3.188:9200]
2018-01-23T16:10:54.004Z INFO [RoleServiceImpl] Admin role is missing or invalid, re-adding it as a built-in role.
2018-01-23T16:10:54.199Z INFO [RoleServiceImpl] Reader role is missing or invalid, re-adding it as a built-in role.
2018-01-23T16:10:54.402Z INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-01-23T16:10:57.207Z INFO [RulesEngineProvider] No static rules file loaded.
2018-01-23T16:10:57.229Z INFO [connection] Opened connection [connectionId{localValue:3, serverValue:3}] to localhost:27017
2018-01-23T16:10:57.405Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-01-23T16:10:57.430Z INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-01-23T16:10:57.465Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-01-23T16:10:57.487Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-01-23T16:10:57.510Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-01-23T16:10:57.540Z WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-01-23T16:10:58.059Z INFO [ServerBootstrap] Graylog server 2.4.1+0f97411 starting up
2018-01-23T16:10:58.060Z INFO [ServerBootstrap] JRE: Oracle Corporation 1.8.0_151 on Linux 4.4.0-1047-aws
2018-01-23T16:10:58.060Z INFO [ServerBootstrap] Deployment: deb
2018-01-23T16:10:58.060Z INFO [ServerBootstrap] OS: Ubuntu 16.04.3 LTS (xenial)
2018-01-23T16:10:58.060Z INFO [ServerBootstrap] Arch: amd64
2018-01-23T16:10:58.068Z WARN [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
/var/log/elasticsearch/graylog.log:
[2018-01-23T16:09:53,853][INFO ][o.e.n.Node ] [graylog-test] initializing ...
[2018-01-23T16:09:53,929][INFO ][o.e.e.NodeEnvironment ] [graylog-test] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [5.4gb], net total_space [7.6gb], spins? [no], types [ext4]
[2018-01-23T16:09:53,929][INFO ][o.e.e.NodeEnvironment ] [graylog-test] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-01-23T16:09:53,931][INFO ][o.e.n.Node ] [graylog-test] node name [graylog-test], node ID [IFEoIUslR5mf5ULYJesdQw]
[2018-01-23T16:09:53,931][INFO ][o.e.n.Node ] [graylog-test] version[5.6.2], pid[18453], build[57e20f3/2017-09-23T13:16:45.703Z], OS[Linux/4.4.0-1047-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-01-23T16:09:53,931][INFO ][o.e.n.Node ] [graylog-test] JVM arguments [-Xms1975m, -Xmx1975m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [aggs-matrix-stats]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [ingest-common]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-expression]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-groovy]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-mustache]
[2018-01-23T16:09:54,960][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-painless]
[2018-01-23T16:09:54,961][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [parent-join]
[2018-01-23T16:09:54,961][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [percolator]
[2018-01-23T16:09:54,961][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [reindex]
[2018-01-23T16:09:54,961][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [transport-netty3]
[2018-01-23T16:09:54,961][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [transport-netty4]
[2018-01-23T16:09:54,967][INFO ][o.e.p.PluginsService ] [graylog-test] no plugins loaded
[2018-01-23T16:09:57,888][INFO ][o.e.d.DiscoveryModule ] [graylog-test] using discovery type [zen]
[2018-01-23T16:09:58,469][INFO ][o.e.n.Node ] [graylog-test] initialized
[2018-01-23T16:09:58,469][INFO ][o.e.n.Node ] [graylog-test] starting ...
[2018-01-23T16:09:58,627][INFO ][o.e.t.TransportService ] [graylog-test] publish_address {172.31.3.188:9300}, bound_addresses {[::]:9300}
[2018-01-23T16:09:58,638][INFO ][o.e.b.BootstrapChecks ] [graylog-test] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-01-23T16:10:01,687][INFO ][o.e.c.s.ClusterService ] [graylog-test] new_master {graylog-test}{IFEoIUslR5mf5ULYJesdQw}{VxkYZf7OSy23TRgIFij7QA}{172.31.3.188}{172.31.3.188:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-01-23T16:10:01,712][INFO ][o.e.h.n.Netty4HttpServerTransport] [graylog-test] publish_address {172.31.3.188:9200}, bound_addresses {[::]:9200}
[2018-01-23T16:10:01,712][INFO ][o.e.n.Node ] [graylog-test] started
[2018-01-23T16:10:01,720][INFO ][o.e.g.GatewayService ] [graylog-test] recovered [0] indices into cluster_state
[2018-01-23T16:14:29,103][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [graylog_0] creating index, cause [api], templates [graylog-internal], shards [1]/[0], mappings [message]
[2018-01-23T16:14:29,358][INFO ][o.e.c.r.a.AllocationService] [graylog-test] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2018-01-23T17:59:29,365][INFO ][o.e.n.Node ] [graylog-test] stopping ...
[2018-01-23T17:59:29,388][INFO ][o.e.n.Node ] [graylog-test] stopped
[2018-01-23T17:59:29,388][INFO ][o.e.n.Node ] [graylog-test] closing ...
[2018-01-23T17:59:29,397][INFO ][o.e.n.Node ] [graylog-test] closed
[2018-01-23T17:59:52,198][INFO ][o.e.n.Node ] [graylog-test] initializing ...
[2018-01-23T17:59:52,379][INFO ][o.e.e.NodeEnvironment ] [graylog-test] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [4.9gb], net total_space [7.6gb], spins? [no], types [ext4]
[2018-01-23T17:59:52,379][INFO ][o.e.e.NodeEnvironment ] [graylog-test] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-01-23T17:59:52,403][INFO ][o.e.n.Node ] [graylog-test] node name [graylog-test], node ID [IFEoIUslR5mf5ULYJesdQw]
[2018-01-23T17:59:52,403][INFO ][o.e.n.Node ] [graylog-test] version[5.6.2], pid[27845], build[57e20f3/2017-09-23T13:16:45.703Z], OS[Linux/4.4.0-1047-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-01-23T17:59:52,403][INFO ][o.e.n.Node ] [graylog-test] JVM arguments [-Xms1975m, -Xmx1975m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [aggs-matrix-stats]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [ingest-common]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-expression]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-groovy]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-mustache]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-painless]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [parent-join]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [percolator]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [reindex]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [transport-netty3]
[2018-01-23T17:59:55,794][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [transport-netty4]
[2018-01-23T17:59:55,795][INFO ][o.e.p.PluginsService ] [graylog-test] loaded plugin [discovery-ec2]
[2018-01-23T17:59:55,795][INFO ][o.e.p.PluginsService ] [graylog-test] loaded plugin [x-pack]
[2018-01-23T17:59:59,979][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin org.elasticsearch.xpack.XPackPlugin
[2018-01-23T18:00:04,043][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/27989] [Main.cc@128] controller (64 bit): Version 5.6.2 (Build 228329870d1c63) Copyright (c) 2017 Elasticsearch BV
[2018-01-23T18:00:04,097][INFO ][o.e.d.DiscoveryModule ] [graylog-test] using discovery type [zen]
[2018-01-23T18:00:05,626][INFO ][o.e.n.Node ] [graylog-test] initialized
[2018-01-23T18:00:05,631][INFO ][o.e.n.Node ] [graylog-test] starting ...
[2018-01-23T18:00:06,035][INFO ][o.e.t.TransportService ] [graylog-test] publish_address {172.31.3.188:9300}, bound_addresses {172.31.3.188:9300}
[2018-01-23T18:00:06,048][INFO ][o.e.b.BootstrapChecks ] [graylog-test] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-01-23T18:00:09,136][INFO ][o.e.c.s.ClusterService ] [graylog-test] new_master {graylog-test}{IFEoIUslR5mf5ULYJesdQw}{AEm6nIOjQ4uRGglvyW0lRQ}{172.31.3.188}{172.31.3.188:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-01-23T18:00:09,208][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [graylog-test] publish_address {172.31.3.188:9200}, bound_addresses {172.31.3.188:9200}
[2018-01-23T18:00:09,208][INFO ][o.e.n.Node ] [graylog-test] started
[2018-01-23T18:00:09,799][INFO ][o.e.g.GatewayService ] [graylog-test] recovered [1] indices into cluster_state
[2018-01-23T18:00:09,871][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [graylog-test] successfully created .ml-meta index template
[2018-01-23T18:00:09,973][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [graylog-test] successfully created .ml-state index template
[2018-01-23T18:00:10,043][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [graylog-test] successfully created .ml-notifications index template
[2018-01-23T18:00:10,289][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [graylog-test] successfully created .ml-anomalies- index template
[2018-01-23T18:00:11,130][INFO ][o.e.l.LicenseService ] [graylog-test] license [ccae04a5-899a-4e5c-b49b-4b7366f83ca3] mode [trial] - valid
[2018-01-23T18:00:11,137][INFO ][o.e.c.r.a.AllocationService] [graylog-test] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2018-01-23T18:00:16,217][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [.monitoring-es-6-2018.01.23] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc]
[2018-01-23T18:00:16,298][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [.watches] creating index, cause [auto(bulk api)], templates [watches], shards [1]/[1], mappings [watch]
[2018-01-23T18:00:16,494][INFO ][o.e.c.m.MetaDataMappingService] [graylog-test] [.watches/C6bAXWo6TOabt-rdzRKizw] update_mapping [watch]
[2018-01-23T18:01:17,017][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [.triggered_watches] creating index, cause [auto(bulk api)], templates [triggered_watches], shards [1]/[1], mappings [triggered_watch]
[2018-01-23T18:01:17,237][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [.watcher-history-6-2018.01.23] creating index, cause [auto(bulk api)], templates [.watch-history-6], shards [1]/[1], mappings [doc]
[2018-01-23T18:01:17,350][INFO ][o.e.c.m.MetaDataMappingService] [graylog-test] [.watcher-history-6-2018.01.23/QhZscqciR5mK6eLqhVpXrA] update_mapping [doc]
[2018-01-23T18:01:17,396][INFO ][o.e.c.m.MetaDataCreateIndexService] [graylog-test] [.monitoring-alerts-6] creating index, cause [auto(bulk api)], templates [.monitoring-alerts], shards [1]/[1], mappings [doc]
[2018-01-23T18:01:17,476][INFO ][o.e.c.m.MetaDataMappingService] [graylog-test] [.watcher-history-6-2018.01.23/QhZscqciR5mK6eLqhVpXrA] update_mapping [doc]
[2018-01-23T18:13:56,869][INFO ][o.e.n.Node ] [graylog-test] stopping ...
[2018-01-23T18:13:56,899][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/27989] [Main.cc@168] Ml controller exiting
[2018-01-23T18:13:56,901][INFO ][o.e.x.w.w.WatchStore ] [graylog-test] stopped watch store
[2018-01-23T18:13:56,901][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
[2018-01-23T18:13:57,411][INFO ][o.e.n.Node ] [graylog-test] stopped
[2018-01-23T18:13:57,412][INFO ][o.e.n.Node ] [graylog-test] closing ...
[2018-01-23T18:13:57,423][INFO ][o.e.n.Node ] [graylog-test] closed
[2018-01-23T18:13:59,607][INFO ][o.e.n.Node ] [graylog-test] initializing ...
[2018-01-23T18:13:59,793][INFO ][o.e.e.NodeEnvironment ] [graylog-test] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [4.9gb], net total_space [7.6gb], spins? [no], types [ext4]
[2018-01-23T18:13:59,794][INFO ][o.e.e.NodeEnvironment ] [graylog-test] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-01-23T18:13:59,830][INFO ][o.e.n.Node ] [graylog-test] node name [graylog-test], node ID [IFEoIUslR5mf5ULYJesdQw]
[2018-01-23T18:13:59,832][INFO ][o.e.n.Node ] [graylog-test] version[5.6.2], pid[28394], build[57e20f3/2017-09-23T13:16:45.703Z], OS[Linux/4.4.0-1047-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-01-23T18:13:59,833][INFO ][o.e.n.Node ] [graylog-test] JVM arguments [-Xms1975m, -Xmx1975m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-01-23T18:14:02,834][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [aggs-matrix-stats]
[2018-01-23T18:14:02,834][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [ingest-common]
[2018-01-23T18:14:02,834][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-expression]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-groovy]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-mustache]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [lang-painless]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [parent-join]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [percolator]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [reindex]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [transport-netty3]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService ] [graylog-test] loaded module [transport-netty4]
[2018-01-23T18:14:02,840][INFO ][o.e.p.PluginsService ] [graylog-test] loaded plugin [discovery-ec2]
[2018-01-23T18:14:02,841][INFO ][o.e.p.PluginsService ] [graylog-test] loaded plugin [x-pack]
[2018-01-23T18:14:06,766][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin org.elasticsearch.xpack.XPackPlugin
[2018-01-23T18:14:08,533][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/28524] [Main.cc@128] controller (64 bit): Version 5.6.2 (Build 228329870d1c63) Copyright (c) 2017 Elasticsearch BV
[2018-01-23T18:14:08,617][INFO ][o.e.d.DiscoveryModule ] [graylog-test] using discovery type [zen]
[2018-01-23T18:14:10,200][INFO ][o.e.n.Node ] [graylog-test] initialized
[2018-01-23T18:14:10,200][INFO ][o.e.n.Node ] [graylog-test] starting ...
[2018-01-23T18:14:10,561][INFO ][o.e.t.TransportService ] [graylog-test] publish_address {172.31.3.188:9300}, bound_addresses {172.31.3.188:9300}
[2018-01-23T18:14:10,574][INFO ][o.e.b.BootstrapChecks ] [graylog-test] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-01-23T18:14:13,674][INFO ][o.e.c.s.ClusterService ] [graylog-test] new_master {graylog-test}{IFEoIUslR5mf5ULYJesdQw}{o8seVW0XSxSTxmYbKGt4eA}{172.31.3.188}{172.31.3.188:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-01-23T18:14:13,767][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [graylog-test] publish_address {172.31.3.188:9200}, bound_addresses {172.31.3.188:9200}
[2018-01-23T18:14:13,767][INFO ][o.e.n.Node ] [graylog-test] started
[2018-01-23T18:14:14,684][INFO ][o.e.l.LicenseService ] [graylog-test] license [ccae04a5-899a-4e5c-b49b-4b7366f83ca3] mode [trial] - valid
[2018-01-23T18:14:14,714][INFO ][o.e.g.GatewayService ] [graylog-test] recovered [6] indices into cluster_state
[2018-01-23T18:14:15,514][INFO ][o.e.c.r.a.AllocationService] [graylog-test] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[graylog_0][0]] ...]).
[2018-01-23T19:26:45,799][INFO ][o.e.x.s.a.f.FileUserPasswdStore] [graylog-test] users file [/etc/elasticsearch/x-pack/users] changed. updating users... )
[2018-01-23T19:26:45,800][INFO ][o.e.x.s.a.f.FileUserRolesStore] [graylog-test] users roles file [/etc/elasticsearch/x-pack/users_roles] changed. updating users roles...
[2018-01-23T19:26:45,801][INFO ][o.e.x.s.a.s.FileRolesStore] [graylog-test] updated roles (roles file [/etc/elasticsearch/x-pack/roles.yml] changed)
/etc/graylog/server/server.conf:
############################
# GRAYLOG CONFIGURATION FILE
############################
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = secret
root_username = custom
root_password_sha2 = secret
root_email = "my@email.com"
root_timezone = America/Detroit
### Plugins Directory
plugin_dir = /usr/share/graylog-server/plugin
### REST Interface
rest_listen_uri = http://0.0.0.0:12900/api/
rest_transport_uri = http://34.238.121.38:12900
rest_enable_cors = true
rest_enable_gzip = true
rest_enable_tls = false
### Web Interface
web_enable = true
web_listen_uri = http://0.0.0.0:9000
web_endpoint_uri = http://127.0.0.1:9000/api
web_enable_cors = true
web_enable_gzip = true
web_enable_tls = false
### Elasticearch
elasticsearch_discovery_enabled = true
elasticsearch_hosts = http://127.0.0.1:9200
elasticsearch_connect_timeout = 10s
elasticsearch_socket_timeout = 60s
# elasticsearch_idle_timeout = -1s
elasticsearch_max_total_connections = 20
elasticsearch_max_total_connections_per_route = 2
elasticsearch_max_retries = 2
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 1
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
### Search Options
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1
elasticsearch_transport_tcp_port = 9300
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
/etc/elasticsearch/elasticsearch.yml:
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# THIS FILE IS MANAGED BY CHEF, DO NOT EDIT MANUALLY, YOUR CHANGES WILL BE OVERWRITTEN!
#
# Please see the documentation for further information on configuration options:
# <https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html>
#
---
cluster.name: graylog
node.name: graylog-test
path.conf: "/etc/elasticsearch"
path.data: "/var/lib/elasticsearch"
path.logs: "/var/log/elasticsearch"
network.host: 172.31.3.188
http.port: 9200
transport.tcp.port: 9300
I am installing everything with a custom wrapper cookbook on a t2.medium. I believe 4 Gigs of RAM is enough. If not, that’s an easy fix, but I still dont know how to resolve the 500 level errors whenever navigating the web UI.