hello community
after having an installed and running on permis Minimum setup.
I’ve tried to install a distributed setup in AWS.
ive installed ES 6.8.4 on ubuntu 18.0.4
and graylog 3.1.3 with mongodb 4.0.13 on a separate ubuntu 18.04.
both on the same vpc and subnet.
all services are up via “systemctl status” commands
and ES is accessible in port 9200/9300 from the graylog machine(see below).
but the web interface is not accessible and checking with “lsof -Pni” it don’t listenning in port 9000.
server.log shows that graceful restart was triggered although the service is still up.
can’t find the reason and I’ve looked at graylog, mongodb and es logs.
I assume I’ve just don’t understand the logs
appreciate the help, I’m a beginner here.
all logs and status are pasted bellow
thanks
root@GL-US:/etc# systemctl status graylog-server.service
● graylog-server.service - Graylog server
Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-11-12 09:33:33 UTC; 30min ago
Docs: http://docs.graylog.org/
Main PID: 16765 (graylog-server)
Tasks: 101 (limit: 4703)
CGroup: /system.slice/graylog-server.service
├─16765 /bin/sh /usr/share/graylog-server/bin/graylog-server
└─16804 /usr/bin/java -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -XX:+UseParN
Nov 12 09:33:33 GL-US systemd[1]: Started Graylog server.
root@GL-US:/etc# systemctl status mongod.service
● mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2019-11-11 11:43:24 UTC; 22h ago
Docs: https://docs.mongodb.org/manual
Main PID: 13264 (mongod)
CGroup: /system.slice/mongod.service
└─13264 /usr/bin/mongod --config /etc/mongod.conf
Nov 11 11:43:24 GL-US systemd[1]: Started MongoDB Database Server.
root@GL-US:/etc#
curl -X GET http://xxxxxxx:9200
{
“name” : “moQ5Ul-”,
“cluster_name” : “graylog”,
“cluster_uuid” : “BhS7is9NQUyfyW4QDZ8CDw”,
“version” : {
“number” : “6.8.4”,
“build_flavor” : “oss”,
“build_type” : “deb”,
“build_hash” : “bca0c8d”,
“build_date” : “2019-10-16T06:19:49.319352Z”,
“build_snapshot” : false,
“lucene_version” : “7.7.2”,
“minimum_wire_compatibility_version” : “5.6.0”,
“minimum_index_compatibility_version” : “5.0.0”
},
“tagline” : “You Know, for Search”
}
graylog log:
2019-11-12T09:34:06.287Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob] complete, took <0ms>.
2019-11-12T09:34:06.288Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp].
2019-11-12T09:34:06.288Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp] complete, took <0ms>.
2019-11-12T09:34:06.288Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2019-11-12T09:34:06.288Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2019-11-12T09:34:06.295Z INFO [GracefulShutdownService] Running graceful shutdown for <1> shutdown hooks
2019-11-12T09:34:06.296Z INFO [GracefulShutdownService] Initiate shutdown for
2019-11-12T09:34:06.296Z INFO [GracefulShutdownService] Finished shutdown for , took 0 ms
2019-11-12T09:34:06.299Z INFO [LogManager] Shutting down.
2019-11-12T09:34:06.309Z INFO [LookupDataAdapterRefreshService] Stopping 0 jobs
2019-11-12T09:34:06.334Z INFO [LogManager] Shutdown complete.
2019-11-12T09:34:06.338Z INFO [Buffers] Waiting until all buffers are empty.
2019-11-12T09:34:06.339Z INFO [Buffers] All buffers are empty. Continuing.
2019-11-12T09:34:06.340Z INFO [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2019-11-12T09:34:06.342Z INFO [JournalReader] Stopping.
2019-11-12T09:34:36.298Z ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], PeriodicalsService [TERMINATED], Mong
oDBProcessingStatusRecorderService [TERMINATED], GracefulShutdownService [TERMINATED], StreamCacheService [TERMINATED], OutputSetupService [TERMINATED], LookupTableService [TERMINATED], EtagService [TERMINATED],
ConfigurationEtagService [TERMINATED], KafkaJournal [TERMINATED], BufferSynchronizerService [TERMINATED], JournalReader [TERMINATED]], FAILED=[JerseyService [FAILED]]}
2019-11-12T09:34:36.298Z ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:148) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]
Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_222]
at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_222]
at sun.nio.ch.Net.bind(Net.java:425) ~[?:1.8.0_222]
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:1.8.0_222]
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[?:1.8.0_222]
at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bindToChannelAndAddress(TCPNIOBindingHandler.java:131) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bind(TCPNIOBindingHandler.java:88) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:238) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:218) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:209) ~[graylog.jar:?]
at org.glassfish.grizzly.http.server.NetworkListener.start(NetworkListener.java:723) ~[graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:277) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:197) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:143) ~[graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222]
2019-11-12T09:34:36.302Z INFO [Server] SIGNAL received. Shutting down.
2019-11-12T09:34:36.308Z INFO [GracefulShutdown] Graceful shutdown initiated.
2019-11-12T09:34:36.308Z INFO [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2019-11-12T09:34:40.310Z INFO [GracefulShutdown] Goodbye.
mongodb log:
019-11-11T13:29:15.387+0000 I NETWORK [listener] connection accepted from 127.0.0.1:44482 #49 (6 connections now open)
2019-11-11T13:29:15.387+0000 I NETWORK [conn49] received client metadata from 127.0.0.1:44482 conn49: { driver: { name: “mongo-java-driver”, version: “unknown” }, os: { type: “Linux”, name: “Linux”, architectur
e: “amd64”, version: “4.15.0-1051-aws” }, platform: “Java/Private Build/1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10” }
2019-11-11T13:29:15.407+0000 I NETWORK [listener] connection accepted from 127.0.0.1:44484 #50 (7 connections now open)
2019-11-11T13:29:15.407+0000 I NETWORK [conn50] received client metadata from 127.0.0.1:44484 conn50: { driver: { name: “mongo-java-driver”, version: “unknown” }, os: { type: “Linux”, name: “Linux”, architectur
e: “amd64”, version: “4.15.0-1051-aws” }, platform: “Java/Private Build/1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10” }
2019-11-12T09:33:33.259+0000 I NETWORK [conn50] end connection 127.0.0.1:44484 (6 connections now open)
2019-11-12T09:33:33.259+0000 I NETWORK [conn48] end connection 127.0.0.1:44480 (5 connections now open)
2019-11-12T09:33:33.260+0000 I NETWORK [conn46] end connection 127.0.0.1:44476 (4 connections now open)
2019-11-12T09:33:33.260+0000 I NETWORK [conn49] end connection 127.0.0.1:44482 (3 connections now open)
2019-11-12T09:33:33.260+0000 I NETWORK [conn44] end connection 127.0.0.1:44472 (2 connections now open)
2019-11-12T09:33:33.260+0000 I NETWORK [conn45] end connection 127.0.0.1:44474 (1 connection now open)
2019-11-12T09:33:33.261+0000 I NETWORK [conn47] end connection 127.0.0.1:44478 (0 connections now open)
2019-11-12T09:33:39.844+0000 I NETWORK [listener] connection accepted from 127.0.0.1:44568 #51 (1 connection now open)
2019-11-12T09:33:39.871+0000 I NETWORK [conn51] received client metadata from 127.0.0.1:44568 conn51: { driver: { name: “mongo-java-driver”, version: “unknown” }, os: { type: “Linux”, name: “Linux”, architectur
e: “amd64”, version: “4.15.0-1051-aws” }, platform: “Java/Private Build/1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10” }
2019-11-12T09:33:39.901+0000 I NETWORK [listener] connection accepted from 127.0.0.1:44570 #52 (2 connections now open)
2019-11-12T09:33:39.902+0000 I NETWORK [conn52] received client metadata from 127.0.0.1:44570 conn52: { driver: { name: “mongo-java-driver”, version: “unknown” }, os: { type: “Linux”, name: “Linux”, architectur
e: “amd64”, version: “4.15.0-1051-aws” }, platform: “Java/Private Build/1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10” }
2019-11-12T09:33:41.262+0000 I NETWORK [listener] connection accepted from 127.0.0.1:44572 #53 (3 connections now open)
2019-11-12T09:33:41.263+0000 I NETWORK [conn53] received client metadata from 127.0.0.1:44572 conn53: { driver: { name: “mongo-java-driver”, version: “unknown” }, os: { type: “Linux”, name: “Linux”, architectur
e: “amd64”, version: “4.15.0-1051-aws” }, platform: “Java/Private Build/1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10” }
2019-11-12T09:33:42.629+0000 I NETWORK [listener] connection accepted from 127.0.0.1:44574 #54 (4 connections now open)
2019-11-12T09:33:42.630+0000 I NETWORK [listener] connection accepted from 127.0.0.1:44576 #55 (5 connections now open)
2019-11-12T09:33:42.630+0000 I NETWORK [conn55] received client metadata from 127.0.0.1:44576 conn55: { driver: { name: “mongo-java-driver”, version: “unknown” }, os: { type: “Linux”, name: “Linux”, architectur
e: “amd64”, version: “4.15.0-1051-aws” }, platform: “Java/Private Build/1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10” }
2019-11-12T09:33:42.631+0000 I NETWORK [conn54] received client metadata from 127.0.0.1:44574 conn54: { driver: { name: “mongo-java-driver”, version: “unknown” }, os: { type: “Linux”, name: “Linux”, architectur
e: “amd64”, version: “4.15.0-1051-aws” }, platform: “Java/Private Build/1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10” }
2019-11-12T09:33:42.769+0000 I NETWORK [listener] connection accepted from 127.0.0.1:44578 #56 (6 connections now open)
2019-11-12T09:33:42.769+0000 I NETWORK [conn56] received client metadata from 127.0.0.1:44578 conn56: { driver: { name: “mongo-java-driver”, version: “unknown” }, os: { type: “Linux”, name: “Linux”, architectur
e: “amd64”, version: “4.15.0-1051-aws” }, platform: “Java/Private Build/1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10” }
2019-11-12T09:33:42.770+0000 I NETWORK [listener] connection accepted from 127.0.0.1:44580 #57 (7 connections now open)
2019-11-12T09:33:42.770+0000 I NETWORK [conn57] received client metadata from 127.0.0.1:44580 conn57: { driver: { name: “mongo-java-driver”, version: “unknown” }, os: { type: “Linux”, name: “Linux”, architectur
e: “amd64”, version: “4.15.0-1051-aws” }, platform: “Java/Private Build/1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10” }
ES log:
[2019-11-11T11:35:31,495][INFO ][o.e.n.Node ] [moQ5Ul-] node name derived from node ID [moQ5Ul-0Tn6rQEV6A5M8hg]; set [node.name] to override
[2019-11-11T11:35:31,495][INFO ][o.e.n.Node ] [moQ5Ul-] version[6.8.4], pid[9737], build[oss/deb/bca0c8d/2019-10-16T06:19:49.319352Z], OS[Linux/4.15.0-1051-aws/amd64], JVM[Private Build/OpenJDK 64-
Bit Server VM/1.8.0_222/25.222-b10]
[2019-11-11T11:35:31,496][INFO ][o.e.n.Node ] [moQ5Ul-] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.netwo
rkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUns
afe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-4945584830514863351,
-XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+
PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/ela
sticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb]
[2019-11-11T11:35:32,385][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [aggs-matrix-stats]
[2019-11-11T11:35:32,386][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [analysis-common]
[2019-11-11T11:35:32,386][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [ingest-common]
[2019-11-11T11:35:32,386][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [ingest-geoip]
[2019-11-11T11:35:32,386][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [ingest-user-agent]
[2019-11-11T11:35:32,386][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [lang-expression]
[2019-11-11T11:35:32,386][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [lang-mustache]
[2019-11-11T11:35:32,386][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [lang-painless]
[2019-11-11T11:35:32,387][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [mapper-extras]
[2019-11-11T11:35:32,387][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [parent-join]
[2019-11-11T11:35:32,387][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [percolator]
[2019-11-11T11:35:32,387][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [rank-eval]
[2019-11-11T11:35:32,387][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [reindex]
[2019-11-11T11:35:32,387][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [repository-url]
[2019-11-11T11:35:32,388][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [transport-netty4]
[2019-11-11T11:35:32,388][INFO ][o.e.p.PluginsService ] [moQ5Ul-] loaded module [tribe]
[2019-11-11T11:35:32,388][INFO ][o.e.p.PluginsService ] [moQ5Ul-] no plugins loaded
[2019-11-11T11:35:36,030][INFO ][o.e.d.DiscoveryModule ] [moQ5Ul-] using discovery type [zen] and host providers [settings]
[2019-11-11T11:35:36,660][INFO ][o.e.n.Node ] [moQ5Ul-] initialized
[2019-11-11T11:35:36,661][INFO ][o.e.n.Node ] [moQ5Ul-] starting …
[2019-11-11T11:35:36,894][INFO ][o.e.t.TransportService ] [moQ5Ul-] publish_address {172.31.80.172:9300}, bound_addresses {[::]:9300}
[2019-11-11T11:35:36,918][INFO ][o.e.b.BootstrapChecks ] [moQ5Ul-] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-11-11T11:35:39,988][INFO ][o.e.c.s.MasterService ] [moQ5Ul-] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {moQ5Ul-}{moQ5Ul-0Tn6rQEV6A5M8hg}{W_wMpyPCT8SQoy53ORi1nA}{172.31.80.172}{1
72.31.80.172:9300}
[2019-11-11T11:35:39,993][INFO ][o.e.c.s.ClusterApplierService] [moQ5Ul-] new_master {moQ5Ul-}{moQ5Ul-0Tn6rQEV6A5M8hg}{W_wMpyPCT8SQoy53ORi1nA}{172.31.80.172}{172.31.80.172:9300}, reason: apply cluster state (fro
m master [master {moQ5Ul-}{moQ5Ul-0Tn6rQEV6A5M8hg}{W_wMpyPCT8SQoy53ORi1nA}{172.31.80.172}{172.31.80.172:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-11-11T11:35:40,011][INFO ][o.e.h.n.Netty4HttpServerTransport] [moQ5Ul-] publish_address {172.31.80.172:9200}, bound_addresses {[::]:9200}
[2019-11-11T11:35:40,011][INFO ][o.e.n.Node ] [moQ5Ul-] started
[2019-11-11T11:35:40,027][INFO ][o.e.g.GatewayService ] [moQ5Ul-] recovered [0] indices into cluster_state
[2019-11-11T13:11:50,111][INFO ][o.e.c.m.MetaDataIndexTemplateService] [moQ5Ul-] adding template [graylog-internal] for index patterns [graylog_]
[2019-11-11T13:11:50,150][INFO ][o.e.c.m.MetaDataCreateIndexService] [moQ5Ul-] [graylog_0] creating index, cause [api], templates [graylog-internal], shards [2]/[0], mappings [message]
[2019-11-11T13:11:50,511][INFO ][o.e.c.r.a.AllocationService] [moQ5Ul-] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[graylog_0][0]] …]).
[2019-11-11T13:11:50,638][INFO ][o.e.c.m.MetaDataIndexTemplateService] [moQ5Ul-] adding template [gl-events-template] for index patterns [gl-events_]
[2019-11-11T13:11:50,662][INFO ][o.e.c.m.MetaDataCreateIndexService] [moQ5Ul-] [gl-events_0] creating index, cause [api], templates [gl-events-template], shards [2]/[0], mappings [message]
[2019-11-11T13:11:50,827][INFO ][o.e.c.r.a.AllocationService] [moQ5Ul-] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[gl-events_0][0]] …]).
[2019-11-11T13:11:50,895][INFO ][o.e.c.m.MetaDataIndexTemplateService] [moQ5Ul-] adding template [gl-system-events-template] for index patterns [gl-system-events_*]
[2019-11-11T13:11:50,920][INFO ][o.e.c.m.MetaDataCreateIndexService] [moQ5Ul-] [gl-system-events_0] creating index, cause [api], templates [gl-system-events-template], shards [2]/[0], mappings [message]
[2019-11-11T13:11:51,077][INFO ][o.e.c.r.a.AllocationService] [moQ5Ul-] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[gl-system-events_0][0]] …]).