Elasticsearch not available error message

Graylog Central (peer support)
1

Can anyone lead me in the right direction here ? What does this error mean ?

2018-03-23T13:07:44.667-04:00 WARN  [ProxiedResource] Unable to call http://10.0.103.14:12900/api/system/metrics/multiple on node <3da6a9b5-f670-4d1e-8028-2ea45740d0e1>
java.net.ConnectException: Failed to connect to /10.0.103.14:12900
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:240) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:158) ~[graylog.jar:?]
	at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256) ~[graylog.jar:?]
	at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134) ~[graylog.jar:?]
	at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113) ~[graylog.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
	at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) ~[graylog.jar:?]
	at okhttp3.RealCall.execute(RealCall.java:77) ~[graylog.jar:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
	at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]
	at okhttp3.internal.platform.Platform.connectSocket(Platform.java:125) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:238) ~[graylog.jar:?]
	... 26 more
2018-03-23T14:03:03.981-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-03-23T15:03:03.981-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-03-23T15:05:33.887-04:00 INFO  [connection] Opened connection [connectionId{localValue:13, serverValue:770}] to graylog3-mgmt.philasd.net:27017
2018-03-23T15:12:51.952-04:00 INFO  [connection] Opened connection [connectionId{localValue:14, serverValue:4}] to graylog2-mgmt.philasd.net:27017
2018-03-23T15:13:03.861-04:00 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2

here is the graylog.log file.

java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna6538669404814768016.tmp: /tmp/jna--1985354563/jna6538669404814768016.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_161]
        at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_161]
        at java.lang.System.load(System.java:1086) ~[?:1.8.0_161]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_161]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_161]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:195) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.8.jar:5.6.8]
[2018-03-23T10:52:05,339][WARN ][o.e.b.Natives            ] cannot check if running as root because JNA is not available
[2018-03-23T10:52:05,339][WARN ][o.e.b.Natives            ] cannot register console handler because JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_AS beacuse JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2018-03-23T10:52:05,451][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-23T10:52:05,604][INFO ][o.e.e.NodeEnvironment    ] [qDUPSsw] using [1] data paths, mounts [[/var (/dev/mapper/centos-var)]], net usable_space [751.8mb], net total_space [1.4gb], spins? [possibly], types [xfs]
[2018-03-23T10:52:05,605][INFO ][o.e.e.NodeEnvironment    ] [qDUPSsw] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-03-23T10:52:05,614][INFO ][o.e.n.Node               ] node name [qDUPSsw] derived from node ID [qDUPSswHQVWUbdZLO-knyQ]; set [node.name] to override
[2018-03-23T10:52:05,614][INFO ][o.e.n.Node               ] version[5.6.8], pid[2611], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-03-23T10:52:05,615][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [aggs-matrix-stats]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [ingest-common]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-expression]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-groovy]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-mustache]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-painless]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [parent-join]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [percolator]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [reindex]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [transport-netty3]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [transport-netty4]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] no plugins loaded
[2018-03-23T10:52:08,754][INFO ][o.e.d.DiscoveryModule    ] [qDUPSsw] using discovery type [zen]
[2018-03-23T10:52:09,461][INFO ][o.e.n.Node               ] initialized
[2018-03-23T10:52:09,462][INFO ][o.e.n.Node               ] [qDUPSsw] starting ...
[2018-03-23T10:52:09,662][INFO ][o.e.t.TransportService   ] [qDUPSsw] publish_address {172.16.5.11:9300}, bound_addresses {172.16.5.11:9300}
[2018-03-23T10:52:09,672][INFO ][o.e.b.BootstrapChecks    ] [qDUPSsw] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-03-23T10:52:12,861][INFO ][o.e.c.s.ClusterService   ] [qDUPSsw] detected_master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300}, added {{ueq7c6I}{ueq7c6IVStKlNtZhePNEOg}{sjvFf7A-T9GBRoMWxvmjaw}{172.16.5.13}{172.16.5.13:9300},{SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300},}, reason: zen-disco-receive(from master [master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300} committed version [65]])
[2018-03-23T10:52:12,949][INFO ][o.e.h.n.Netty4HttpServerTransport] [qDUPSsw] publish_address {172.16.5.11:9200}, bound_addresses {172.16.5.11:9200}
[2018-03-23T10:52:12,949][INFO ][o.e.n.Node               ] [qDUPSsw] started
3

First of all, please properly format the log and configuration snippets in your posts for better readability.
See Creating and highlighting code blocks - GitHub Docs for details.

Check your Graylog configuration files for mismatching configuration settings for rest_listen_uri and rest_transport_uri.

4

all three have this ;
graylog1
rest_listen_uri = http://10.0.103.11:12900/api/
rest_transport_uri = http://10.0.103.11:12900/api/

graylog3
rest_listen_uri = http://10.0.103.14:12900/api/
rest_transport_uri = http://10.0.103.14:12900/api/

graylog2
rest_listen_uri = http://10.0.103.13:12900/api/
rest_transport_uri = http://10.0.103.13:12900/api/

When I had the just single node this fixed the issue.

5

I’m wondering , should they all point to graylog1 ?

6

here is the logs entry for graylog1. Am I on the right track ?

2018-03-23T17:08:31.234-04:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2018-03-23T17:08:31.432-04:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2018-03-23T17:08:42.298-04:00 INFO [NetworkListener] Started listener bound to [10.0.103.11:12900]
2018-03-23T17:08:42.300-04:00 INFO [HttpServer] [HttpServer] Started.
2018-03-23T17:08:42.300-04:00 INFO [JerseyService] Started REST API at http://10.0.103.11:12900/api/
2018-03-23T17:08:45.184-04:00 INFO [NetworkListener] Started listener bound to [10.0.103.11:9000]
2018-03-23T17:08:45.184-04:00 INFO [HttpServer] [HttpServer-1] Started.
2018-03-23T17:08:45.184-04:00 INFO [JerseyService] Started Web Interface at http://10.0.103.11:9000/
2018-03-23T17:08:45.185-04:00 INFO [ServiceManagerListener] Services are healthy

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.