Could not load field information

I have been struggling with this for 3 weeks now but I am unable to get the solution. Could someone please help with this ?
I have 3 nodes all of which are green in elasticsearch.

graylog1. server.conf

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxxxxxxxxx
root_username = sdpadmin
root_password_sha2 = xxxxxxxxxx
root_timezone = America/New_York
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://10.0.103.11:9000/api/
rest_transport_uri = http://10.0.103.11:9000/api/
web_listen_uri = http://10.0.103.11:9000/
elasticsearch_hosts = http://graylog1.philasd.net:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://user:password@172.16.5.11:27017,172.16.5.13:27017,172.16.5.14:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

elasticsearch.yml on graylog1

cluster.name: graylog
discovery.zen.ping.unicast.hosts: ["172.16.5.11", "172.16.5.13","172.16.5.14"]
discovery.zen.minimum_master_nodes: 1
node.master: true
network.host: 172.16.5.11
http.bind_host: 172.16.5.11
http.publish_host: 172.16.5.11
bootstrap.system_call_filter: false

graylog.log on graylog1

[2018-03-23T10:52:02,442][INFO ][o.e.n.Node               ] [qDUPSsw] closed
[2018-03-23T10:52:05,332][WARN ][o.e.b.Natives            ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna6538669404814768016.tmp: /tmp/jna--1985354563/jna6538669404814768016.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_161]
        at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_161]
        at java.lang.System.load(System.java:1086) ~[?:1.8.0_161]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_161]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_161]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:195) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.8.jar:5.6.8]
[2018-03-23T10:52:05,339][WARN ][o.e.b.Natives            ] cannot check if running as root because JNA is not available
[2018-03-23T10:52:05,339][WARN ][o.e.b.Natives            ] cannot register console handler because JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_AS beacuse JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2018-03-23T10:52:05,451][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-23T10:52:05,604][INFO ][o.e.e.NodeEnvironment    ] [qDUPSsw] using [1] data paths, mounts [[/var (/dev/mapper/centos-var)]], net usable_space [751.8mb], net total_space [1.4gb], spins? [possibly], types [xfs]
[2018-03-23T10:52:05,605][INFO ][o.e.e.NodeEnvironment    ] [qDUPSsw] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-03-23T10:52:05,614][INFO ][o.e.n.Node               ] node name [qDUPSsw] derived from node ID [qDUPSswHQVWUbdZLO-knyQ]; set [node.name] to override
[2018-03-23T10:52:05,614][INFO ][o.e.n.Node               ] version[5.6.8], pid[2611], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-03-23T10:52:05,615][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [aggs-matrix-stats]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [ingest-common]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-expression]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-groovy]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-mustache]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-painless]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [parent-join]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [percolator]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [reindex]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [transport-netty3]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [transport-netty4]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] no plugins loaded
[2018-03-23T10:52:08,754][INFO ][o.e.d.DiscoveryModule    ] [qDUPSsw] using discovery type [zen]
[2018-03-23T10:52:09,461][INFO ][o.e.n.Node               ] initialized
[2018-03-23T10:52:09,462][INFO ][o.e.n.Node               ] [qDUPSsw] starting ...
[2018-03-23T10:52:09,662][INFO ][o.e.t.TransportService   ] [qDUPSsw] publish_address {172.16.5.11:9300}, bound_addresses {172.16.5.11:9300}
[2018-03-23T10:52:09,672][INFO ][o.e.b.BootstrapChecks    ] [qDUPSsw] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-03-23T10:52:12,861][INFO ][o.e.c.s.ClusterService   ] [qDUPSsw] detected_master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300}, added {{ueq7c6I}{ueq7c6IVStKlNtZhePNEOg}{sjvFf7A-T9GBRoMWxvmjaw}{172.16.5.13}{172.16.5.13:9300},{SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300},}, reason: zen-disco-receive(from master [master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300} committed version [65]])
[2018-03-23T10:52:12,949][INFO ][o.e.h.n.Netty4HttpServerTransport] [qDUPSsw] publish_address {172.16.5.11:9200}, bound_addresses {172.16.5.11:9200}
[2018-03-23T10:52:12,949][INFO ][o.e.n.Node               ] [qDUPSsw] started

server.log on graylog1

[2018-03-23T10:52:02,442][INFO ][o.e.n.Node               ] [qDUPSsw] closed
[2018-03-23T10:52:05,332][WARN ][o.e.b.Natives            ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna6538669404814768016.tmp: /tmp/jna--1985354563/jna6538669404814768016.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_161]
        at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_161]
        at java.lang.System.load(System.java:1086) ~[?:1.8.0_161]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_161]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_161]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:195) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.8.jar:5.6.8]
[2018-03-23T10:52:05,339][WARN ][o.e.b.Natives            ] cannot check if running as root because JNA is not available
[2018-03-23T10:52:05,339][WARN ][o.e.b.Natives            ] cannot register console handler because JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_AS beacuse JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2018-03-23T10:52:05,451][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-23T10:52:05,604][INFO ][o.e.e.NodeEnvironment    ] [qDUPSsw] using [1] data paths, mounts [[/var (/dev/mapper/centos-var)]], net usable_space [751.8mb], net total_space [1.4gb], spins? [possibly], types [xfs]
[2018-03-23T10:52:05,605][INFO ][o.e.e.NodeEnvironment    ] [qDUPSsw] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-03-23T10:52:05,614][INFO ][o.e.n.Node               ] node name [qDUPSsw] derived from node ID [qDUPSswHQVWUbdZLO-knyQ]; set [node.name] to override
[2018-03-23T10:52:05,614][INFO ][o.e.n.Node               ] version[5.6.8], pid[2611], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-03-23T10:52:05,615][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [aggs-matrix-stats]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [ingest-common]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-expression]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-groovy]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-mustache]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-painless]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [parent-join]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [percolator]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [reindex]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [transport-netty3]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [transport-netty4]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] no plugins loaded
[2018-03-23T10:52:08,754][INFO ][o.e.d.DiscoveryModule    ] [qDUPSsw] using discovery type [zen]
[2018-03-23T10:52:09,461][INFO ][o.e.n.Node               ] initialized
[2018-03-23T10:52:09,462][INFO ][o.e.n.Node               ] [qDUPSsw] starting ...
[2018-03-23T10:52:09,662][INFO ][o.e.t.TransportService   ] [qDUPSsw] publish_address {172.16.5.11:9300}, bound_addresses {172.16.5.11:9300}
[2018-03-23T10:52:09,672][INFO ][o.e.b.BootstrapChecks    ] [qDUPSsw] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-03-23T10:52:12,861][INFO ][o.e.c.s.ClusterService   ] [qDUPSsw] detected_master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300}, added {{ueq7c6I}{ueq7c6IVStKlNtZhePNEOg}{sjvFf7A-T9GBRoMWxvmjaw}{172.16.5.13}{172.16.5.13:9300},{SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300},}, reason: zen-disco-receive(from master [master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300} committed version [65]])
[2018-03-23T10:52:12,949][INFO ][o.e.h.n.Netty4HttpServerTransport] [qDUPSsw] publish_address {172.16.5.11:9200}, bound_addresses {172.16.5.11:9200}
[2018-03-23T10:52:12,949][INFO ][o.e.n.Node               ] [qDUPSsw] started

What Java Version did you run?

Graylog and Elasticsearch do Support Java 8, but not Java 9 in your current used Versions.

[2018-03-23T10:52:05,332][WARN ][o.e.b.Natives ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna–1985354563/jna6538669404814768016.tmp: /tmp/jna–1985354563

and

[2018-03-23T10:52:05,332][WARN ][o.e.b.Natives ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna–1985354563/jna6538669404814768016.tmp: /tmp/jna–1985354563/jna6538669404814768016.tmp: failed to map segment from shared object: Operation not permitted

When I use google with load JNA native it redirects me to https://github.com/elastic/elasticsearch/issues/18406 where a potential fix is suggested ( set -Djava.io.tmpdir ) that might help you to resolve your issue.

[root@graylog1 ~]# which java
/bin/java
[root@graylog1 ~]# java -version
openjdk version “1.8.0_161”
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)
[root@graylog1 ~]#

I think it has something to do with graylog “talking” with elasticsearch. When I configured graylog1 as a stand alone it worked but now with 3 nodes it fails.

I implemented this on the singe node to get it to work :slight_smile:
Graylog not able to connect to Elasticsearch cluster

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.