Could not load field information

unixinthebox · March 23, 2018, 4:38pm

I have been struggling with this for 3 weeks now but I am unable to get the solution. Could someone please help with this ?
I have 3 nodes all of which are green in elasticsearch.

graylog1. server.conf

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxxxxxxxxx
root_username = sdpadmin
root_password_sha2 = xxxxxxxxxx
root_timezone = America/New_York
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://10.0.103.11:9000/api/
rest_transport_uri = http://10.0.103.11:9000/api/
web_listen_uri = http://10.0.103.11:9000/
elasticsearch_hosts = http://graylog1.philasd.net:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://user:password@172.16.5.11:27017,172.16.5.13:27017,172.16.5.14:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

elasticsearch.yml on graylog1

cluster.name: graylog
discovery.zen.ping.unicast.hosts: ["172.16.5.11", "172.16.5.13","172.16.5.14"]
discovery.zen.minimum_master_nodes: 1
node.master: true
network.host: 172.16.5.11
http.bind_host: 172.16.5.11
http.publish_host: 172.16.5.11
bootstrap.system_call_filter: false

graylog.log on graylog1

[2018-03-23T10:52:02,442][INFO ][o.e.n.Node               ] [qDUPSsw] closed
[2018-03-23T10:52:05,332][WARN ][o.e.b.Natives            ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna6538669404814768016.tmp: /tmp/jna--1985354563/jna6538669404814768016.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_161]
        at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_161]
        at java.lang.System.load(System.java:1086) ~[?:1.8.0_161]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_161]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_161]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:195) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.8.jar:5.6.8]
[2018-03-23T10:52:05,339][WARN ][o.e.b.Natives            ] cannot check if running as root because JNA is not available
[2018-03-23T10:52:05,339][WARN ][o.e.b.Natives            ] cannot register console handler because JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_AS beacuse JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2018-03-23T10:52:05,451][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-23T10:52:05,604][INFO ][o.e.e.NodeEnvironment    ] [qDUPSsw] using [1] data paths, mounts [[/var (/dev/mapper/centos-var)]], net usable_space [751.8mb], net total_space [1.4gb], spins? [possibly], types [xfs]
[2018-03-23T10:52:05,605][INFO ][o.e.e.NodeEnvironment    ] [qDUPSsw] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-03-23T10:52:05,614][INFO ][o.e.n.Node               ] node name [qDUPSsw] derived from node ID [qDUPSswHQVWUbdZLO-knyQ]; set [node.name] to override
[2018-03-23T10:52:05,614][INFO ][o.e.n.Node               ] version[5.6.8], pid[2611], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-03-23T10:52:05,615][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [aggs-matrix-stats]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [ingest-common]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-expression]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-groovy]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-mustache]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-painless]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [parent-join]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [percolator]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [reindex]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [transport-netty3]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [transport-netty4]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] no plugins loaded
[2018-03-23T10:52:08,754][INFO ][o.e.d.DiscoveryModule    ] [qDUPSsw] using discovery type [zen]
[2018-03-23T10:52:09,461][INFO ][o.e.n.Node               ] initialized
[2018-03-23T10:52:09,462][INFO ][o.e.n.Node               ] [qDUPSsw] starting ...
[2018-03-23T10:52:09,662][INFO ][o.e.t.TransportService   ] [qDUPSsw] publish_address {172.16.5.11:9300}, bound_addresses {172.16.5.11:9300}
[2018-03-23T10:52:09,672][INFO ][o.e.b.BootstrapChecks    ] [qDUPSsw] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-03-23T10:52:12,861][INFO ][o.e.c.s.ClusterService   ] [qDUPSsw] detected_master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300}, added {{ueq7c6I}{ueq7c6IVStKlNtZhePNEOg}{sjvFf7A-T9GBRoMWxvmjaw}{172.16.5.13}{172.16.5.13:9300},{SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300},}, reason: zen-disco-receive(from master [master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300} committed version [65]])
[2018-03-23T10:52:12,949][INFO ][o.e.h.n.Netty4HttpServerTransport] [qDUPSsw] publish_address {172.16.5.11:9200}, bound_addresses {172.16.5.11:9200}
[2018-03-23T10:52:12,949][INFO ][o.e.n.Node               ] [qDUPSsw] started

server.log on graylog1

[2018-03-23T10:52:02,442][INFO ][o.e.n.Node               ] [qDUPSsw] closed
[2018-03-23T10:52:05,332][WARN ][o.e.b.Natives            ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna6538669404814768016.tmp: /tmp/jna--1985354563/jna6538669404814768016.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_161]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_161]
        at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_161]
        at java.lang.System.load(System.java:1086) ~[?:1.8.0_161]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_161]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_161]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:195) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.8.jar:5.6.8]
[2018-03-23T10:52:05,339][WARN ][o.e.b.Natives            ] cannot check if running as root because JNA is not available
[2018-03-23T10:52:05,339][WARN ][o.e.b.Natives            ] cannot register console handler because JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_AS beacuse JNA is not available
[2018-03-23T10:52:05,340][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2018-03-23T10:52:05,451][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-23T10:52:05,604][INFO ][o.e.e.NodeEnvironment    ] [qDUPSsw] using [1] data paths, mounts [[/var (/dev/mapper/centos-var)]], net usable_space [751.8mb], net total_space [1.4gb], spins? [possibly], types [xfs]
[2018-03-23T10:52:05,605][INFO ][o.e.e.NodeEnvironment    ] [qDUPSsw] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-03-23T10:52:05,614][INFO ][o.e.n.Node               ] node name [qDUPSsw] derived from node ID [qDUPSswHQVWUbdZLO-knyQ]; set [node.name] to override
[2018-03-23T10:52:05,614][INFO ][o.e.n.Node               ] version[5.6.8], pid[2611], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-03-23T10:52:05,615][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [aggs-matrix-stats]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [ingest-common]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-expression]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-groovy]
[2018-03-23T10:52:06,763][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-mustache]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [lang-painless]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [parent-join]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [percolator]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [reindex]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [transport-netty3]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] loaded module [transport-netty4]
[2018-03-23T10:52:06,764][INFO ][o.e.p.PluginsService     ] [qDUPSsw] no plugins loaded
[2018-03-23T10:52:08,754][INFO ][o.e.d.DiscoveryModule    ] [qDUPSsw] using discovery type [zen]
[2018-03-23T10:52:09,461][INFO ][o.e.n.Node               ] initialized
[2018-03-23T10:52:09,462][INFO ][o.e.n.Node               ] [qDUPSsw] starting ...
[2018-03-23T10:52:09,662][INFO ][o.e.t.TransportService   ] [qDUPSsw] publish_address {172.16.5.11:9300}, bound_addresses {172.16.5.11:9300}
[2018-03-23T10:52:09,672][INFO ][o.e.b.BootstrapChecks    ] [qDUPSsw] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-03-23T10:52:12,861][INFO ][o.e.c.s.ClusterService   ] [qDUPSsw] detected_master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300}, added {{ueq7c6I}{ueq7c6IVStKlNtZhePNEOg}{sjvFf7A-T9GBRoMWxvmjaw}{172.16.5.13}{172.16.5.13:9300},{SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300},}, reason: zen-disco-receive(from master [master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{EJ5QXHFoTL2cjtMMgpmi8g}{172.16.5.14}{172.16.5.14:9300} committed version [65]])
[2018-03-23T10:52:12,949][INFO ][o.e.h.n.Netty4HttpServerTransport] [qDUPSsw] publish_address {172.16.5.11:9200}, bound_addresses {172.16.5.11:9200}
[2018-03-23T10:52:12,949][INFO ][o.e.n.Node               ] [qDUPSsw] started

jan · March 23, 2018, 4:47pm

What Java Version did you run?

Graylog and Elasticsearch do Support Java 8, but not Java 9 in your current used Versions.

[2018-03-23T10:52:05,332][WARN ][o.e.b.Natives ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna–1985354563/jna6538669404814768016.tmp: /tmp/jna–1985354563

and

[2018-03-23T10:52:05,332][WARN ][o.e.b.Natives ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna–1985354563/jna6538669404814768016.tmp: /tmp/jna–1985354563/jna6538669404814768016.tmp: failed to map segment from shared object: Operation not permitted

When I use google with load JNA native it redirects me to https://github.com/elastic/elasticsearch/issues/18406 where a potential fix is suggested ( set -Djava.io.tmpdir ) that might help you to resolve your issue.

unixinthebox · March 23, 2018, 6:09pm

[root@graylog1 ~]# which java
/bin/java
[root@graylog1 ~]# java -version
openjdk version “1.8.0_161”
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)
[root@graylog1 ~]#

unixinthebox · March 23, 2018, 6:12pm

I think it has something to do with graylog “talking” with elasticsearch. When I configured graylog1 as a stand alone it worked but now with 3 nodes it fails.

I implemented this on the singe node to get it to work
Graylog not able to connect to Elasticsearch cluster

system · April 6, 2018, 6:12pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Could not load field information. Loading field information failed Graylog Central (peer support)	2	1297	June 14, 2019
Graylog could not load field information: 500 Graylog Central (peer support)	3	1943	May 24, 2019
Could not load field Graylog Central (peer support)	9	6380	March 13, 2018
Graylog web interface problem! Graylog Central (peer support)	12	4790	February 19, 2019
Can't Connect to ElasticSearch? Graylog Central (peer support)	18	4964	August 29, 2017

Could not load field information

graylog1. server.conf

elasticsearch.yml on graylog1

graylog.log on graylog1

server.log on graylog1

Related topics