Fairly new Graylog install on FreeBSD; graylog 2.3.3 / elasticsearch 2.4.4; I have a couple basic inputs running and tcpdump shows data flowing from clients to the graylog server on the expected ports. However, any attempts to search events gives me a “Loading…” message and nothing happens. The ES index is growing, but I can’t seem to get any data to display.
What’s the configuration of Graylog?
What’s the configuration of Elasticsearch?
What’s in the logs of your Graylog and Elasticsearch nodes?
gl server.conf
is_master = true
node_id_file = /var/graylog/server/node-id
plugin_dir = /usr/local/share/graylog/plugin
rest_listen_uri = http://10.31.0.60:12900/
elasticsearch_config_file = /usr/local/etc/elasticsearch/elasticsearch.yml
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog2
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_cluster_name = graylog2
elasticsearch_node_name = graylog2-server
elasticsearch_transport_tcp_port = 9350
elasticsearch_discovery_zen_ping_multicast_enabled = false
elasticsearch_discovery_zen_ping_unicast_hosts = 10.31.0.60:9300
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/db/graylog/journal
dead_letters_enabled = false
lb_recognition_period_seconds = 3
mongodb_useauth = false
mongodb_host = 127.0.0.1
mongodb_database = graylog2
mongodb_port = 27017
es yml
cluster.name: graylog2
node.name: graylog2-server
path.data: /var/db/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 10.31.0.60
http.port: 9200
discovery.zone.ping.multicast.enable: false
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300", "10.31.0.60:9300"]
node.max_local_storage_nodes: 1
es log
[2017-07-07 15:48:19,760][INFO ][cluster.metadata ] [graylog2-server] [graylog2_4] update_mapping [message]
[2017-07-07 15:48:23,741][INFO ][cluster.metadata ] [graylog2-server] [graylog2_4] update_mapping [message]
[2017-07-07 16:26:59,740][INFO ][cluster.metadata ] [graylog2-server] [graylog2_4] update_mapping [message]
[2017-07-07 20:48:09,739][INFO ][cluster.metadata ] [graylog2-server] [graylog2_4] update_mapping [message]
Errors in GL log
2017-07-07 15:07:44,515 ERROR o.g.r.r.s.i.IndicesResource [rest-worker-5] Index [graylog2_0] not found.
2017-07-07 15:07:44,516 ERROR o.g.r.r.s.i.IndicesResource [rest-worker-7] Index [graylog2_0] not found.
2017-07-07 15:07:52,907 ERROR o.g.r.r.s.i.IndicesResource [rest-worker-5] Index [graylog2_1] not found.
2017-07-07 15:08:00,930 ERROR o.g.r.r.s.i.IndicesResource [rest-worker-2] Index [graylog2_2] not found.
2017-07-07 15:08:34,483 ERROR o.g.r.r.s.i.IndicesResource [rest-worker-12] Index [graylog2_3] not found.
2017-07-07 15:08:53,786 ERROR o.g.s.j.SystemJobManager [systemjob-executor-0] Unhandled error while running SystemJob <a60ce420-6347-11e7-b826-00237d96a2b8> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
This is (with 99.99999% certainty) wrong. Please read the annotated configuration file and the docs:
http://docs.graylog.org/en/2.2/pages/configuration/elasticsearch.html
Additionally, the name of the standalone Elasticsearch node(s) and the embedded Elasticsearch node in Graylog shouldn’t be identical to make identifying each node easier (see elasticsearch_node_name and node.name settings respectively).
In general, you have quite a few deprecated settings in your Graylog configuration file. You should probably start with the matching version of the annotated configuration file and only change the required settings: graylog2-server/misc/graylog.conf at 2.2.3 · Graylog2/graylog2-server · GitHub
root@secdev1:/var/log/graylog # ls /usr/local/etc/elasticsearch/
elasticsearch.yml elasticsearch.yml.sample logging.yml logging.yml.sample
So the above has been determined to be fine. 0.000001% FTW
I grabbed the updated graylog config from github and transferred the necessary bits to it, to be honest there seemed to be very few differences save a couple lines. I copied over the important parts of the previous config and restarted and got things working.
It’s not the path that’s wrong, it’s the fact that it’s the configuration file for your (standalone) Elasticsearch node, not for the embedded Elasticsearch node in Graylog.
Would the embedded ES node run under the graylog user, or elasticsearch user? I’m trying to find evidence of it, but it doesn’t seem to exist. Graylog is working perfectly right now, but if there is something fundamentally incorrect about this setup I would like to address it before I spin up another one.
Any pointers on how to locate the elusive embedded ES node?
Searched the entire filesystems for another elasticsearch.yml and came up empty.
It’s part of the Graylog process, so it would run in the context of the system user running that process.
One of these?
graylog java 53065 89 tcp4 10.31.0.60:9350 :
graylog java 53065 91 tcp4 10.31.0.60:9000 :
graylog java 53065 120 tcp4 10.31.0.60:9443 :
graylog java 53065 194 udp4 *:5140 :
graylog java 53065 204 udp4 *:12201 :
graylog java 53065 205 tcp4 *:5043 :
Yes, probably.
I’m not the administrator of that system, you know? 
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.