Fairly new Graylog install on FreeBSD; graylog 2.3.3 / elasticsearch 2.4.4; I have a couple basic inputs running and tcpdump shows data flowing from clients to the graylog server on the expected ports. However, any attempts to search events gives me a “Loading…” message and nothing happens. The ES index is growing, but I can’t seem to get any data to display.
What’s the configuration of Graylog?
What’s the configuration of Elasticsearch?
What’s in the logs of your Graylog and Elasticsearch nodes?
is_master = true node_id_file = /var/graylog/server/node-id plugin_dir = /usr/local/share/graylog/plugin rest_listen_uri = http://10.31.0.60:12900/ elasticsearch_config_file = /usr/local/etc/elasticsearch/elasticsearch.yml rotation_strategy = count elasticsearch_max_docs_per_index = 20000000 elasticsearch_max_number_of_indices = 20 retention_strategy = delete elasticsearch_shards = 4 elasticsearch_replicas = 0 elasticsearch_index_prefix = graylog2 allow_leading_wildcard_searches = false allow_highlighting = false elasticsearch_cluster_name = graylog2 elasticsearch_node_name = graylog2-server elasticsearch_transport_tcp_port = 9350 elasticsearch_discovery_zen_ping_multicast_enabled = false elasticsearch_discovery_zen_ping_unicast_hosts = 10.31.0.60:9300 elasticsearch_analyzer = standard output_batch_size = 500 output_flush_interval = 1 output_fault_count_threshold = 5 output_fault_penalty_seconds = 30 processbuffer_processors = 5 outputbuffer_processors = 3 processor_wait_strategy = blocking ring_size = 65536 inputbuffer_ring_size = 65536 inputbuffer_processors = 2 inputbuffer_wait_strategy = blocking message_journal_enabled = true message_journal_dir = /var/db/graylog/journal dead_letters_enabled = false lb_recognition_period_seconds = 3 mongodb_useauth = false mongodb_host = 127.0.0.1 mongodb_database = graylog2 mongodb_port = 27017
cluster.name: graylog2 node.name: graylog2-server path.data: /var/db/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory_lock: true network.host: 10.31.0.60 http.port: 9200 discovery.zone.ping.multicast.enable: false discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300", "10.31.0.60:9300"] node.max_local_storage_nodes: 1
[2017-07-07 15:48:19,760][INFO ][cluster.metadata ] [graylog2-server] [graylog2_4] update_mapping [message] [2017-07-07 15:48:23,741][INFO ][cluster.metadata ] [graylog2-server] [graylog2_4] update_mapping [message] [2017-07-07 16:26:59,740][INFO ][cluster.metadata ] [graylog2-server] [graylog2_4] update_mapping [message] [2017-07-07 20:48:09,739][INFO ][cluster.metadata ] [graylog2-server] [graylog2_4] update_mapping [message]
Errors in GL log
2017-07-07 15:07:44,515 ERROR o.g.r.r.s.i.IndicesResource [rest-worker-5] Index [graylog2_0] not found. 2017-07-07 15:07:44,516 ERROR o.g.r.r.s.i.IndicesResource [rest-worker-7] Index [graylog2_0] not found. 2017-07-07 15:07:52,907 ERROR o.g.r.r.s.i.IndicesResource [rest-worker-5] Index [graylog2_1] not found. 2017-07-07 15:08:00,930 ERROR o.g.r.r.s.i.IndicesResource [rest-worker-2] Index [graylog2_2] not found. 2017-07-07 15:08:34,483 ERROR o.g.r.r.s.i.IndicesResource [rest-worker-12] Index [graylog2_3] not found. 2017-07-07 15:08:53,786 ERROR o.g.s.j.SystemJobManager [systemjob-executor-0] Unhandled error while running SystemJob <a60ce420-6347-11e7-b826-00237d96a2b8> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
This is (with 99.99999% certainty) wrong. Please read the annotated configuration file and the docs:
Additionally, the name of the standalone Elasticsearch node(s) and the embedded Elasticsearch node in Graylog shouldn’t be identical to make identifying each node easier (see
node.name settings respectively).
In general, you have quite a few deprecated settings in your Graylog configuration file. You should probably start with the matching version of the annotated configuration file and only change the required settings: https://github.com/Graylog2/graylog2-server/blob/2.2.3/misc/graylog.conf
root@secdev1:/var/log/graylog # ls /usr/local/etc/elasticsearch/
elasticsearch.yml elasticsearch.yml.sample logging.yml logging.yml.sample
So the above has been determined to be fine. 0.000001% FTW
I grabbed the updated graylog config from github and transferred the necessary bits to it, to be honest there seemed to be very few differences save a couple lines. I copied over the important parts of the previous config and restarted and got things working.
It’s not the path that’s wrong, it’s the fact that it’s the configuration file for your (standalone) Elasticsearch node, not for the embedded Elasticsearch node in Graylog.
Would the embedded ES node run under the graylog user, or elasticsearch user? I’m trying to find evidence of it, but it doesn’t seem to exist. Graylog is working perfectly right now, but if there is something fundamentally incorrect about this setup I would like to address it before I spin up another one.
Any pointers on how to locate the elusive embedded ES node?
Searched the entire filesystems for another elasticsearch.yml and came up empty.
It’s part of the Graylog process, so it would run in the context of the system user running that process.
One of these?
graylog java 53065 89 tcp4 10.31.0.60:9350 :
graylog java 53065 91 tcp4 10.31.0.60:9000 :
graylog java 53065 120 tcp4 10.31.0.60:9443 :
graylog java 53065 194 udp4 *:5140 :
graylog java 53065 204 udp4 *:12201 :
graylog java 53065 205 tcp4 *:5043 :
I’m not the administrator of that system, you know?
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.