Hi and thanks for helping out!
I did find this in server.log yesterday and scratched on the surface of it:
022-04-30T11:38:24.638+02:00 WARN [ProxiedResource] Unable to call http://192.168.44.92:9000/api/system on node <d4137266-8358-41f5-9f7e-901f208850a1>: timeout
curl -i http://192.168.44.92:9000/api/system
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="Graylog Server"
X-Graylog-Node-ID: d4137266-8358-41f5-9f7e-901f208850a1
X-Runtime-Microseconds: 350
Content-Length: 0
Nothing superobvious in elastic logs. These are the last few lines.
[2022-04-30T04:09:37,284][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [vmware-all-msg_124/j1RNL7Z7STS9_32l_gk49g] update_mapping [_doc]
[2022-04-30T04:09:39,291][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [sec-winbeats-short_101/HGkC4WqgTnukygoaWYsi_g] update_mapping [_doc]
[2022-04-30T04:09:39,401][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [sec-winbeats-short_101/HGkC4WqgTnukygoaWYsi_g] update_mapping [_doc]
[2022-04-30T04:09:47,283][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [vmware-all-msg_124/j1RNL7Z7STS9_32l_gk49g] update_mapping [_doc]
[2022-04-30T04:09:47,285][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [vmware-all-msg_124/j1RNL7Z7STS9_32l_gk49g] update_mapping [_doc]
[2022-04-30T04:14:53,277][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [vmware-all-msg_124/j1RNL7Z7STS9_32l_gk49g] update_mapping [_doc]
[2022-04-30T08:25:35,280][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [sec-winbeats-short_101/HGkC4WqgTnukygoaWYsi_g] update_mapping [_doc]
[2022-04-30T10:35:01,290][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [sec-winbeats-short_101/HGkC4WqgTnukygoaWYsi_g] update_mapping [_doc]
[2022-04-30T10:35:01,310][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [sec-winbeats-short_101/HGkC4WqgTnukygoaWYsi_g] update_mapping [_doc]
[2022-04-30T10:49:55,282][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [vmware-all-msg_124/j1RNL7Z7STS9_32l_gk49g] update_mapping [_doc]
[2022-04-30T10:50:09,280][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [vmware-all-msg_124/j1RNL7Z7STS9_32l_gk49g] update_mapping [_doc]
[2022-04-30T10:50:09,308][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [vmware-all-msg_124/j1RNL7Z7STS9_32l_gk49g] update_mapping [_doc]
[2022-04-30T10:50:50,283][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [sec-winbeats-short_101/HGkC4WqgTnukygoaWYsi_g] update_mapping [_doc]
[2022-04-30T10:53:54,292][INFO ][o.e.c.m.MetadataMappingService] [m5-logger01.localdomain] [sec-winbeats-short_101/HGkC4WqgTnukygoaWYsi_g] update_mapping [_doc]
Graylog conf. Passwords and such removed.
grep -v ^\# /etc/graylog/server/server.conf | grep .
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxx
root_password_sha2 = xxx
root_timezone = Europe/Stockholm
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.44.92:9000
elasticsearch_hosts = http://127.0.0.1:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://127.0.0.1/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = true
transport_email_hostname = 192.168.xx.xx
transport_email_port = 25
transport_email_use_auth = false
transport_email_subject_prefix = [graylog]
transport_email_from_email =xxx@xxx.xx
proxied_requests_thread_pool_size = 32
Elastic conf:
grep -v ^\# /etc/elasticsearch/elasticsearch.yml | grep .
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
cluster.name: graylog
action.auto_create_index: false
No difference when disabling firewall.
Htop screenshot. Not sure it would paste properly as text.
Thanks again @gsmith