Hello everyone,
I am fairly new to graylog but already took one day of debugging without figuring out why my graylog does not process anymore.
I started of with a Single-Server environment (GL + ES on one server) and ran into the issue that nothing was stored to elastic anymore.
I figured it might be a performance problem (although it does not make sense to me that nothing is handle…with performance issue I would expect a growing backlog but not nothing at all to be saved)
So I extended to a separate ES server.
Server obviously is Throttled as Journal (was extended to 20GB in the hope to find a fix before it would run full) is fully utilized.
Elastic is shown green with no indexer failures
Both servers are run on ubuntu 20.04LTS
- Graylog 4.1.1+27dec96
- VM with 12 Cores 12GB RAM
- has >20GB free space
- JVM Heap is set to 4GB
- proxied via Apache2 (config below)
- only HTTP/HTTPS + GL Input-Ports exposed via firewall
- Elasticsearch 7.10.2
- has 600GB free space
- VM with 8 Cores 8GB RAM
- Only port 9200+9300 exposed via firewall to GL-server
Some Information about my current setup (comments and default values et cetera removed, Hostnames changed):
/etc/graylog/server/server.conf
is_master = true
http_bind_address = 0.0.0.0:9000
http_external_uri = https://graylog.monitoring.domain.local/
elasticsearch_hosts = http://es-server:9200
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
message_journal_max_age = 48h
message_journal_max_size = 20gb
/etc/apache2/sites-enabled/000-default.conf
<VirtualHost *:80>
ServerName graylog.monitoring.domain.local
ServerAlias graylog.monitoring.domain.local
Redirect permanent / https://graylog.monitoring.domain.local/
</VirtualHost>
<VirtualHost *:443>
ServerName graylog.monitoring.domain.local
ProxyRequests Off
SSLEngine on
SSLOptions +StrictRequire
SSLCertificateFile "/etc/ssl/certs/graylog.monitoring.domain.local.crt"
SSLCertificateKeyFile "/etc/ssl/private/graylog.monitoring.domain.local.key"
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
<Location />
RequestHeader set X-Graylog-Server-URL "https://graylog.monitoring.domain.local"
ProxyPass http://127.0.0.1:9000/
ProxyPassReverse http://127.0.0.1:9000/
</Location>
</VirtualHost>
/etc/elasticsearch/elasticsearch.yml
cluster.name: graylog
path.data: /graylog-data/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.type: single-node
curl http://es-server:9200/_cluster/health?pretty (executed from GL-server)
{
"cluster_name" : "graylog",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 20,
"active_shards" : 20,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
GL /api/cluster
{
"327f2801-d3ee-4215-bfe3-36016b482b56": {
"facility": "graylog-server",
"codename": "Noir",
"node_id": "327f2801-d3ee-4215-bfe3-36016b482b56",
"cluster_id": "9751f3c2-24bb-4523-9eb9-61c8902c7aec",
"version": "4.1.1+27dec96",
"started_at": "2021-07-08T20:18:16.969Z",
"hostname": "gl-server",
"lifecycle": "throttled",
"lb_status": "throttled",
"timezone": "Europe/Berlin",
"operating_system": "Linux 5.4.0-77-generic",
"is_processing": true
}
}
Where could/should I look to figure out why nothing is written out?
Nothing is processed even if all inputs are stopped and system has no load at all.
So I assume not a load problem
Thanks for any help
