Journal utilization is too high Error

Graylog Central (peer support)
,
1

Hi,

i installed graylog docker version on ubuntu, and trying to get Windows Events with Winlogbeat . Ubuntu host has 8 GB Ram 4 Core Cpu 256 GB disk, this is what it is look like;

image
image943×174 13.4 KB

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS"

This is graylog conf file

is_master = true
node_id_file = /usr/share/graylog/data/config/node-id
root_username = admin
root_timezone = UTC
bin_dir = /usr/share/graylog/bin
data_dir = /usr/share/graylog/data
plugin_dir = /usr/share/graylog/plugin
elasticsearch_version_probe_attempts = 5
elasticsearch_version_probe_delay = 5s
elasticsearch_connect_timeout = 10s
elasticsearch_socket_timeout = 60s
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = data/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://mongodb/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32
enabled_tls_protocols= TLSv1.2,TLSv1.3
prometheus_exporter_enabled = true
prometheus_exporter_bind_address = 127.0.0.1:9833
transport_email_enabled = true
transport_email_hostname = outbound.mailhop.org
transport_email_port = 587
transport_email_use_auth = true
transport_email_use_tls = true
transport_email_use_ssl = false
transport_email_auth_username = xxxxx
transport_email_auth_password = xxxxxx
transport_email_subject_prefix = [graylog]
transport_email_from_email = graylog@example.com
transport_email_web_interface_url = https://graylog.example.com

Winlogbeat conf file (installed with Graylog sidecar)

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: DESKTOP-LOGPROVIDER
fields.gl2_source_collector: 9591####-####-####-####-fd7#####da54


output.logstash:
   hosts: ["192.168.5.50:1514"] (fake ip, not real)
path:
  data: C:\Program Files\Graylog\sidecar\generated\68665db149a37237d72eaf7e\data
  logs: C:\Program Files\Graylog\sidecar\generated\68665db149a37237d72eaf7e\logs
tags:
 - windows
winlogbeat:
  event_logs:
   - name: Application
     ignore_older: 96h
   - name: System
     ignore_older: 96h
   - name: Security
     ignore_older: 96h
   - name: Setup
     ignore_older: 96h
   - name: ForwardedEvents
     forwarded: true
     ignore_older: 96h
   - name: Microsoft-Windows-Windows Defender/Operational
     ignore_older: 96h
   - name: Microsoft-Windows-Sysmon/Operational
     ignore_older: 96h
   - name: Microsoft-Windows-TerminalServices-LocalSessionManager/Operational
     ignore_older: 96h
   - name: Microsoft-Windows-PowerShell/Operational
     ignore_older: 96h
   - name: windows PowerShell
     ignore_older: 96h

This is the Error i Receive;

Journal utilization is too high
(triggered 3 minutes ago)
Journal utilization is too high and may go over the limit soon. Please verify that your Elasticsearch cluster is healthy and fast enough. You may also want to review your Graylog journal settings and set a higher limit. (Node: 0eef#########################e54fe)

Uncommited messages deleted from journal
(triggered a few seconds ago)
Some messages were deleted from the Graylog journal before they could be written to Elasticsearch. Please verify that your Elasticsearch cluster is healthy and fast enough. You may also want to review your Graylog journal settings and set a higher limit. (Node: 0eef#########################17e54fe)

I tried to read old topics but i didn’t understand what to do, because suggestions are not for docker, and i dont have Elasticsearch confs.

Can you help me please.

Thanks.

2

This is the answer. Your storage is full, and therefore no new logs can be ingested. You can delete old logs to free up space for new logs.

3

Hi,
Thanks for helping, i quit using winlogbeat (produces too much logs) and stopped services on computers and deleted inputs on graylog but couldn’t delete sidecars, i am not sure if i deleted the logs because warning is still there but i am receiving logs form nxlog. How can i completely delete sidecars and winlogbeat logs?

4

I found it,

image
image1307×668 45.2 KB

Need to change Expiration threshold, after 14 days sidecars will be deleted.

5

nope, you need to have a look at your index sets:

  1. go on one of the logs, check on which stream it is (left side of the log)
  2. to on the streams overview. search for that stream and go on the index set of that stream
  3. check the data retention there.