Nothing showing in any streams since OS patching

Graylog Central (peer support)
1

**1. we patched our CentOS operating systems, and now graylog isn’t showing any messages

we are getting lots of activity, lots of messages In, but none out.

-35,381,135,306 unprocessed messages are currently in the journal, in 1 segments.
165 messages have been appended in the last second, 0 messages have been read in the last second.

2. Describe your environment:
CentOS 8

  • Package Version:
    gray log - 4.3.7
    elasticsearch - 6.8.23

**3. checked config in graylog, rebooted servers

4. How can the community help?
further troubleshooting available?
any known issues others have had with this problem

2

Hello @gemidriver and welcome

Sorry to hear your having issues, looks journal is filling up a lot. I would check your elasticsearch status and/or logs from Elasticsearch & Graylog.

3

elasticsearch status looks good from graylog
Elasticsearch cluster graylog is green. Shards: 392 active, 0 initializing, 0 relocating, 0 unassigned,

how would we delete the journal?
The journal contains -35,770,843,964 unprocessed messages in 2 segments. 156 messages appended, 0 messages read in the last second
we are running on CentOS 8

4

hello,

I must warn you its not the proper way to handle this situation.

You should see you journal location under System/Node

image
image1173×232 22.9 KB

  • Stop graylog service

  • Delete everything in the directory “Journal”

  • Start graylog service back up

EDIT: Since Elasticsearch is not indexing those logs, your probably going to run into the same issue.

5

@gemidriver

I noticed that was a negative number, found out why here.

For a reference in this forum.

And here

6

Disk Journal

Incoming messages are written to the disk journal to ensure they are kept safe in case of a server failure. The journal also helps keeping Graylog working if any of the outputs is too slow to keep up with the message rate or whenever there is a peak in incoming messages. It makes sure that Graylog does not buffer all of those messages in main memory and avoids overly long garbage collection pauses that way.

Configuration

Path:
file:///var/lib/graylog-server/journal/
Earliest entry:
a few seconds ago
Maximum size:
10.0GiB
Maximum age:
12 hours 0 minutes
Flush policy:
Every 1,000,000 messages or 1 minute 0 seconds

Utilization

0.07%

-35,769,212,100 unprocessed messages are currently in the journal, in 1 segments.
307 messages have been appended in the last second, 0 messages have been read in the last second.

7

performed the tasks - all working again
cheers

2 Likes
8

awesome-yes-will-ferrell

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.