Journal utilization is too high - process buffer 100%

[gsmith]

Hello,

@Chase

Again, the weird thing is that once we reboot, all messages that have built up are then flushed out.

I assume your taking about your Journal? If so not sure why your server would do that. The journal was created for this purpose if Graylog crashed. By rebooting you mimic a crash.

@Chase

what’s weird is instead when we hit that mark, is that it will never go back to its normal performance without a reboot

I assume there is a wait time that is given before the next reboot? Basically not just rebooting the server randomly and giving time for Graylog to catch up?

@Chase

is that I can shut down all 3 services (elasticsearch, graylog, and mongodb), and then start them again, and performance is still degraded.

Is ulimit properly configured?

@Chase

As far as elasticsearch goes, we do have error messages in the logs, however, most of these are related to our desktop environment, and is saying that there is over a 1000 fields.

Seams that this problem has been around before.

• Problems with limit of total fields greater than 1000

I looked over your other posts and they all look similar, Either to many fields, Output buffer fills up, Processor buffer files up or Journal is filling up.

• Graylog high journal utilization is too high

• Graylog high journaling and low output rate

These are all pointing to your elasticsearch/configurations but looking at your ES & GL configuration files comparing it to mine, I really don’t see anything sticking out that I can Identify the source of the issue.

@Chase

reason=failed to parse field [Message1] of type [date] in document

@Chase

failed to parse date field [Power.EnergyEstimationEngine.Wifi.ppkg] with format

@Chase

strict_date_optional_time||epoch_millis]]]; nested: ElasticsearchException[Elasticsearch exception [type=date_time_parse_exception, reason=Failed to parse with all enclosed parsers]];

Seams Elasticsearch is having hard time with the message coming in and perhaps Indexing them. I would really look into this. It maybe the source or part of the issue.

Some Ideas:

• Perhaps look into GL garbage collection this could have an impacted on performance.
• Not sure how your ingesting log/s or shipping them. Perhaps, adjust your Log shippers to send the minimal amount of logs.
• Insure the correct input is used for each device sending log to Graylog.
• Ensure the Elasticsearch version is no greater the 7.10

If none of that works, I would start small for example:

• Input Syslog UDP for my Windows devices.
• Input Raw/Plaintext for my network work devices.

Double check other logs on this system /var/log maybe you can get a clue or some type of direction on why this issue is happening.