Incomplete message body after arrival to Graylog

[m_mlk]

1. Describe your incident:

One application generates log entries when it runs OK and when it throws an error:

The following entries have been taken from the local log file:

• When OK, it logs:

Jan 03 15:32:11 foobar-vm application-name[5569]: time=“2023-01-03T15:32:11Z” level=info msg=“http: GET /path/to/application/status 200” duration=0.000050 method=GET path=/path/to/application/status status=200 version=1.0

This is rendered like this (timestamps are taken from different examples):

• When an error occurs, this is logged instead:

Jan 03 15:32:06 foobar-vm application-name[5569]: time=“2023-01-03T15:32:06Z” level=warning msg=“error calling someFunction” error=“Post "https://some-long-url": context deadline exceeded (Client.Timeout exceeded while awaiting headers)” method=someFunction value=100 rv=okhttp/4.9.3 url=“https://some-long-url” version=1.0

This is rendered like this (timestamps are taken from different examples):

As you can see, in both cases, the original messages get extremely shortened on arrival.

2. Describe your environment:

• OS Information: Ubuntu 20.04 LTS

• Package Version: GL 4.3.9, OpenSearch 1.3.6

• Service logs, configurations, and environment variables:

rsyslog client configuration:

$ cat /etc/rsyslog.d/graylog.conf
.;auth,authpriv.none -/var/log/syslog
. @@graylog-server:1515;RSYSLOG_SyslogProtocol23Format

3. What steps have you already taken to try and solve the problem?

I checked the messages at source and compared them with that Graylog shows.
Also read some posts in the forums - unfortunately, they didn’t help.

4. How can the community help?

Is it possible to get the original messages as they actually are?

TIA!

[tmacgbay]

My first guess would be the Input you are using is pre-parsing the data. If you switch the input to a RAW input , there is nothing being done to the data and it likely won’t pick up msg= improperly.

[joe.gross]

@tmacgbay is right. If I had to guess, I would say your syslog is malformed. Try the raw input as he suggested and see if it’s better.

[m_mlk]

Hello @tmacgbay and @joe.gross

Thank you guys for your answers.

Creating a new raw input is not a problem… but I observed that the VMs where the application whose logs aren’t displayed correctly on Graylog also ship other logs:

In green: fully well-formed logs
In red: problem at hand

Both types of messages are ingested into Graylog using the Syslog-TCP input.

Could it be that the application writes a malformed syslog message and therefore Syslog-TCP doesn’t understand it?

TIA

[joe.gross]

“Could it be that the application writes a malformed syslog message and therefore Syslog-TCP doesn’t understand it?:”

Absolutely. If you have logs from the same system coming in correctly, it’s likely the application causing the error.

[m_mlk]

Our developers will love that answer - LOL.

Cheers

[tmacgbay]

Let us know what happens!

[m_mlk]

Enabling JSON formatting in the application for the logs seemed to have solved the issue…
Weird… but at least it’s solved.

Cheers