Incomplete message body after arrival to Graylog

1. Describe your incident:

One application generates log entries when it runs OK and when it throws an error:

The following entries have been taken from the local log file:

  • When OK, it logs:

Jan 03 15:32:11 foobar-vm application-name[5569]: time=“2023-01-03T15:32:11Z” level=info msg=“http: GET /path/to/application/status 200” duration=0.000050 method=GET path=/path/to/application/status status=200 version=1.0

This is rendered like this (timestamps are taken from different examples):

  • When an error occurs, this is logged instead:

Jan 03 15:32:06 foobar-vm application-name[5569]: time=“2023-01-03T15:32:06Z” level=warning msg=“error calling someFunction” error=“Post "https://some-long-url": context deadline exceeded (Client.Timeout exceeded while awaiting headers)” method=someFunction value=100 rv=okhttp/4.9.3 url=“https://some-long-url” version=1.0

This is rendered like this (timestamps are taken from different examples):

As you can see, in both cases, the original messages get extremely shortened on arrival.

2. Describe your environment:

  • OS Information: Ubuntu 20.04 LTS

  • Package Version: GL 4.3.9, OpenSearch 1.3.6

  • Service logs, configurations, and environment variables:

rsyslog client configuration:

$ cat /etc/rsyslog.d/graylog.conf
.;auth,authpriv.none -/var/log/syslog
. @@graylog-server:1515;RSYSLOG_SyslogProtocol23Format

3. What steps have you already taken to try and solve the problem?

I checked the messages at source and compared them with that Graylog shows.
Also read some posts in the forums - unfortunately, they didn’t help.

4. How can the community help?

Is it possible to get the original messages as they actually are?

TIA!

My first guess would be the Input you are using is pre-parsing the data. If you switch the input to a RAW input , there is nothing being done to the data and it likely won’t pick up msg= improperly.

3 Likes

@tmacgbay is right. If I had to guess, I would say your syslog is malformed. Try the raw input as he suggested and see if it’s better.

Hello @tmacgbay and @chris.black

Thank you guys for your answers.

Creating a new raw input is not a problem… but I observed that the VMs where the application whose logs aren’t displayed correctly on Graylog also ship other logs:

In green: fully well-formed logs
In red: problem at hand

Both types of messages are ingested into Graylog using the Syslog-TCP input.

Could it be that the application writes a malformed syslog message and therefore Syslog-TCP doesn’t understand it?

TIA

“Could it be that the application writes a malformed syslog message and therefore Syslog-TCP doesn’t understand it?:”

Absolutely. If you have logs from the same system coming in correctly, it’s likely the application causing the error.

1 Like

Our developers will love that answer - LOL.

Cheers

Let us know what happens!

1 Like

Enabling JSON formatting in the application for the logs seemed to have solved the issue…
Weird… but at least it’s solved.

Cheers

2 Likes