Internal logs filling after update to 2.3.1

Graylog Central (peer support)
1

Basically I have my logs filled by this issue: Has empty mandatory "short_message" field for OVA 2.3.1

What’s the solution? It’s cute to know the GELF is strict and so on but what is the solution so I stop seeing 50msg/s about this all the time… also it seems to just drop the messages and fill the process buffer so this is a problem.

2

Just as my comment in the referenced topic says:

Messages like these were silently dropped before Graylog 2.3.0 and now at least trigger a log entry.

In short, you have to fix (or update) the GELF client sending these invalid messages.

1 Like
3

Yes, I understand you initial comment but didn’t understand the implications, which you just clarified :slight_smile:
Unfortunately I believe I have to go thru ALL my clients to fix this… sigh

As an example of a typical nxlog.conf output settings:

<Output 578f97f40ae2f10b1139b093>
Module om_udp
Host 192.168.20.210
Port 5441
OutputType  GELF
Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
Exec $gl2_source_collector = '2e87ac7d-eeba-45a6-9ac4-5e56e8b9cd5b';
Exec $Hostname = hostname_fqdn();
</Output>

What should I do with the ‘$short_message’ line? remove it altogether?

4

It looks like $raw_event (and then ultimately $short_message) is blank.

Make sure that the short_message (or the message) field always contains some non-blank string.

You can see the validation logic for GELF messages in Graylog 2.3.1 here:

5

Thanks for the detailed information. However I don’t understand the design choice of throwing an exception rather than simply letting the field be an empty string.

Nonetheless, I now have to go thru my 200+ clients, for each output, and see which has an empty ‘$short_message’ and replace/remove it. At least if the exception included the Output ID it would help me narrow down which clients/output I have to investigate.

With this issue, the new ES 5.5 custom mappings to adjust and the etcd update issue, our team decided to skip version 2.3 altogether and stick to 2.2.3 which works just fine.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.