Graylog 6.1.6. Input configured as RAW (because Graylog in Syslog Input automatically creates not necessary hundreds, if not thousands, of fields from Fortigate logs). In Search mode I see 3 fields - message, source and timestamp. The message field contains, of course, the entire original log. The problem starts in the pipeline, where I want to create a couple of fields with data extracted from the message. Any reference to $message.message causes the logs to disappear and not be saved. Here’s an example of even such a simple rule that causes the logs to no longer be available, not to mention more complex pipeline rules
rule “Reset Message”
when true
then
set_field(“new_message”, to_string($message.message));
end
It’s strange because in Search mode, I can search using the message: field, but in the pipeline, it seems like that field doesn’t exist (?). Any ideas on how to diagnose this? Alternatively, I could go back to the Syslog UDP Input, but I would have to remove all those fields and leave only the ones I need—which I don’t know how to do (because I’m not going to manually enter each field name in the pipeline (just to remind you, hundreds of them are being created)).
Are there any errors in server.log or the OpenSearch logs. Most often when messages simply disappear it is because they cannot be stored properly into OpenSearch for some reason.
The awareness that something might be wrong with the message field kept bothering me all night, so first thing in the morning — as a test — I set a condition
in the pipeline to check whether such a field exists. It turned out that the condition was not met because the rest of the pipeline was being ignored.
So I figured something might be wrong with the input itself. (I must admit that earlier, I enabled RAW mode before disabling Syslog mode—it only lasted a few seconds,
and I’m not sure if that matters, but for the sake of transparency, I’m mentioning it. The inputs were on the same port, and I didn’t restart graylog-server.)
Therefore, I removed the RAW input from the configuration and redefined it. And bingo—the problem disappeared! The message field is now being processed correctly,
and references to $message.message no longer cause logs to vanish.
Thank you very much for your interest and for sharing the valuable insight that:
“Most often when messages simply disappear, it is because they cannot be stored properly into OpenSearch for some reason.”
Maybe my case will save someone a few hours in the future.
— as a test — I set a condition