I have Graylog 2.3.xx and 2.4.xx in my system for logging separate entities, and before I do a long overdue upgrade, I need to solve some issues. There are in both a difficulty to fetch messages from RabbitMQ queue in any other format than raw input. Those events are from Linux machines and in beats. Similar configuration from Windows registries with Winlogbeat gets consumed nicely as GELF AMQP.
Have I missed something important?
Second problem stems from parsing said raw input message. I end up with a “message”:“which includes everything sent”. I don’t care of the rest but I want to find a value for a source withing that message.
How do I access a sub field “source”?
Thank you in advance. 2020-01-19T22:00:00Z #graylog
Beats is only supported via native protocol and not via kafka or amqp. Depending how you ingest the RAW is the only option So you need to parse that yourself.
The second question is more about how you work with that. Means processing pipeplines or extractors? What did you want todo with that?
I have a service which is load balanced on many hosts but which are sending their log events using the same route. At a receiving end there is Graylog which should set a source field value for each event from nested JSON in message field. I hope I don’t need a lot of grokking at this point and there is a simple way to copy a field host or a hostname to a new value of a source field.
At the moment, I am playing around with pipelines, their rules and trying to find the neat way to replace a value of a field.
a simple json parser in the processing pipeline could be like:
rule "extract-json"
when
starts_with(to_string($message.message), "{") && ends_with(to_string($message.message), "}")
then
let json = parse_json(to_string($message.message));
let map = to_map(json);
set_fields(map);
end
If you have more specific needs, the community will be able to help you if you can’t continue yourself.
