RHEL 7, Graylog 2.3 and 2.4: only raw input for linux beats and its parsing

I have Graylog 2.3.xx and 2.4.xx in my system for logging separate entities, and before I do a long overdue upgrade, I need to solve some issues. There are in both a difficulty to fetch messages from RabbitMQ queue in any other format than raw input. Those events are from Linux machines and in beats. Similar configuration from Windows registries with Winlogbeat gets consumed nicely as GELF AMQP.

Have I missed something important?

Second problem stems from parsing said raw input message. I end up with a “message”:“which includes everything sent”. I don’t care of the rest but I want to find a value for a source withing that message.

How do I access a sub field “source”?

Thank you in advance.

he @viutilo

Beats is only supported via native protocol and not via kafka or amqp. Depending how you ingest the RAW is the only option So you need to parse that yourself.

The second question is more about how you work with that. Means processing pipeplines or extractors? What did you want todo with that?

Thank you, that first was as I feared. :slightly_smiling_face:

I have a service which is load balanced on many hosts but which are sending their log events using the same route. At a receiving end there is Graylog which should set a source field value for each event from nested JSON in message field. I hope I don’t need a lot of grokking at this point and there is a simple way to copy a field host or a hostname to a new value of a source field.

At the moment, I am playing around with pipelines, their rules and trying to find the neat way to replace a value of a field.

Please send some examples, what do you want to achive.

Anyway, check documentation about JSON parsing, using extractors or pipeline rules first:

a simple json parser in the processing pipeline could be like:

rule "extract-json"
    starts_with(to_string($message.message), "{") && ends_with(to_string($message.message), "}")
    let json = parse_json(to_string($message.message));
    let map = to_map(json);

If you have more specific needs, the community will be able to help you if you can’t continue yourself.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.