Systemd journal ingestion?

Graylog Central (peer support)
1

I have a very small use case for something like graylog: Power it up; ingest a system’s entire [systemd] journal for search, analysis, and troubleshooting of a specific problem. Shut down graylog when done. Fire it up next time it’s needed and have ingestion pick up from where it last left off.

I simply don’t need a full-time log archiving+management solution but having powerful search features would be nice once in a while.

I could simply ingest journalctl json output but the “pick up where it left off” part is puzzling. I didn’t see a marketplace product specific to systemd-journald so I’m wondering if this kind of thing exists.

My other thought is to have a log agent installed and ready-to-go on the target system (I know these can be intelligent about bookmarking, log rolls, etc) and use it to feed a graylog instance.

Thanks for any advice…

Edit: I may already have something for this.. rsyslog is installed on this system, configured with StateFile="imjournal.state" for cursor bookmarking. I could leverage omfwd with a large disk queue to handle forwarding to an intermittent receiver, infinite retries, and a reasonably long retry interval. I suspect this will work…

2

I have used filebeat with the journald input configuration, and it works great Journald input | Filebeat Reference [8.19] | Elastic

Filebeat is a log shipping agent in the Elasticsearch suite of products – runs as a daemon on the origin host, and is lightweight and reliable in my experience. With the usual package install on linux you have a systemd unit and manage the process with systemctl. You can enable/disable or run in on a schedule or whatever.

It keeps a local metadata store and the default behavior is to “pick up where it left off” – so you get a continuous log stream (eventually) even if flow is interrupted. You can blow away the metadata to re-start from scratch.

This might be a good fit for your use case.

3

Also meant to say, you definitely could do this with rsyslog but based on my experience that’ll be a lot more painful to get working. I’ve used rsyslog at times for various purposes when it’s impossible or difficult to use filebeat or a similar agent.