So, I have GrayLog running in a Docker instance. I’ve successfully tested the ability to send to GELF endpoints over HTTP. Now, I’d like to have it pull all Docker log traffic from other instances (in unrelated containers). I’ve used SigNoz and it does this by default, out of the box.
Can someone point me in the direction of documentation that clearly outlines how to achieve Any/All Docker Containers → Graylog Docker Instance? I’d like to do this without excluding other sources from reading logs.
Also, I’m going to be using this for a few Node JS applications, so if someone has quality configurations that they want to recommend (e.g. Winston/Pino/Bunyan transports to Graylog), I’m definitely interested).
Also logs are normally located here. /var/lib/docker/containers/<container_id>/<container_id>-json.log
EDIT:
Not sure about this statement
By Archive are you refering to long term backups of the whole container? If so, I just create a checkpoit or backup from the node the containers are on , normally through Veeam free version.
Not “archive” – achieve (successfully accomplish ).
One other caveat – I’m using MacOS, so log files are . . .well, I have no idea where, and I’m honestly not sure how Signoz is reading the logs (other than perhaps accessing /var/run/docker.sock/ directly).
Would certainly be nice if this process was a bit more straightforward.
So I was digging around on the internet, I did see Nxlog and Filebeat is able to be installed on MacOS M1 Arm chip, but looks like there were some issues (i.e., 2020).
So I looked here…
If you have a signoz running on a different host then you will have to run a otel-collector to export logs from your host to the host where SigNoz is running.
Actually researching SigNoz , very simialer in the configurations need like Rsyslog/ Filebeat config.
After reading over there documention for SigNoz this can be achieve NOT archived
By using another service to send logs to SigNoz (otel-collector) then shipped out from there to Graylog.
Just different names but the same principle like the rest of the log shippers. TBH SigNoz reminds me of Rsyslog/Rsyslog server.
Eeeeeeesh! Yeah, I don’t necessarily care to go from Graylog to Signoz. Just need to pick one or the other.
Man, who would have thought the idea of streaming/parsing logs could get so convoluted? I could have written my own logging/metrics dashboard app in the few days I’ve spent looking over these options.
I’ve actually got it successfully pulling data via Loki atm. And I can use /var/run/docker.sock on Linux or MacOS. No need for additional drivers or otherwise!
Nah, you were definitely helpful. Sometimes just having a springboard is all you need! Thanks again!
Aaaaaaah, very nice! If you have any quality Vector “remap” or “transform” techniques you’d like to share, I’m all ears.
I’m in the fortunate position that I can generate logs in any “shape” that I like. I just need to be able to plot data points from them at the end of the day. Currently, I’m submitting nested JSON objects, but they’re getting pushed through as strings. I’d like to be able to pipe an entire field (not label) back to JSON and query on the object if at all possible.
Anyway, thanks again, and feel free to spam ideas.