Receiving all Docker traffic to Graylog Docker Instance . . .?

[coder]

Hi All,

So, I have GrayLog running in a Docker instance. I’ve successfully tested the ability to send to GELF endpoints over HTTP. Now, I’d like to have it pull all Docker log traffic from other instances (in unrelated containers). I’ve used SigNoz and it does this by default, out of the box.

Can someone point me in the direction of documentation that clearly outlines how to achieve Any/All Docker Containers → Graylog Docker Instance? I’d like to do this without excluding other sources from reading logs.

Also, I’m going to be using this for a few Node JS applications, so if someone has quality configurations that they want to recommend (e.g. Winston/Pino/Bunyan transports to Graylog), I’m definitely interested).

Best

[gsmith]

Hello && Welcome @coder

I have by using Filebeat /w Graylog Sidecar. Just filebeat would be fine but I perfer to adjust my setting on the Web UI becuase Im lazy

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- type: docker
  containers.ids: 
    -  '*'
  type: log
output.logstash:
   hosts: ["192.168.1.100:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

Container input | Filebeat Reference [8.11] | Elastic

Also logs are normally located here.
/var/lib/docker/containers/<container_id>/<container_id>-json.log

EDIT:
Not sure about this statement

@coder

achieve Any/All Docker Containers → Graylog Docker Instance?

By Archive are you refering to long term backups of the whole container? If so, I just create a checkpoit or backup from the node the containers are on , normally through Veeam free version.

[coder]

Not “archive” – achieve (successfully accomplish ).

One other caveat – I’m using MacOS, so log files are . . .well, I have no idea where, and I’m honestly not sure how Signoz is reading the logs (other than perhaps accessing /var/run/docker.sock/ directly).

Would certainly be nice if this process was a bit more straightforward.

Thanks for the input btw!

[gsmith]

Hey

@coder

Not “archive” – achieve (successfully accomplish ).

Sorry HAHA I just woke up

@coder

One other caveat – I’m using MacOS, so log files are . .

Oh sorry , I dont touch Mac stuff, tbh apple /Docker here you may or may not get an anwser. But ill ask around for ya

[gsmith]

hey

Just an FYI, I found this.

Here.

Filebeat quick start: installation and configuration | Filebeat Reference...

[coder]

All good. Yeah, I’m also on M1 ARM, so there’s an extra

Definitely don’t lose any sleep over it, but any input is appreciated. Thanks again.

[gsmith]

Holy cow man is there anything else we should know?
Nah, I’m all good challenge accepted

[gsmith]

Hey,

@coder

SigNoz

So I was digging around on the internet, I did see Nxlog and Filebeat is able to be installed on MacOS M1 Arm chip, but looks like there were some issues (i.e., 2020).

So I looked here…

Collecting Docker container logs | SigNoz

If you have a signoz running on a different host then you will have to run a otel-collector to export logs from your host to the host where SigNoz is running.

Actually researching SigNoz , very simialer in the configurations need like Rsyslog/ Filebeat config.

Logs | SigNoz

After reading over there documention for SigNoz this can be achieve NOT archived

By using another service to send logs to SigNoz (otel-collector) then shipped out from there to Graylog.

Just different names but the same principle like the rest of the log shippers. TBH SigNoz reminds me of Rsyslog/Rsyslog server.

[coder]

Eeeeeeesh! Yeah, I don’t necessarily care to go from Graylog to Signoz. Just need to pick one or the other.

Man, who would have thought the idea of streaming/parsing logs could get so convoluted? I could have written my own logging/metrics dashboard app in the few days I’ve spent looking over these options.

Appreciate those links. I’ll take a closer look.

[gsmith]

@coder

I don’t necessarily care to go from Graylog to Signoz

I was refering to from Signox to graylog, By using another service to send logs to SigNoz (otel-collector) then to Graylog Im awake now LOL.

otel-collector --> SigNoz --> Graylog

Or

another_log_shipper/s --> Graylog

@coder

Appreciate those links. I’ll take a closer look.

Of cource, sorry I cant be more help.

[coder]

Wow! So I found this GH post about Vector.

I’ve actually got it successfully pulling data via Loki atm. And I can use /var/run/docker.sock on Linux or MacOS. No need for additional drivers or otherwise!

Nah, you were definitely helpful. Sometimes just having a springboard is all you need! Thanks again!

[gsmith]

I use that also

Not only for Graylog metrics, Opensearch, Zabbix, Nextcloud, Bookstack, etc… pretty much with everthing. Good stuff

[coder]

Aaaaaaah, very nice! If you have any quality Vector “remap” or “transform” techniques you’d like to share, I’m all ears.

I’m in the fortunate position that I can generate logs in any “shape” that I like. I just need to be able to plot data points from them at the end of the day. Currently, I’m submitting nested JSON objects, but they’re getting pushed through as strings. I’d like to be able to pipe an entire field (not label) back to JSON and query on the object if at all possible.

Anyway, thanks again, and feel free to spam ideas.