Receiving all Docker traffic to Graylog Docker Instance . . .?

Hi All,

So, I have GrayLog running in a Docker instance. I’ve successfully tested the ability to send to GELF endpoints over HTTP. Now, I’d like to have it pull all Docker log traffic from other instances (in unrelated containers). I’ve used SigNoz and it does this by default, out of the box.

Can someone point me in the direction of documentation that clearly outlines how to achieve Any/All Docker ContainersGraylog Docker Instance? I’d like to do this without excluding other sources from reading logs.

Also, I’m going to be using this for a few Node JS applications, so if someone has quality configurations that they want to recommend (e.g. Winston/Pino/Bunyan transports to Graylog), I’m definitely interested).

Best

Hello && Welcome @coder

I have by using Filebeat /w Graylog Sidecar. Just filebeat would be fine but I perfer to adjust my setting on the Web UI becuase Im lazy :laughing:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- type: docker
  containers.ids: 
    -  '*'
  type: log
output.logstash:
   hosts: ["192.168.1.100:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

Also logs are normally located here.
/var/lib/docker/containers/<container_id>/<container_id>-json.log

EDIT:
Not sure about this statement

By Archive are you refering to long term backups of the whole container? If so, I just create a checkpoit or backup from the node the containers are on , normally through Veeam free version.

Not “archive” – achieve (successfully accomplish :laughing: ).

One other caveat – I’m using MacOS, so log files are . . .well, I have no idea where, and I’m honestly not sure how Signoz is reading the logs (other than perhaps accessing /var/run/docker.sock/ directly).

Would certainly be nice if this process was a bit more straightforward. :melting_face:

Thanks for the input btw!

Hey

Sorry HAHA I just woke up

Oh sorry , I dont touch Mac stuff, tbh apple /Docker here you may or may not get an anwser. But ill ask around for ya

hey

Just an FYI, I found this.

Here.

All good. Yeah, I’m also on M1 ARM, so there’s an extra :wrench:

:laughing:

Definitely don’t lose any sleep over it, but any input is appreciated. Thanks again.

1 Like

Holy cow man :laughing: is there anything else we should know?
Nah, I’m all good challenge accepted :+1:

Hey,

So I was digging around on the internet, I did see Nxlog and Filebeat is able to be installed on MacOS M1 Arm chip, but looks like there were some issues (i.e., 2020).

So I looked here…

If you have a signoz running on a different host then you will have to run a otel-collector to export logs from your host to the host where SigNoz is running.

Actually researching SigNoz , very simialer in the configurations need like Rsyslog/ Filebeat config.

After reading over there documention for SigNoz this can be achieve NOT archived :laughing:

By using another service to send logs to SigNoz (otel-collector) then shipped out from there to Graylog.

Just different names but the same principle like the rest of the log shippers. TBH SigNoz reminds me of Rsyslog/Rsyslog server.

Eeeeeeesh! Yeah, I don’t necessarily care to go from Graylog to Signoz. Just need to pick one or the other.

Man, who would have thought the idea of streaming/parsing logs could get so convoluted? I could have written my own logging/metrics dashboard app in the few days I’ve spent looking over these options. :laughing:

Appreciate those links. I’ll take a closer look.

I was refering to from Signox to graylog, By using another service to send logs to SigNoz (otel-collector) then to Graylog Im awake now LOL.

otel-collector --> SigNoz --> Graylog

Or

another_log_shipper/s --> Graylog

Of cource, :+1: sorry I cant be more help.

Wow! So I found this GH post about Vector.

I’ve actually got it successfully pulling data via Loki atm. And I can use /var/run/docker.sock on Linux or MacOS. No need for additional drivers or otherwise!

Nah, you were definitely helpful. Sometimes just having a springboard is all you need! Thanks again!

1 Like

I use that also :joy:

Not only for Graylog metrics, Opensearch, Zabbix, Nextcloud, Bookstack, etc… pretty much with everthing. Good stuff

1 Like

Aaaaaaah, very nice! If you have any quality Vector “remap” or “transform” techniques you’d like to share, I’m all ears.

I’m in the fortunate position that I can generate logs in any “shape” that I like. I just need to be able to plot data points from them at the end of the day. Currently, I’m submitting nested JSON objects, but they’re getting pushed through as strings. I’d like to be able to pipe an entire field (not label) back to JSON and query on the object if at all possible. :thinking:

Anyway, thanks again, and feel free to spam ideas. :rofl:

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.