Continuing the topic - 'Graylog log ingestion from file path and not from port listening'

I had created a topic with the name Graylog log ingestion from file path and not from port listening
I can’t comment on it as the the topic is closed.

Right now on my graylog I am ingesting logs from local file using filebeat.
I am not using Graylog-sidecar.

I am trying to understand why Graylogsidewar+filebeat is preferred and not standalone filebeat which I have setup right now?

Graylog sidecar allows you to manage all the settings (other than the base connection settings) in the Graylog interface. This in turn allows you to apply a single consistent log configuration to multiple machines with one pane of glass for creation, editing and deployment. it is not required but it makes things easier.

Hello @spandey

Hey thats me :laughing: Thx for the screenshot.

Adding on to @tmacgbay statement about controlling nodes in a central area (CIC).

Graylog Sidecar is a wrapper for those other log shippers, So all that is need is to download and install Graylog_Sidecar and insure you able to see the “Phone Home” on Graylogs Web UI. Then all that is need is to create a configuration and add it to the sidecar via Web UI.

Example of a configuration for 100 Linux servers.

This is really great for big environments.

2 Likes

Thanks @gsmith @tmacgbay for the response.
@gsmith You have replied in all of my questions before so thanks for that!

So if I understand correctly , graylog-sidecar is like a abstraction for different log shipper and any configuration for these log shipper can be done on Graylog-sidecar itself ie., on the UI itself instead of logging into the vm and configuring there.
Also same configuration can be applied to ‘n’ number of devices through graylog-sidecar instead of individual configurations for devices.

Yes that is correct :+1:

@ttsandrew @gsmith I have suggestion to ask regarding graylog design.

We have a syslog server where all the vms send there logs at port 514.
On syslog server all the logs are stored at /import/system/logs.

I have setup the graylog on syslog server and instead of listening port I am importing all the logs present in /import/system/logs to the graylog using filebeat.

One other way of implementing this is below:-
I could have instead setup the graylog to listen to say port 1514 and forward all the logs coming at 514 to 1514. But then the logs won’t get saved at ‘/import/system/logs’.
The use-case is that we want to save these logs at nfs mounted at above location.

So that’s why I have implemented importing logs into graylog using filebeat instead of listening to ports.

Do you guys think using filebeat + graylog sidecar is correct design or is there any other way like listening to port etc ., could be better implementation?

Looking forward to your responses!
Thanks again!

Couple things you can do is …

Iptables add a re-route rule for port 514 to port 5141.

or graylog-sidecar-filebeat

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    -/import/system/*.logs
  type: log
output.logstash:
   hosts: ["batman.com:5141"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

Depends on what you want to do either way you will get your messages to Graylog.

2 Likes

Google DNS and Batman.com accept logs! :stuck_out_tongue_winking_eye:

2 Likes

:rofl: That’s a good one :+1:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.