Continuing the topic - 'Graylog log ingestion from file path and not from port listening'

[spandey]

I had created a topic with the name Graylog log ingestion from file path and not from port listening
I can’t comment on it as the the topic is closed.

Right now on my graylog I am ingesting logs from local file using filebeat.
I am not using Graylog-sidecar.

I am trying to understand why Graylogsidewar+filebeat is preferred and not standalone filebeat which I have setup right now?

[tmacgbay]

Graylog sidecar allows you to manage all the settings (other than the base connection settings) in the Graylog interface. This in turn allows you to apply a single consistent log configuration to multiple machines with one pane of glass for creation, editing and deployment. it is not required but it makes things easier.

[gsmith]

Hello @spandey

Hey thats me Thx for the screenshot.

Adding on to @tmacgbay statement about controlling nodes in a central area (CIC).

Graylog Sidecar is a wrapper for those other log shippers, So all that is need is to download and install Graylog_Sidecar and insure you able to see the “Phone Home” on Graylogs Web UI. Then all that is need is to create a configuration and add it to the sidecar via Web UI.

Example of a configuration for 100 Linux servers.

This is really great for big environments.

[spandey]

Thanks @gsmith @tmacgbay for the response.
@gsmith You have replied in all of my questions before so thanks for that!

So if I understand correctly , graylog-sidecar is like a abstraction for different log shipper and any configuration for these log shipper can be done on Graylog-sidecar itself ie., on the UI itself instead of logging into the vm and configuring there.
Also same configuration can be applied to ‘n’ number of devices through graylog-sidecar instead of individual configurations for devices.

[gsmith]

@spandey

So if I understand correctly , graylog-sidecar is like a abstraction for different log shipper and any configuration for these log shipper can be done on Graylog-sidecar itself ie., on the UI itself instead of logging into the vm and configuring there.

Yes that is correct

[spandey]

@ttsandrew @gsmith I have suggestion to ask regarding graylog design.

We have a syslog server where all the vms send there logs at port 514.
On syslog server all the logs are stored at /import/system/logs.

I have setup the graylog on syslog server and instead of listening port I am importing all the logs present in /import/system/logs to the graylog using filebeat.

One other way of implementing this is below:-
I could have instead setup the graylog to listen to say port 1514 and forward all the logs coming at 514 to 1514. But then the logs won’t get saved at ‘/import/system/logs’.
The use-case is that we want to save these logs at nfs mounted at above location.

So that’s why I have implemented importing logs into graylog using filebeat instead of listening to ports.

Do you guys think using filebeat + graylog sidecar is correct design or is there any other way like listening to port etc ., could be better implementation?

Looking forward to your responses!
Thanks again!

[gsmith]

Couple things you can do is …

Iptables add a re-route rule for port 514 to port 5141.

or graylog-sidecar-filebeat

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    -/import/system/*.logs
  type: log
output.logstash:
   hosts: ["batman.com:5141"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

Depends on what you want to do either way you will get your messages to Graylog.

[tmacgbay]

Google DNS and Batman.com accept logs!

[gsmith]

@tmacgbay

Google DNS and Batman.com accept logs!

That’s a good one