Can someone tell me if I got this right?

First of all I am new to Graylog.

I have Graylog 3 configured on an instance in google cloud and it appears to be running, however when looking at configuring everything there appears to be a number of basic ways to do the same thing and because of this I’m confused about where to being troubleshooting and how to test if everything is working.

-From what I understand there is graylog which is a service I have setup to ingest logs sent to it via inputs (beats in my case).
-There is also the sidecar feature, which appears to be something used to configure outputs/clients on machines that need their logs sent to graylog.
-For inputs there are a number of options but filebeat seems to be the one most recommend using and that is what I’m trying to get setup.

Here is where I get confused. What part does logstash play in all of this? I’ve seen step by step guides that refer to it as necessary then posts here in the community that say its no longer necessary. Any kind of posts or troubleshooting of beats seems to relay me to an ELK forum and I’m not sure how that translates to Graylog.

Honestly, I’ve spent a week now on and off trying to get this setup.

Could anyone give me some tips on what exactly I need, where the basic log files are for these essential services are, and what I can do to troubleshoot on the clients?

Logstash shows up in the beats configuration because those Beasts is for Elasticsearch/Logstash and was not syntactically built for Graylog… on the other hand Graylog was built to receive data output by the various beats (winlogbeat, filebeat, auditbeat… etc). While you do need elasticsearch for backend storage of data, there is no need for logstash.

Rough outline - Graylog receives messages from servers set via beats/nexlog/syslog etc… and manipulates the data (beaks out fields and potentially takes action on the detail) with the help of a underlying Mongo DB. When all manipulation is done the message the result is shipped out to a Elasticsearch DB for future retrieval in a search… The sidecar system allows you to centrally create and deploy configurations to sidecar clients but the sidecar has to connect to Graylog first (local configuration) before Graylog can start controlling configuration - it does not do this by default.

Logfile paths for Graylog are in the documentation and easy to find, for details on Mongo/elasticsearch log files… best to find those at their web sites.- beats logs are on the client in the directory where beats is installed. The beats log files do a pretty good job of telling you what it’s trying to do.

1 Like

OK, thanks for your reply. I think I’m missing a step or something. I’ve got the sidecar setup and making changes to the filebeat.conf file on my test server. The issue is, that file does nothing to my filebeat configuration as the default location for filebeat’s config file appears to be /etc/filebeat/filebeat.yml and the sidecar creates a file called filebeat.conf in the folder “/var/lib/graylog-sidecar/generated”. I know I could manually copy and rename the file but that seems counter intuitive to the way the sidecar system should work.

I’ve been using the following command to run filebeat:
sudo filebeat -e -c /var/lib/graylog-sidecar/generated/filebeat.conf
which looks like this:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: {sidecar.nodeName} fields.gl2_source_collector: {sidecar.nodeId}

- input_type: log
    - /var/log/*.log
  type: log
#  hosts: ["IP-HERE:5044"]
#  ssl.enabled: true
#  ssl.certificate: "/etc/ssh/"
#  ssl.key: "/etc/ssh/"
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

I commented out the logstash part but now when I look at the logs I get this:

2020-04-06T21:43:30.748Z        INFO    instance/beat.go:280    Setup Beat: filebeat; Version: 6.8.8
2020-04-06T21:43:30.748Z        INFO    instance/beat.go:309    No outputs are defined. Please define one under the output section.
2020-04-06T21:43:30.748Z        INFO    instance/beat.go:359    filebeat stopped.
2020-04-06T21:43:30.748Z        ERROR   instance/beat.go:906    Exiting: No outputs are defined. Please define one under the output section.

If I uncomment the logstash part, I get:

2020-04-06T21:46:08.342Z        ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://3EditedOutIP::5044)): read tcp>EditedOutIP:: read: connection reset by peer
2020-04-06T21:46:08.342Z        INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://EditedOutIP::5044)) with 1 reconnect attempt(s)
2020-04-06T21:46:12.236Z        ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://EditedOutIP::5044)): read tcp>EditedOutIP:5044: read: connection reset by peer

You need the logstash part. Beats doesn’t have an output.graylog command so you use output.logstash.

Do you have an Input set up on Graylog to receive the sidecar output on port 5044?

I figured it out. I needed the CA for the ssl key in the logstash config.

Could you explain the best practice for applying the sidecar filebeat.conf to the remote systems? I’m hoping to use ansible to automate the deployment process but right now now the sidecar creates filebeat.conf in /var/lib/graylog-sidecar/generated and filebeat is expected filebeat.yml in /etc/filebeat.

According to the documentation the sidecar should be creating a .yml file but instead it’s creating a .conf file. Is there an option I missed somewhere?

I don’t work in the linux sidecar side that much but here is my short take. Don’t touch anything in generated - that is the stuff pushed from Graylog. When you do an initial install, have ansible mod/swap the sidecar.yml file so that it specifically points to your graylog server(s). When the sidecar starts it will use that (And other settings I am not mentioning) to connect to Graylog. It’s a base configuration the real configuration is handled in the Graylog UI. Below is my initial windows C:\Program Files\graylog\sidecar\sidecar.yml which is very similar:

server_url: http://GRAYLOG-001:9000/api/
server_api_token: "jumble-o-stuff" 
update_interval: 10
tls_skip_verify: true
send_status: true
collector_id: file:C:\Program Files\Graylog\sidecar\collector-id
cache_path: C:\Program Files\Graylog\sidecar\cache
log_path: C:\Program Files\Graylog\sidecar\logs
log_rotation_time: 86400
log_max_age: 604800
tags: [windows]
collector_binaries_whitelist: []
    - name: nxlog
      enabled: false
      binary_path: C:\Program Files (x86)\nxlog\nxlog.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\nxlog.conf
    - name: winlogbeat
      enabled: true
      binary_path: C:\Program Files\Graylog\sidecar\winlogbeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\winlogbeat.yml
    - name: filebeat
      enabled: true
      binary_path: C:\Program Files\Graylog\sidecar\filebeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\filebeat.yml
    - name: auditbeat
      enabled: true
      binary_path: C:\Program Files\Graylog\sidecar\auditbeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\auditbeat.yml

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.