I have Graylog 3 configured on an instance in google cloud and it appears to be running, however when looking at configuring everything there appears to be a number of basic ways to do the same thing and because of this I’m confused about where to being troubleshooting and how to test if everything is working.
-From what I understand there is graylog which is a service I have setup to ingest logs sent to it via inputs (beats in my case).
-There is also the sidecar feature, which appears to be something used to configure outputs/clients on machines that need their logs sent to graylog.
-For inputs there are a number of options but filebeat seems to be the one most recommend using and that is what I’m trying to get setup.
Here is where I get confused. What part does logstash play in all of this? I’ve seen step by step guides that refer to it as necessary then posts here in the community that say its no longer necessary. Any kind of posts or troubleshooting of beats seems to relay me to an ELK forum and I’m not sure how that translates to Graylog.
Honestly, I’ve spent a week now on and off trying to get this setup.
Could anyone give me some tips on what exactly I need, where the basic log files are for these essential services are, and what I can do to troubleshoot on the clients?
Logstash shows up in the beats configuration because those Beasts is for Elasticsearch/Logstash and was not syntactically built for Graylog… on the other hand Graylog was built to receive data output by the various beats (winlogbeat, filebeat, auditbeat… etc). While you do need elasticsearch for backend storage of data, there is no need for logstash.
Rough outline - Graylog receives messages from servers set via beats/nexlog/syslog etc… and manipulates the data (beaks out fields and potentially takes action on the detail) with the help of a underlying Mongo DB. When all manipulation is done the message the result is shipped out to a Elasticsearch DB for future retrieval in a search… The sidecar system allows you to centrally create and deploy configurations to sidecar clients but the sidecar has to connect to Graylog first (local configuration) before Graylog can start controlling configuration - it does not do this by default.
Logfile paths for Graylog are in the documentation and easy to find, for details on Mongo/elasticsearch log files… best to find those at their web sites.- beats logs are on the client in the directory where beats is installed. The beats log files do a pretty good job of telling you what it’s trying to do.
OK, thanks for your reply. I think I’m missing a step or something. I’ve got the sidecar setup and making changes to the filebeat.conf file on my test server. The issue is, that file does nothing to my filebeat configuration as the default location for filebeat’s config file appears to be /etc/filebeat/filebeat.yml and the sidecar creates a file called filebeat.conf in the folder “/var/lib/graylog-sidecar/generated”. I know I could manually copy and rename the file but that seems counter intuitive to the way the sidecar system should work.
I’ve been using the following command to run filebeat: sudo filebeat -e -c /var/lib/graylog-sidecar/generated/filebeat.conf
which looks like this:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: {sidecar.nodeName}
fields.gl2_source_collector: {sidecar.nodeId}
I commented out the logstash part but now when I look at the logs I get this:
2020-04-06T21:43:30.748Z INFO instance/beat.go:280 Setup Beat: filebeat; Version: 6.8.8
2020-04-06T21:43:30.748Z INFO instance/beat.go:309 No outputs are defined. Please define one under the output section.
2020-04-06T21:43:30.748Z INFO instance/beat.go:359 filebeat stopped.
2020-04-06T21:43:30.748Z ERROR instance/beat.go:906 Exiting: No outputs are defined. Please define one under the output section.
2020-04-06T21:46:08.342Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://3EditedOutIP::5044)): read tcp 10.100.2.137:49430->EditedOutIP:: read: connection reset by peer
2020-04-06T21:46:08.342Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://EditedOutIP::5044)) with 1 reconnect attempt(s)
2020-04-06T21:46:12.236Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://EditedOutIP::5044)): read tcp 10.100.2.137:49436->EditedOutIP:5044: read: connection reset by peer
I figured it out. I needed the CA for the ssl key in the logstash config.
Could you explain the best practice for applying the sidecar filebeat.conf to the remote systems? I’m hoping to use ansible to automate the deployment process but right now now the sidecar creates filebeat.conf in /var/lib/graylog-sidecar/generated and filebeat is expected filebeat.yml in /etc/filebeat.
According to the documentation the sidecar should be creating a .yml file but instead it’s creating a .conf file. Is there an option I missed somewhere?
I don’t work in the linux sidecar side that much but here is my short take. Don’t touch anything in generated - that is the stuff pushed from Graylog. When you do an initial install, have ansible mod/swap the sidecar.yml file so that it specifically points to your graylog server(s). When the sidecar starts it will use that (And other settings I am not mentioning) to connect to Graylog. It’s a base configuration the real configuration is handled in the Graylog UI. Below is my initial windows C:\Program Files\graylog\sidecar\sidecar.yml which is very similar:
