Filebeat config not found. Tags - no config found


(Ofentse) #1

I see that this topic has been covered a couple of times. And in each time I go through thread I replicate the solutions/checks and still have this issue.

a) Is it normal for file = /etc/graylog/collector-sidecar/generated/filebeat.yml to be missing after installing collector-sidecar 0.1.6? Is it OK if I take filebeat config from forum and customize it. Then replace the missing filebeat.yml?

b) My config for tags is as below. The tags show in Graylog 2.4 appliance I am using for demo. Does this mean this is OK?

server_url: https://gl1.mylogs.com:9000/api/
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files: /var/log/chrony
node_id: clr.mylogs.com
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
linuxcollector
backends:
- name: nxlog
enabled: false
binary_path: /usr/bin/nxlog
configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
- name: filebeat
enabled: true
binary_path: /usr/bin/filebeat
configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml


Windows events to graylog server
(Jan Doberstein) #2

when you have the sidecar installed (current stable 0.1.7 as time of writing) you need to configure filebeat from the Graylog UI as documented: http://docs.graylog.org/en/2.4/pages/collector_sidecar.html#step-by-step-guide

In addition you need to create a BEATS Input on Graylog that is the target for your logs.


(Tess) #3

Yes this is normal, because in my experience the Sidecar generates the file for you, based on the configuration you make in Graylog. As Jan points out, you need to first do some config in the Graylog GU.

  1. Define a collector config, in System > Collectors > Manage configurations.
  2. For example, a config called “Windows”, with the tag “windows”.
  3. You edit the config by setting up an output (for example Winlogbeat targeted at the Graylog server) and an input (for example Window Event Log).
  4. Once that is setup, you edit the Sidecar config file and i the tags section you add the new “windows” tag.

Once the Sidecar gets restarted, it’ll pick up that it has a config to retrieve. It’ll pull it from Graylog and then arrange the local filebeat/winlogbeat config.


(Ofentse) #4

Thank you @jan and @Totally_Not_A_Robot. The process has become a bit clearer for me.

@Totally_Not_A_Robot is it ok if I configure the collector then configure the GUI with the tags? Restarting the collector should then get it to collect the config?


(Ofentse) #5

in fact it seems like the server does attempt to restart the collector when you click restart in the collector configuration side.


(Ben van Staveren) #6

As long as the collector sidecar is configured with tags, when you create the configs in the Graylog UI they will automatically be synched within a minute or so and the collector will restart filebeat and start sending. No need to restart the collector - unless, of course, you added tags to it’s configuration :slight_smile:


(Ofentse) #7

Thank you. I think I have picked up some mistakes in my dealing with the matter. So I hope the sync will happen shortly :slight_smile:


(Ofentse) #8

So I have followed step by step. Not sure why node-id was an issue. Filebeat is failing to start in background. Graylog Server is not generating the required filebeat.yml file. Is it reasonable to create my own filebeat.yml as workaround? It seems like it will never be overwritten.


(Ofentse) #9

For config I have the below:

image


(Jan Doberstein) #10

you latest error loglines give a clear picture … you have not configured anything in the Graylog UI OR you just have not have any tags given to the configuration that matches the tags you have in the sidecar configuration at the server.


(Ofentse) #11

Thanks @jan for feedback.

From below image I got the impression that the tags have been configured as per config files above. “linux” and “ussdrica”. The names I configured in collector*.yml automatically showed up in the GUI.

image


(Jan Doberstein) #12

yes but you have not done that for the configuration http://docs.graylog.org/en/2.5/pages/collector_sidecar.html#step-by-step-guide

This step 6 is missing - because you need to assign a tag to the configuration that Graylog knows what configuration should be written to what sidecar.


(Tess) #13

Well that, and you need to configure the Beats output and the inputs :slight_smile: As I listed in my earlier pst.


(Ofentse) #14

Thanks @jan, from the below, it may seem like I added the tags but the below picture clearly disagrees. So I need to spend more time on this step. Also I note that each time I look for the tags, I also see that there is no auto-suggest for the tag, while it shows the “windows” tag of a successful sidecar collector I did for windows. I will try again


(Ofentse) #15

Thanks @Totally_Not_A_Robot. I will also review those steps. Hopefully I do not need to specify the tags in other steps too (besides the above mentioned and the collector*.yml)


(Ofentse) #16

I recall that for Windows I use the collector with NXLog. I see there is backend snippet configured. Does this need to be done for filebeat for Linux?


(Ofentse) #17

Thank you all. I think this topic can be closed.

I made a mistake I made before. Perhaps it is because I last did a config about 6 months ago. I did not CREATE the tags. I only added the tag words and pressed update. I actually need to insert tag name, create option and then select update.

I have learnt a lesson I will never forget. Many thanks for your assistance.


(system) closed #18

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.