Sidecar cannot get the configuration for filebeat

cdeng · February 19, 2018, 1:22pm

and i got this:

cdeng · February 19, 2018, 1:25pm

also noted that i have opened the firewall for the Beats Input port on each server of the graylog cluster now that the selinux is enforcing:

firewall-cmd --list-all

public (active)
target: default
icmp-block-inversion: no
interfaces: eth0 eth1
sources:
services: ssh
ports: 9000/tcp 12201/tcp 5044/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:

jan · February 19, 2018, 1:25pm

you do not have a tag for your configuration.

please check http://docs.graylog.org/en/2.4/pages/collector_sidecar.html#step-by-step-guide special the part with the tags

cdeng · February 19, 2018, 1:46pm

then i configure the collector host:

put the CA certificate file ca-cert.pem under directory /etc/pki/ca-trust/source/anchors/

# pwd
 /etc/pki/ca-trust/source/anchors
# ls -l
总用量 4
-rw-r--r--. 1 root root 2143 2月  19 20:08 ca-cert.pem

add the CA ceritficate into CA trusted by update-ca-trust extract and check it works by curl access to the rest interface and web interface of graylog:

# update-ca-trust extract
# curl -I https://gl1.mylogs.com:9000/api
HTTP/1.1 200 OK
X-Graylog-Node-ID: bcb2f984-5c5d-4e83-81cd-102c4a299b37
X-Runtime-Microseconds: 1213
Content-Length: 232
Content-Type: application/json
Date: Mon, 19 Feb 2018 12:10:00 GMT

# curl -I https://gl1.mylogs.com:9000
HTTP/1.1 200 OK
X-UA-Compatible: IE=edge
X-Graylog-Node-ID: bcb2f984-5c5d-4e83-81cd-102c4a299b37
Content-Length: 1640
Content-Type: text/html
Date: Mon, 19 Feb 2018 12:10:17 GMT

install collector-sider by:

# rpm -Uvh https://github.com/Graylog2/collector-sidecar/releases/download/0.1.4/collector-sidecar-0.1.4-1.x86_64.rpm

put the collector’s key and certificate which signed by CA certificate and CA certifcate itself under directory /etc/graylog/collector-sidecar/

# ls -l /etc/graylog/collector-sidecar/
总用量 16
-rw-r--r--. 1 root root 2143 2月  19 21:38 ca-cert.pem
-rw-r--r--. 1 root root 2029 2月  19 21:38 clr-node-cert.pem
-rw-r--r--. 1 root root 3272 2月  19 21:39 clr-node-key.pem
-rw-rw-r--. 1 root root  703 7月  31 2017 collector_sidecar.yml
drwxrwxr-x. 2 root root    6 7月  31 2017 generated

configure the /etc/graylog/collector-sidecar/collect_sidecar.yml

# cat /etc/graylog/collector-sidecar/collector_sidecar.yml
server_url: https://gl1.mylogs.com:9000/api/
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files: /var/log/chrony
node_id: clr.mylogs.com
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
    - ntp
    - chronyd
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

run the sidecar by:

# graylog-collector-sidecar -service install
# systemctl start collector-sidecar
# systemctl -l status collector-sidecar
● collector-sidecar.service - Wrapper service for Graylog controlled collector
   Loaded: loaded (/etc/systemd/system/collector-sidecar.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2018-02-19 21:44:45 CST; 12s ago
 Main PID: 18324 (graylog-collect)
   CGroup: /system.slice/collector-sidecar.service
           └─18324 /usr/bin/graylog-collector-sidecar

2月 19 21:44:46 clr.mylogs.com graylog-collector-sidecar[18324]: time="2018-02-19T21:44:46+08:00" level=info msg="[filebeat] Stopping"
2月 19 21:44:48 clr.mylogs.com graylog-collector-sidecar[18324]: time="2018-02-19T21:44:48+08:00" level=info msg="[filebeat] Starting (exec driver)"
2月 19 21:44:49 clr.mylogs.com graylog-collector-sidecar[18324]: time="2018-02-19T21:44:49+08:00" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 2/3."
2月 19 21:44:49 clr.mylogs.com graylog-collector-sidecar[18324]: time="2018-02-19T21:44:49+08:00" level=info msg="[filebeat] Stopping"
2月 19 21:44:51 clr.mylogs.com graylog-collector-sidecar[18324]: time="2018-02-19T21:44:51+08:00" level=info msg="[filebeat] Starting (exec driver)"
2月 19 21:44:52 clr.mylogs.com graylog-collector-sidecar[18324]: time="2018-02-19T21:44:52+08:00" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 3/3."
2月 19 21:44:52 clr.mylogs.com graylog-collector-sidecar[18324]: time="2018-02-19T21:44:52+08:00" level=info msg="[filebeat] Stopping"
2月 19 21:44:54 clr.mylogs.com graylog-collector-sidecar[18324]: time="2018-02-19T21:44:54+08:00" level=info msg="[filebeat] Starting (exec driver)"
2月 19 21:44:55 clr.mylogs.com graylog-collector-sidecar[18324]: time="2018-02-19T21:44:55+08:00" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!"
2月 19 21:44:55 clr.mylogs.com graylog-collector-sidecar[18324]: time="2018-02-19T21:44:55+08:00" level=info msg="[RequestConfiguration] No configuration found for configured tags!"

noticed that it report No configuration found …

jan · February 19, 2018, 1:48pm

please:

cdeng · February 19, 2018, 1:48pm

and check against the collector status on the graylog:

cdeng · February 19, 2018, 1:50pm

and more information on the collector status:

cdeng · February 19, 2018, 1:52pm

and indeed, for each time i restart the sidecar service, the collector status on graylog will turn to running shortly and then to failing:

cdeng · February 19, 2018, 1:54pm

then failing again

cdeng · February 19, 2018, 1:58pm

the only things i suspect if it is related to my dual homed hosts: clr.mylogs.com:

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:7f:8a:34 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.51/24 brd 10.10.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe7f:8a34/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:99:2f:72 brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.4/24 brd 192.168.3.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe99:2f72/64 scope link 
       valid_lft forever preferred_lft forever
# ip route
default via 192.168.3.1 dev eth1 proto static metric 100 
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.51 metric 100 
192.168.3.0/24 dev eth1 proto kernel scope link src 192.168.3.4 metric 100

it seems sidecar already communicate with graylog for uploading the list of the log directory, but failed to download the configuration.

any idea ?

jan · February 19, 2018, 1:59pm

cdeng · February 19, 2018, 2:04pm

yes. you are right. it is so trickly. i need to press enter before press update the tags. thank you very much !

system · March 5, 2018, 2:04pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failed loading configuration Graylog Central (peer support) sidecar , filebeat-linux , nosendlogfblx , failoadcongfigfblx	15	5425	August 2, 2018
UBUNTU Sidecar log Config fail Graylog Central (peer support)	12	567	October 12, 2022
Unable to run collector-sidecar Graylog Central (peer support) sidecar , filebeat-linux , nosendlogfblx	18	9886	July 20, 2018
Collector side car using filebeat Graylog Central (peer support) sidecar , filebeat-linux , nosendlogfblx , failoadcongfigfblx	23	5310	April 12, 2018
Graylog sidecar and filebeat Graylog Central (peer support) pipeline-rules , sidecar , filebeat-linux , filebeat-windows , nosendlogfblx	57	10590	September 16, 2021

Sidecar cannot get the configuration for filebeat

firewall-cmd --list-all

Related topics