Sidecar cannot get the configuration for filebeat

Graylog Central (peer support)
, , , ,
1

Hello, I have failed to make filebeat work with SSL/TLS with a private self-signed CA in a graylog-2.4.3-1 .

in my configuration, the key and certs are put under /etc/graylog/server for graylog server as:

[root@gl1 ~]# ls -l /etc/graylog/server/*.pem
-rw-rw----. 1 root graylog 2143 2月   8 12:18 /etc/graylog/server/ca-cert.pem
-rw-rw----. 1 root graylog 2029 2月  10 04:05 /etc/graylog/server/rest-cert.pem
-rw-rw----. 1 root graylog 3272 2月  10 04:04 /etc/graylog/server/rest-key.pem
-rw-rw----. 1 root graylog 2029 2月  10 04:05 /etc/graylog/server/web-cert.pem
-rw-rw----. 1 root graylog 3272 2月  10 04:05 /etc/graylog/server/web-key.pem

in the collector host i put the key and certs for filebeat under /etc/graylog/collect-sidecar/ and the ca certificate has been already added to the system trust ca list by update-ca-trust command:

ls -l /etc/pki/ca-trust/source/anchors/ca-cert.pem
-rw-r--r--. 1 root root 2143 2月  13 15:37 /etc/pki/ca-trust/source/anchors/ca-cert.pem

ls -l /etc/graylog/collector-sidecar/*.pem
-rw-r--r--. 1 root root 2143 2月  13 15:46 /etc/graylog/collector-sidecar/ca-cert.pem
-rw-r--r--. 1 root root 2049 2月  13 15:46 /etc/graylog/collector-sidecar/clr-node-cert.pem
-rw-r--r--. 1 root root 3272 2月  13 15:46 /etc/graylog/collector-sidecar/clr-node-key.pem

and the output of sidecar:

graylog-collector-sidecar -debug -c /etc/graylog/collector-sidecar/collector_sidecar.yml
INFO[0000] Using collector-id: 8fb06d83-1cd9-4373-a12e-e1bad4a3f246 
INFO[0000] Fetching configurations tagged by: [linux ntp chronyd] 
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/collector_sidecar.log.%Y%m%d%H%M 
DEBU[0000] Add collector backend: filebeat              
INFO[0000] Starting signal distributor                  
DEBU[0000] [signal-processor] (seq=1) handling cmd: restart 
INFO[0000] [filebeat] Starting (exec driver)            
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0000] [signal-processor] (seq=1) cmd done: restart 
ERRO[0001] [filebeat] Backend finished unexpectedly, trying to restart 1/3. 
DEBU[0001] [signal-processor] (seq=2) handling cmd: restart 
INFO[0001] [filebeat] Stopping                          
DEBU[0003] [filebeat] SIGHUP ignored, killing process   
DEBU[0003] [filebeat] Failed to kill process os: process already finished 
INFO[0003] [filebeat] Starting (exec driver)            
DEBU[0003] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0003] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0003] [signal-processor] (seq=2) cmd done: restart 
ERRO[0004] [filebeat] Backend finished unexpectedly, trying to restart 2/3. 
DEBU[0004] [signal-processor] (seq=3) handling cmd: restart 
INFO[0004] [filebeat] Stopping                          
DEBU[0006] [filebeat] SIGHUP ignored, killing process   
DEBU[0006] [filebeat] Failed to kill process os: process already finished 
INFO[0006] [filebeat] Starting (exec driver)            
DEBU[0006] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0006] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0006] [signal-processor] (seq=3) cmd done: restart 
ERRO[0007] [filebeat] Backend finished unexpectedly, trying to restart 3/3. 
DEBU[0007] [signal-processor] (seq=4) handling cmd: restart 
INFO[0007] [filebeat] Stopping                          
DEBU[0009] [filebeat] SIGHUP ignored, killing process   
DEBU[0009] [filebeat] Failed to kill process os: process already finished 
INFO[0009] [filebeat] Starting (exec driver)            
DEBU[0009] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0009] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0009] [signal-processor] (seq=4) cmd done: restart 
ERRO[0010] [filebeat] Unable to start collector after 3 tries, giving up! 
INFO[0010] [RequestConfiguration] No configuration found for configured tags! 
INFO[0020] [RequestConfiguration] No configuration found for configured tags!

and the status report failed for the collector on the graylog server, but log files have been listed there.

2018-02-13 15:32:32 72 /var/log/chrony
2018-02-13 17:20:23 15070 /var/log/chrony/measurements.log
2018-02-13 17:20:23 11934 /var/log/chrony/statistics.log
2018-02-13 17:20:23 10496 /var/log/chrony/tracking.log

is there anything i missed ? thanks.

–charles

2

the sidecar configuration:

cat /etc/graylog/collector-sidecar/collector_sidecar.yml
server_url: https://gl1.mylogs.com:9000/api/
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files: /var/log/chrony
node_id: clr.mylogs.com
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
    - ntp
    - chronyd
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml
3

what did your filebeat logfile in /var/log/graylog/collector-sidecar gives you back? What is the configuration in /etc/graylog/collector-sidecar/generated/filebeat.yml ?

4

sidecar doesn’t create the filebeat.yml file. and the log:

cat collector_sidecar.log
time="2018-02-13T15:51:22+08:00" level=info msg="Starting signal distributor" 
time="2018-02-13T15:51:22+08:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-02-13T15:51:23+08:00" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 1/3." 
time="2018-02-13T15:51:23+08:00" level=info msg="[filebeat] Stopping" 
time="2018-02-13T15:51:25+08:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-02-13T15:51:26+08:00" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 2/3." 
time="2018-02-13T15:51:26+08:00" level=info msg="[filebeat] Stopping" 
time="2018-02-13T15:51:28+08:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-02-13T15:51:29+08:00" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 3/3." 
time="2018-02-13T15:51:29+08:00" level=info msg="[filebeat] Stopping" 
time="2018-02-13T15:51:31+08:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-02-13T15:51:32+08:00" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!" 
time="2018-02-13T15:51:32+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://gl1.mylogs.com:9000/api/plugins/org.graylog.plugins.collector/8fb06d83-1cd9-4373-a12e-e1bad4a3f246?tags=%5B%22linux%22%2C%22ntp%22%2C%22chronyd%22%5D: EOF" 
time="2018-02-13T15:51:32+08:00" level=error msg="[UpdateRegistration]
5

sorry, the previous are old one, the new logs:

time="2018-02-13T16:56:38+08:00" level=info msg="Stopping signal distributor" 
time="2018-02-13T16:57:05+08:00" level=debug msg="Add collector backend: filebeat" 
time="2018-02-13T16:57:05+08:00" level=info msg="Starting signal distributor" 
time="2018-02-13T16:57:05+08:00" level=debug msg="[signal-processor] (seq=1) handling cmd: restart" 
time="2018-02-13T16:57:05+08:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-02-13T16:57:05+08:00" level=debug msg="Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M" 
time="2018-02-13T16:57:05+08:00" level=debug msg="Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M" 
time="2018-02-13T16:57:05+08:00" level=debug msg="[signal-processor] (seq=1) cmd done: restart" 
time="2018-02-13T16:57:06+08:00" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 1/3." 
time="2018-02-13T16:57:06+08:00" level=debug msg="[signal-processor] (seq=2) handling cmd: restart" 
time="2018-02-13T16:57:06+08:00" level=info msg="[filebeat] Stopping" 
time="2018-02-13T16:57:08+08:00" level=debug msg="[filebeat] SIGHUP ignored, killing process" 
time="2018-02-13T16:57:08+08:00" level=debug msg="[filebeat] Failed to kill process os: process already finished" 
time="2018-02-13T16:57:08+08:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-02-13T16:57:08+08:00" level=debug msg="Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M" 
time="2018-02-13T16:57:08+08:00" level=debug msg="Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M" 
time="2018-02-13T16:57:08+08:00" level=debug msg="[signal-processor] (seq=2) cmd done: restart" 
time="2018-02-13T16:57:09+08:00" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 2/3." 
time="2018-02-13T16:57:09+08:00" level=debug msg="[signal-processor] (seq=3) handling cmd: restart" 
time="2018-02-13T16:57:09+08:00" level=info msg="[filebeat] Stopping" 
time="2018-02-13T16:57:11+08:00" level=debug msg="[filebeat] SIGHUP ignored, killing process" 
time="2018-02-13T16:57:11+08:00" level=debug msg="[filebeat] Failed to kill process os: process already finished" 
time="2018-02-13T16:57:11+08:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-02-13T16:57:11+08:00" level=debug msg="Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M" 
time="2018-02-13T16:57:11+08:00" level=debug msg="Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M" 
time="2018-02-13T16:57:11+08:00" level=debug msg="[signal-processor] (seq=3) cmd done: restart" 
time="2018-02-13T16:57:12+08:00" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 3/3." 
time="2018-02-13T16:57:12+08:00" level=debug msg="[signal-processor] (seq=4) handling cmd: restart" 
time="2018-02-13T16:57:12+08:00" level=info msg="[filebeat] Stopping" 
time="2018-02-13T16:57:14+08:00" level=debug msg="[filebeat] SIGHUP ignored, killing process" 
time="2018-02-13T16:57:14+08:00" level=debug msg="[filebeat] Failed to kill process os: process already finished" 
time="2018-02-13T16:57:14+08:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-02-13T16:57:14+08:00" level=debug msg="Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M" 
time="2018-02-13T16:57:14+08:00" level=debug msg="Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M" 
time="2018-02-13T16:57:14+08:00" level=debug msg="[signal-processor] (seq=4) cmd done: restart" 
time="2018-02-13T16:57:15+08:00" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!" 
time="2018-02-13T16:57:15+08:00" level=info msg="[RequestConfiguration] No configuration found for configured tags!" 
time="2018-02-13T16:57:25+08:00" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
6

please answer my question

7

no, there is no filebeat.yml generated. this is the problem.

8 
cat filebeat_stderr.log
filebeat2018/02/13 10:13:46.077438 beat.go:339: CRIT Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
filebeat2018/02/13 10:13:49.080472 beat.go:339: CRIT Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
filebeat2018/02/13 10:13:52.080407 beat.go:339: CRIT Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
filebeat2018/02/13 10:13:55.084897 beat.go:339: CRIT Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
9

Should the key and certs for filebeat put on graylog server for sidebar download or manually put on filebeat host?

10

is your configured server_url in the sidecar configuration your Graylogs rest_listen_uri ?

Did you checked your collector_sidecar.yml that is is a valid .yml file?

Are you able to reach the configured server_url from the server where you run the collector sidecar? Did you add your CA in a proper way to that Servers trust store?

Given your certifiate is named foo.crt

Debian / Ubuntu

Copy your CA to dir `/usr/local/share/ca-certificates/`
Use command like: `sudo cp foo.crt /usr/local/share/ca-certificates/foo.crt`
Update the CA store: `sudo update-ca-certificates`

CentOS / RedHat

Install the ca-certificates package: `yum install ca-certificates`
Enable the dynamic CA configuration feature: `update-ca-trust force-enable`
Add it as a new file to /etc/pki/ca-trust/source/anchors/: `cp foo.crt /etc/pki/ca-trust/source/anchors/`
Use command: `update-ca-trust extract`

Are you able to reach the configure server_uri with curl? What is the output of curl -I YOURURI ?

11

Yes. i have configured server_url to the rest_listen_uri, and collector sidecar can reach the server_url.

i have add the CA cert into Java Trusted CA store on the graylog-server, but not the system’s trust CA store:

cat /etc/graylog/collector-sidecar/collector_sidecar.yml
server_url: https://gl1.mylogs.com:9000/api/
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files: /var/log/chrony
node_id: clr.mylogs.com
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
    - ntp
    - chronyd
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml
12

@cdeng Please properly format your configuration and text snippets for readability: http://commonmark.org/help/

Example:

``` 
Some text
More text
```
13

curl -I https://gl1.mylogs.com:9000/api/
HTTP/1.1 200 OK
X-Graylog-Node-ID: f9818ad6-9b6b-41d0-83ae-0bfe8ab4fa72
X-Runtime-Microseconds: 1325
Content-Length: 232
Content-Type: application/json
Date: Tue, 13 Feb 2018 10:54:14 GMT

14

i have add the CA cert into Java Trusted CA store on the graylog-server

Your client server - where the collector-sidecar runs - need to have the CA which is used to sign the certificate that is used to secure your Graylog server. How should a go program running on another server know that the CA certificate you have used to sign the certificate can be trusted?

15

the collector side already add the CA to the system trusted CA list.

16

and you have configurations in Graylog for your configured tags?

tags:
    - linux
    - ntp
    - chronyd

with some kind of configuration?

17

The strange is that on the graylog server’s collector’s status, we can see the log files list, but it seems the configuration does not download.

18

yes. i have attached all those tags with the configuration on graylog server side.

19

It seems the SSL/TLS between sidecar and graylog works, but when make a configuration for collector failed.

i am not trying to use the server self signed certificate, but input the key file and cert file for collector. does the key/cert file in the output configuration should point to path on the graylog server or collector host ? is there any ownership requirements for it ?

20

did you have created a beats input on Graylog to receive the messages?

The certificates needs to be on the server where they need to be used (filebeat on the sending server, beats input on all Graylog servers).

Did you tried the communication between filebeat and graylog without tls to verify that everything else is working?