Failed loading configuration

PitoneMaledetto · July 17, 2018, 3:40pm

Hi all,
After battling with certificates now I am unable to start collector-sidecar.
I get the following error:
Failed loading configuration file: /etc/graylog/collector-sidecar/collector_sidecar.yml

The version I am using is:
Graylog Collector Sidecar version 0.1.6 (77045b2) [go1.9.3/amd64]

And this is my configuration file:

server_url: https://myserver/api
update_interval: 10
tls_skip_verify: false
send_status: true
list_log_files:
	- /srv/galileoLive/log
node_id: mynode_id
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
#    - apache
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

I must add that at the very beginning I supplied the filebeat.yml configuration file and all worked ok (not sure if it has any relevance) but then it all stopped working even when I moved the filebeat.yml under /etc/graylog/collector-sidecar/generated. It just does not start any more.

Any help would be appreciated.
Thank you

jochen · July 17, 2018, 4:02pm

Please post the complete logs of the Graylog Collector Sidecar and remove your manually crafted filebeat.yml from the configured configuration_path.

PitoneMaledetto · July 18, 2018, 8:24am

Hi jochen,
Thank you for the reply. To be complete with my information I am using Logstash to output to Graylog from my Beats. As far as I understand this should be fine since the configuration will be managed by Graylog and the log output independently to Logstash for filtering down to Graylog.

Here is the edited configuration file:

server_url: https://myserver/api
update_interval: 10
tls_skip_verify: false
send_status: true
list_log_files:
        - /srv/galileoLive/log
node_id: mynode_id
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
#    - apache
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/

There is nothing much in the log really and nothing at all for yesterday (17-07-2018) but here you go. I can’t post the whole log since is too big but anyway mainly is just a repeat of the same message, so I post only relevant entries:

time="2018-07-16T13:20:33+02:00" level=info msg="[RequestConfiguration] No configuration found for configured tags!"

time="2018-07-16T14:30:48+02:00" level=info msg="Stopping signal distributor" 
time="2018-07-16T14:30:48+02:00" level=info msg="[filebeat] Stopping" 
time="2018-07-16T14:30:49+02:00" level=info msg="Starting signal distributor" 
time="2018-07-16T14:30:49+02:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-07-16T14:30:50+02:00" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 1/3." 
time="2018-07-16T14:30:50+02:00" level=info msg="[filebeat] Stopping" 
time="2018-07-16T14:30:52+02:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-07-16T14:30:53+02:00" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 2/3." 
time="2018-07-16T14:30:53+02:00" level=info msg="[filebeat] Stopping" 
time="2018-07-16T14:30:55+02:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-07-16T14:30:56+02:00" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 3/3." 
time="2018-07-16T14:30:56+02:00" level=info msg="[filebeat] Stopping" 
time="2018-07-16T14:30:58+02:00" level=info msg="[filebeat] Starting (exec driver)" 
time="2018-07-16T14:30:59+02:00" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!" 

time="2018-07-16T14:41:51+02:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get https://qa-gb-graylog-01.stanleybet.com/api/plugins/org.graylog.plugins.collector/6d0adaf5-7c04-459c-bf30-3adbbf849b3f?tags=%5B%22linux%22%5D: x509: certificate signed by unknown authority" 
time="2018-07-16T14:41:58+02:00" level=info msg="Stopping signal distributor" 

Thank you for your help.
Regards

jochen · July 18, 2018, 8:35am

You’ll have to disable certificate authentication (tls_skip_verify ): http://docs.graylog.org/en/2.4/pages/collector_sidecar.html#configuration

PitoneMaledetto · July 18, 2018, 8:40am

This is an old entry 16-07-2018 since then I have solved the certificate issue by importing and trust our own signed certificate used by nginx.
Nonetheless I will try by:

tls_skip_verify: true

Thanks

PitoneMaledetto · July 18, 2018, 8:46am

Nothing:

root@linuxbox: systemctl status collector-sidecar
● collector-sidecar.service - Wrapper service for Graylog controlled collector
   Loaded: loaded (/etc/systemd/system/collector-sidecar.service; enabled)
   Active: activating (auto-restart) (Result: exit-code) since Wed 2018-07-18 10:42:13 CEST; 2s ago
  Process: 32067 ExecStart=/usr/bin/graylog-collector-sidecar (code=exited, status=1/FAILURE)
 Main PID: 32067 (code=exited, status=1/FAILURE)

Jul 18 10:42:13 linuxbox systemd[1]: Unit collector-sidecar.service entered failed state.

Even manually invoking the start command like so:

root@linuxbox: /usr/bin/graylog-collector-sidecar -debug
Failed loading configuration file: /etc/graylog/collector-sidecar/collector_sidecar.yml

Just does not work anymore and since it does not even attempt to start the log file has nothing regarding the issue more than what I have already posted above.
It seem that it cannot find the configuration any-more.
Thanks

jochen · July 18, 2018, 9:01am

Try starting the Graylog Collector Sidecar manually and check the output.
http://docs.graylog.org/en/2.4/pages/collector_sidecar.html#debug

PitoneMaledetto · July 19, 2018, 9:11am

Hi,
I have already tried it:

root@linuxbox:/usr/bin# graylog-collector-sidecar -debug -c /etc/graylog/collector-sidecar/collector_sidecar.yml
Failed loading configuration file: /etc/graylog/collector-sidecar/collector_sidecar.yml

Thanks

jochen · July 19, 2018, 9:24am

Please post the output of the following command and the current contents of the configuration file (collector_sidecar.yml).

# namei -l /etc/graylog/collector-sidecar/collector_sidecar.yml

PitoneMaledetto · July 19, 2018, 11:12am

root@linuxbox:/usr/bin# namei -l /etc/graylog/collector-sidecar/collector_sidecar.yml
f: /etc/graylog/collector-sidecar/collector_sidecar.yml
drwxr-xr-x root root /
drwxr-xr-x root root etc
drwxrwxr-x root root graylog
drwxrwxr-x root root collector-sidecar
-rw-rw-r-- root root collector_sidecar.yml

Configuration:

server_url: https://mygraylogserver/api
update_interval: 10
tls_skip_verify: true 
send_status: true
list_log_files:
	- /srv/galileoLive/log
node_id: mynode_id
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
#    - apache
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/

Thanks

jochen · July 19, 2018, 11:20am

You’re using a tab in line 6, which isn’t allowed in YAML.

Why does YAML forbid tabs?

Tabs have been outlawed since they are treated differently by different editors and tools. And since indentation is so critical to proper interpretation of YAML, this issue is just too tricky to even attempt. Indeed Guido van Rossum of Python has acknowledged that allowing TABs in Python source is a headache for many people and that were he to design Python again, he would forbid them.

Source: YAML Ain't Markup Language

Check out http://www.yamllint.com/ to validate your YAML files.

PitoneMaledetto · July 19, 2018, 3:15pm

Thank you!
Now I have:

INFO[0000] Using collector-id: 6d0adaf5-7c04-459c-bf30-3adbbf849b3f 
INFO[0000] Fetching configurations tagged by: [voffer]  
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/collector_sidecar.log.%Y%m%d%H%M 
DEBU[0000] Add collector backend: filebeat              
INFO[0000] Starting signal distributor                  
DEBU[0000] [signal-processor] (seq=1) handling cmd: restart 
INFO[0000] [filebeat] Starting (exec driver)            
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0000] [signal-processor] (seq=1) cmd done: restart 
ERRO[0001] [filebeat] Backend finished unexpectedly, trying to restart 1/3. 
DEBU[0001] [signal-processor] (seq=2) handling cmd: restart 
INFO[0001] [filebeat] Stopping                          
DEBU[0003] [filebeat] SIGHUP ignored, killing process   
DEBU[0003] [filebeat] Failed to kill process os: process already finished 
INFO[0003] [filebeat] Starting (exec driver)            
DEBU[0003] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0003] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0003] [signal-processor] (seq=2) cmd done: restart 
ERRO[0004] [filebeat] Backend finished unexpectedly, trying to restart 2/3. 
DEBU[0004] [signal-processor] (seq=3) handling cmd: restart 
INFO[0004] [filebeat] Stopping                          
DEBU[0006] [filebeat] SIGHUP ignored, killing process   
DEBU[0006] [filebeat] Failed to kill process os: process already finished 
INFO[0006] [filebeat] Starting (exec driver)            
DEBU[0006] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0006] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0006] [signal-processor] (seq=3) cmd done: restart 
ERRO[0007] [filebeat] Backend finished unexpectedly, trying to restart 3/3. 
DEBU[0007] [signal-processor] (seq=4) handling cmd: restart 
INFO[0007] [filebeat] Stopping                          
DEBU[0009] [filebeat] SIGHUP ignored, killing process   
DEBU[0009] [filebeat] Failed to kill process os: process already finished 
INFO[0009] [filebeat] Starting (exec driver)            
DEBU[0009] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0009] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M 
DEBU[0009] [signal-processor] (seq=4) cmd done: restart 
ERRO[0010] [filebeat] Unable to start collector after 3 tries, giving up! 
INFO[0010] [filebeat] Configuration change detected, rewriting configuration file. 
ERRO[0010] [filebeat] Error during configuration validation: filebeat2018/07/19 15:05:07.395026 beat.go:635: CRIT Exiting: error loading config file: read /etc/graylog/collector-sidecar/generated/: is a directory
Exiting: error loading config file: read /etc/graylog/collector-sidecar/generated/: is a directory
 
ERRO[0010] [filebeat] Collector configuration file is not valid, waiting for the next update. 
DEBU[0020] [RequestConfiguration] No configuration update available, skipping update. 
DEBU[0030] [RequestConfiguration] No configuration update available, skipping update. 
DEBU[0040] [RequestConfiguration] No configuration update available, skipping update.

I must mention that the Global Input in my case is GELF UDP since i would like to filter logs before sending them to Graylog/Elasticsearch. I would like to have Sidecar to take care of the configuration of Filebeat but Filebeat output should go to Logstash first.

P.S. the filebeat.yml has not been generated yet.
Thanks

jochen · July 19, 2018, 3:19pm

Compare your modified configuration to the default configuration:

github.com

Graylog2/collector-sidecar/blob/0.1.6/collector_sidecar.yml#L19-L22


      
          - name: filebeat
            enabled: true
            binary_path: /usr/bin/filebeat
            configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

PitoneMaledetto · July 19, 2018, 3:43pm

I misunderstood you when you asked me to:
“…and remove your manually crafted filebeat.yml from the configured configuration_path.”
Now is working the configuration is the following:

filebeat:
  prospectors:
  - encoding: plain
    exclude_files:
    - .gz
    fields:
      collector_node_id: mynode_id
      gl2_source_collector: 6d0adaf5-7c04-459c-bf30-3adbbf849b3f
      type: log
    ignore_older: 0
    paths:
    - /var//log/voffer*.log
    scan_frequency: 10s
    tail_files: true
    type: log
output:
  logstash:
    hosts:
    - logstash_ip:12201
path:
  data: /var/cache/graylog/collector-sidecar/filebeat/data
  logs: /var/log/graylog/collector-sidecar
tags:
- voffer

but Logstash does not receive anything…

PitoneMaledetto · July 19, 2018, 3:48pm

Silly me! Wrong port!
Now all works!
Thank you very much for your time and help!
Really appreciated

system · August 2, 2018, 3:48pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog-collector-sidecar failing Graylog Central (peer support) sidecar , filebeat-linux , nosendlogfblx , clfblx	2	3295	July 5, 2017
Graylog backend not starting Graylog Central (peer support) sidecar , filebeat-linux , nosendlogfblx , failoadcongfigfblx	2	1288	March 28, 2018
New sidecar filebeat failing Graylog Central (peer support) sidecar , filebeat-windows	2	2313	November 6, 2019
Graylog Collector Sidecar Linux Filebeat Failing Graylog Central (peer support) sidecar , filebeat-linux , filebeat-windows	2	2106	June 15, 2017
Sidecar confusion Graylog Central (peer support) sidecar , nxlog , multi-linenx	2	1740	December 6, 2017

Failed loading configuration

Why does YAML forbid tabs?

Related topics