Unable to run collector-sidecar

wlt · July 5, 2018, 11:13am

Hi graylog,

I am trying sidecar collector to work with beats to collect logs.

I am downloading tar package for sidecar because my server do not allow internet install.

Currently i have created sidecar-collector.yml and specify the settings as below.

server_url: http://graylog_server:9000/api/
update_interval: 30
tls_skip_verify: true
send_status: true
list_log_files:
  - /var/log
node_id: graylog-collector-sidecar
collector_id: file:/etc/graylog/collector-sidecar/collector-id (unsure what is this, empty file)
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
  - linux
  - apache
  - redis
backends:
    - name: filebeat
      enabled: true
      binary_path: /etc/filebeat/
      configuration_path: /etc/filebeat/filebeat.yml

When i run ./graylog-collector-sidecar,

it shows fatal error : no API token was configured.

Help is appreciated. thanks !

jan · July 5, 2018, 11:18am

What Version of Graylog did you use and what version of the collector-sidecar did you use?

wlt · July 6, 2018, 2:31am

Hi Jan,

Thanks for replying.

Graylog version : 2.4.5
Sidecar : collector-sidecar-1.0.0-alpha.1.tar
OS : Red Hat Linux 7

jan · July 6, 2018, 6:32am

Did you read the additional text when you download the collector-sidecar?

ATTENTION: Requires Graylog v3.0-alpha

You need to go with the latest stable release (at time of writing this)

wlt · July 6, 2018, 7:53am

Hi Jan,

Thanks for the reply, i managed to run collector. However i encounter this error.

level=error msg="[UpdateRegistration] Bad response from Graylog server: 405 Method Not Allowed"
level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found"
level=error msg=“Can’t fetch configuration from Graylog API: GET http://graylog_server:9000/api/plugins/org.graylog.plugins.collector?tags=%5B%22linux%22%5D: 404 HTTP 404 Not Found”

My collector_sidecar.yml is as follows :

server_url: http://graylog_server:9000/api/
update_interval: 10
tls_skip_verify: false
send_status: true
list_log_files:
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

Graylog : 2.4.5
Sidecar : 0.1.6

jochen · July 6, 2018, 8:10am

Is http://graylog_server:9000/api/ the correct URI of the Graylog REST API?
Is the Collector plugin installed in your Graylog cluster? It’s being shipped by default, but check on the System/Node/Details page.

What’s the complete output of the following command on the machine running the Collector Sidecar?

# curl -i -H 'Accept: application/json' 'http://graylog_server:9000/api/?pretty=true'

wlt · July 6, 2018, 8:12am

Hi Jochen,

Thanks for the reply.

Yes, collector plugin is there. I able to see collector under system/node.

Below is the results of the curl request :

HTTP/1.1 200 OK
X-Graylog-Node-ID: 573cc295-80f5-4447-8e71-cb29cc0e525b
X-Runtime-Microseconds: 1313
Content-Type: application/json
Date: Fri, 06 Jul 2018 08:11:34 GMT
Content-Length: 253

{
  "cluster_id" : "f170227c-9520-42fa-bbdf-ee6d48b907d9",
  "node_id" : "573cc295-80f5-4447-8e71-cb29cc0e525b",
  "version" : "2.4.5+8e18e6a",
  "tagline" : "Manage your logs in the dark and have lasers going and make it look like you're from space!"
}

Thanks again.

jochen · July 6, 2018, 8:38am

What are you seeing exactly?

wlt · July 6, 2018, 8:43am

Under the web interface, system -> nodes - > details -> installed plugins

-> Collector is there.

Thanks.

Edited : added graylog server initialization message.

2018-07-06 16:07:13,104 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at http://graylog_server:9000/api/
2018-07-06 16:07:13,105 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at http://graylog_server:9000/

jochen · July 6, 2018, 8:53am

Please post the complete content of the plugin section.

What’s the complete output of the following command on the machine running the Collector Sidecar (you have to use your own admin credentials)?

# curl -i -u admin:password -H 'Accept: application/json' 'http://graylog_server:9000/api/plugins/org.graylog.plugins.collector/configurations/tags'

wlt · July 6, 2018, 9:03am

Hi,

The curl responds as below :

HTTP/1.1 200 OK
X-Graylog-Node-ID: 573cc295-80f5-4447-8e71-cb29cc0e525b
Content-Type: application/json
Date: Fri, 06 Jul 2018 09:00:38 GMT
Content-Length: 9

["linux"]

The complete content of the plugin section as below :

Installed plugins 9 plugins installed

Name	Version	Author	Description
AWS plugins	2.4.5	Graylog, Inc.	Collection of plugins to read data from or interact with the Amazon Web Services (AWS).  Website 
CEF Input	2.4.5	Graylog, Inc.	Input plugin to receive CEF (Common Event Format) messages.  Website 
Collector	2.4.5	Graylog, Inc.	Collectors plugin  Website 
Elastic Beats Input	2.4.5	Graylog, Inc.	Input plugin for Elastic Beats (Beats/Lumberjack protocol).  Website 
Enterprise Integration Plugin	2.4.5	Graylog, Inc	Provides basic integration with Graylog Enterprise  Website 
MapWidgetPlugin	2.4.5	Graylog, Inc.	Map widget for Graylog  Website 
NetFlow Plugin	2.4.5	Graylog, Inc.	Provides NetFlow inputs  Website 
Pipeline Processor Plugin	2.4.5	Graylog, Inc.	Pluggable pipeline processing framework  Website 
Threat Intelligence Plugin	2.4.5	Graylog, Inc.	Threat intelligence database lookup functions for the Graylog Pipeline Processor  Website

This is the log generated when i start sidecar by using

./graylog-collector-sidecar

INFO[0000] No node-id was configured, falling back to hostname
INFO[0000] Fetching configurations tagged by: [linux]
INFO[0000] Starting signal distributor
INFO[0000] [filebeat] Starting (exec driver)
ERRO[0001] [filebeat] Backend finished unexpectedly, trying to restart 1/3.
INFO[0001] [filebeat] Stopping
INFO[0003] [filebeat] Starting (exec driver)
ERRO[0004] [filebeat] Backend finished unexpectedly, trying to restart 2/3.
INFO[0004] [filebeat] Stopping
INFO[0006] [filebeat] Starting (exec driver)
ERRO[0007] [filebeat] Backend finished unexpectedly, trying to restart 3/3.
INFO[0007] [filebeat] Stopping
INFO[0009] [filebeat] Starting (exec driver)
ERRO[0010] [filebeat] Unable to start collector after 3 tries, giving up!
ERRO[0010] [RequestConfiguration] Bad response status from Graylog server: 404 Not Found
ERRO[0010] Can't fetch configuration from Graylog API: GET http://graylog_server:9000/api/plugins/org.graylog.plugins.collector?tags=%5B%22linux%22%5D: 404 HTTP 404 Not Found
ERRO[0010] [UpdateRegistration] Bad response from Graylog server: 405 Method Not Allowed

jochen · July 6, 2018, 9:54am

Are you running a load-balancer or a reverse proxy in front of Graylog’s REST API?
What’s the content of /etc/graylog/collector-sidecar/collector-id? See http://docs.graylog.org/en/2.4/pages/collector_sidecar.html#configuration for details.
What’s the output of the following commands on the machine running the Collector Sidecar?

# namei -l /etc/graylog/collector-sidecar/collector-id
# stat /etc/graylog/collector-sidecar/collector-id
# hostname
# hostname -f
# hostname -a

wlt · July 6, 2018, 10:04am

Hi,

When i run

vi /etc/graylog/collector-sidecar/collector-id

it is empty, do i need to fill any details inside ?

for # namei -l /etc/graylog/collector-sidecar/collector-id

f: /etc/graylog/collector-sidecar/collector-id
dr-xr-xr-x root root /
drwxr-xr-x root root etc
drwxr-xr-x root root graylog
drwxr-xr-x root root collector-sidecar
-rw-r----- root root collector-id

for # stat /etc/graylog/collector-sidecar/collector-id

  File: ‘/etc/graylog/collector-sidecar/collector-id’
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: fd00h/64768d    Inode: 7477        Links: 1
Access: (0640/-rw-r-----)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-07-06 09:18:49.023106661 +0200
Modify: 2018-07-06 09:17:48.976895852 +0200
Change: 2018-07-06 09:17:48.976895852 +0200
 Birth: -

for hostname return xxx hostname,
hostname -f return xxx.xxx.com.
hostname -a return xxx

marius · July 6, 2018, 10:10am

Hi,
yes the collector-id should contain a uuid that identifies this instance. You can remove the file and let the Sidecar create a fresh ID for you (needs write access to /etc/graylog/collector-sidecar for the sidecar process).

wlt · July 6, 2018, 10:16am

Hi Marius.

Thanks for your reply, i removed the existing collector-id.

When i restart the collector service, i receive this error.

INFO[0000] collector-id file doesn't exist, generating a new one
INFO[0000] Using collector-id: c173dd92-c546-4892-903d-51f98d5938bb
INFO[0000] No node-id was configured, falling back to hostname
INFO[0000] Fetching configurations tagged by: [linux]
INFO[0000] Starting signal distributor
INFO[0000] [filebeat] Starting (exec driver)
ERRO[0001] [filebeat] Backend finished unexpectedly, trying to restart 1/3.
INFO[0001] [filebeat] Stopping
INFO[0003] [filebeat] Starting (exec driver)
ERRO[0004] [filebeat] Backend finished unexpectedly, trying to restart 2/3.
INFO[0004] [filebeat] Stopping
INFO[0006] [filebeat] Starting (exec driver)
ERRO[0007] [filebeat] Backend finished unexpectedly, trying to restart 3/3.
INFO[0007] [filebeat] Stopping
INFO[0009] [filebeat] Starting (exec driver)
ERRO[0010] [filebeat] Unable to start collector after 3 tries, giving up!
ERRO[0010] stat /var/cache/graylog: no such file or directory
INFO[0010] Trying to create directory for: /var/cache/graylog/collector-sidecar
INFO[0010] [filebeat] Configuration change detected, rewriting configuration file.
ERRO[0010] [filebeat] Error during configuration validation: Flag --configtest has been deprecated, configtest flag has been deprecated, use test config subcommand
Exiting: error loading config file: config file ("/etc/graylog/collector-sidecar/generated/filebeat.yml") must be owned by the beat user (uid=0) or root

ERRO[0010] [filebeat] Collector configuration file is not valid, waiting for the next update.

How should i proceed? Thanks again.

Edited : i removed the filebeat.yml and restart the service. Now everything is working. Thanks lot Graylog !

marius · July 6, 2018, 10:22am

Make sure that you use the beat version that is bundled with the sidecar version, looks like you are using a newer version where the command line options did change.

Run the Sidecar as root user to make sure the generated configuration file is accepted by the beat process.

wlt · July 6, 2018, 10:23am

Hi Marius,

I removed the filebeat.yml from generated and rerun the service again.

Now everything is working as of now.

Thanks again for support.

jochen · July 6, 2018, 10:49am

For reference:
https://github.com/Graylog2/collector-sidecar/issues/253

system · July 20, 2018, 10:49am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog backend not starting Graylog Central (peer support) sidecar , filebeat-linux , nosendlogfblx , failoadcongfigfblx	2	1225	March 28, 2018
Graylog Collector Sidecar Linux Filebeat Failing Graylog Central (peer support) sidecar , filebeat-linux , filebeat-windows	2	2034	June 15, 2017
Failed loading configuration Graylog Central (peer support) sidecar , filebeat-linux , nosendlogfblx , failoadcongfigfblx	15	5099	August 2, 2018
Collector side car using filebeat Graylog Central (peer support) sidecar , filebeat-linux , nosendlogfblx , failoadcongfigfblx	23	4993	April 12, 2018
Sidecar 1.5 can't start the filebeat that it is shipped with Graylog Central (peer support) filebeat-linux	2	243	December 1, 2023

Unable to run collector-sidecar

Related Topics