Unable to run collector-sidecar


#1

Hi graylog,

I am trying sidecar collector to work with beats to collect logs.

I am downloading tar package for sidecar because my server do not allow internet install.

Currently i have created sidecar-collector.yml and specify the settings as below.

server_url: http://graylog_server:9000/api/
update_interval: 30
tls_skip_verify: true
send_status: true
list_log_files:
  - /var/log
node_id: graylog-collector-sidecar
collector_id: file:/etc/graylog/collector-sidecar/collector-id (unsure what is this, empty file)
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
  - linux
  - apache
  - redis
backends:
    - name: filebeat
      enabled: true
      binary_path: /etc/filebeat/
      configuration_path: /etc/filebeat/filebeat.yml

When i run ./graylog-collector-sidecar,

it shows fatal error : no API token was configured.

Help is appreciated. thanks !


(Jan Doberstein) #2

What Version of Graylog did you use and what version of the collector-sidecar did you use?


#3

Hi Jan,

Thanks for replying.

Graylog version : 2.4.5
Sidecar : collector-sidecar-1.0.0-alpha.1.tar
OS : Red Hat Linux 7


(Jan Doberstein) #4

Did you read the additional text when you download the collector-sidecar?

ATTENTION: Requires Graylog v3.0-alpha

You need to go with the latest stable release (at time of writing this)


#5

Hi Jan,

Thanks for the reply, i managed to run collector. However i encounter this error.

level=error msg="[UpdateRegistration] Bad response from Graylog server: 405 Method Not Allowed"
level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found"
level=error msg=“Can’t fetch configuration from Graylog API: GET http://graylog_server:9000/api/plugins/org.graylog.plugins.collector?tags=%5B%22linux%22%5D: 404 HTTP 404 Not Found”

My collector_sidecar.yml is as follows :

server_url: http://graylog_server:9000/api/
update_interval: 10
tls_skip_verify: false
send_status: true
list_log_files:
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

Graylog : 2.4.5
Sidecar : 0.1.6


(Jochen) #6

Is http://graylog_server:9000/api/ the correct URI of the Graylog REST API?
Is the Collector plugin installed in your Graylog cluster? It’s being shipped by default, but check on the System/Node/Details page.

What’s the complete output of the following command on the machine running the Collector Sidecar?

# curl -i -H 'Accept: application/json' 'http://graylog_server:9000/api/?pretty=true'

#7

Hi Jochen,

Thanks for the reply.

Yes, collector plugin is there. I able to see collector under system/node.

Below is the results of the curl request :

HTTP/1.1 200 OK
X-Graylog-Node-ID: 573cc295-80f5-4447-8e71-cb29cc0e525b
X-Runtime-Microseconds: 1313
Content-Type: application/json
Date: Fri, 06 Jul 2018 08:11:34 GMT
Content-Length: 253

{
  "cluster_id" : "f170227c-9520-42fa-bbdf-ee6d48b907d9",
  "node_id" : "573cc295-80f5-4447-8e71-cb29cc0e525b",
  "version" : "2.4.5+8e18e6a",
  "tagline" : "Manage your logs in the dark and have lasers going and make it look like you're from space!"
} 

Thanks again.


(Jochen) #8

What are you seeing exactly?


#9

Under the web interface, system -> nodes - > details -> installed plugins

-> Collector is there.

Thanks.

Edited : added graylog server initialization message.

2018-07-06 16:07:13,104 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at http://graylog_server:9000/api/
2018-07-06 16:07:13,105 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at http://graylog_server:9000/


(Jochen) #10

Please post the complete content of the plugin section.

What’s the complete output of the following command on the machine running the Collector Sidecar (you have to use your own admin credentials)?

# curl -i -u admin:password -H 'Accept: application/json' 'http://graylog_server:9000/api/plugins/org.graylog.plugins.collector/configurations/tags'

#11

Hi,

The curl responds as below :

HTTP/1.1 200 OK
X-Graylog-Node-ID: 573cc295-80f5-4447-8e71-cb29cc0e525b
Content-Type: application/json
Date: Fri, 06 Jul 2018 09:00:38 GMT
Content-Length: 9

["linux"] 

The complete content of the plugin section as below :

Installed plugins 9 plugins installed

Name	Version	Author	Description
AWS plugins	2.4.5	Graylog, Inc.	Collection of plugins to read data from or interact with the Amazon Web Services (AWS).  Website 
CEF Input	2.4.5	Graylog, Inc.	Input plugin to receive CEF (Common Event Format) messages.  Website 
Collector	2.4.5	Graylog, Inc.	Collectors plugin  Website 
Elastic Beats Input	2.4.5	Graylog, Inc.	Input plugin for Elastic Beats (Beats/Lumberjack protocol).  Website 
Enterprise Integration Plugin	2.4.5	Graylog, Inc	Provides basic integration with Graylog Enterprise  Website 
MapWidgetPlugin	2.4.5	Graylog, Inc.	Map widget for Graylog  Website 
NetFlow Plugin	2.4.5	Graylog, Inc.	Provides NetFlow inputs  Website 
Pipeline Processor Plugin	2.4.5	Graylog, Inc.	Pluggable pipeline processing framework  Website 
Threat Intelligence Plugin	2.4.5	Graylog, Inc.	Threat intelligence database lookup functions for the Graylog Pipeline Processor  Website 

This is the log generated when i start sidecar by using

./graylog-collector-sidecar

INFO[0000] No node-id was configured, falling back to hostname
INFO[0000] Fetching configurations tagged by: [linux]
INFO[0000] Starting signal distributor
INFO[0000] [filebeat] Starting (exec driver)
ERRO[0001] [filebeat] Backend finished unexpectedly, trying to restart 1/3.
INFO[0001] [filebeat] Stopping
INFO[0003] [filebeat] Starting (exec driver)
ERRO[0004] [filebeat] Backend finished unexpectedly, trying to restart 2/3.
INFO[0004] [filebeat] Stopping
INFO[0006] [filebeat] Starting (exec driver)
ERRO[0007] [filebeat] Backend finished unexpectedly, trying to restart 3/3.
INFO[0007] [filebeat] Stopping
INFO[0009] [filebeat] Starting (exec driver)
ERRO[0010] [filebeat] Unable to start collector after 3 tries, giving up!
ERRO[0010] [RequestConfiguration] Bad response status from Graylog server: 404 Not Found
ERRO[0010] Can't fetch configuration from Graylog API: GET http://graylog_server:9000/api/plugins/org.graylog.plugins.collector?tags=%5B%22linux%22%5D: 404 HTTP 404 Not Found
ERRO[0010] [UpdateRegistration] Bad response from Graylog server: 405 Method Not Allowed

(Jochen) #12

Are you running a load-balancer or a reverse proxy in front of Graylog’s REST API?
What’s the content of /etc/graylog/collector-sidecar/collector-id? See http://docs.graylog.org/en/2.4/pages/collector_sidecar.html#configuration for details.
What’s the output of the following commands on the machine running the Collector Sidecar?

# namei -l /etc/graylog/collector-sidecar/collector-id
# stat /etc/graylog/collector-sidecar/collector-id
# hostname
# hostname -f
# hostname -a

#13

Hi,

When i run

vi /etc/graylog/collector-sidecar/collector-id

it is empty, do i need to fill any details inside ?

for # namei -l /etc/graylog/collector-sidecar/collector-id

f: /etc/graylog/collector-sidecar/collector-id
dr-xr-xr-x root root /
drwxr-xr-x root root etc
drwxr-xr-x root root graylog
drwxr-xr-x root root collector-sidecar
-rw-r----- root root collector-id

for # stat /etc/graylog/collector-sidecar/collector-id

  File: ‘/etc/graylog/collector-sidecar/collector-id’
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: fd00h/64768d    Inode: 7477        Links: 1
Access: (0640/-rw-r-----)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-07-06 09:18:49.023106661 +0200
Modify: 2018-07-06 09:17:48.976895852 +0200
Change: 2018-07-06 09:17:48.976895852 +0200
 Birth: -

for hostname return xxx hostname,
hostname -f return xxx.xxx.com.
hostname -a return xxx


(marius) #14

Hi,
yes the collector-id should contain a uuid that identifies this instance. You can remove the file and let the Sidecar create a fresh ID for you (needs write access to /etc/graylog/collector-sidecar for the sidecar process).


#15

Hi Marius.

Thanks for your reply, i removed the existing collector-id.

When i restart the collector service, i receive this error.

INFO[0000] collector-id file doesn't exist, generating a new one
INFO[0000] Using collector-id: c173dd92-c546-4892-903d-51f98d5938bb
INFO[0000] No node-id was configured, falling back to hostname
INFO[0000] Fetching configurations tagged by: [linux]
INFO[0000] Starting signal distributor
INFO[0000] [filebeat] Starting (exec driver)
ERRO[0001] [filebeat] Backend finished unexpectedly, trying to restart 1/3.
INFO[0001] [filebeat] Stopping
INFO[0003] [filebeat] Starting (exec driver)
ERRO[0004] [filebeat] Backend finished unexpectedly, trying to restart 2/3.
INFO[0004] [filebeat] Stopping
INFO[0006] [filebeat] Starting (exec driver)
ERRO[0007] [filebeat] Backend finished unexpectedly, trying to restart 3/3.
INFO[0007] [filebeat] Stopping
INFO[0009] [filebeat] Starting (exec driver)
ERRO[0010] [filebeat] Unable to start collector after 3 tries, giving up!
ERRO[0010] stat /var/cache/graylog: no such file or directory
INFO[0010] Trying to create directory for: /var/cache/graylog/collector-sidecar
INFO[0010] [filebeat] Configuration change detected, rewriting configuration file.
ERRO[0010] [filebeat] Error during configuration validation: Flag --configtest has been deprecated, configtest flag has been deprecated, use test config subcommand
Exiting: error loading config file: config file ("/etc/graylog/collector-sidecar/generated/filebeat.yml") must be owned by the beat user (uid=0) or root

ERRO[0010] [filebeat] Collector configuration file is not valid, waiting for the next update.

How should i proceed? Thanks again.

Edited : i removed the filebeat.yml and restart the service. Now everything is working. Thanks lot Graylog !


(marius) #16

Make sure that you use the beat version that is bundled with the sidecar version, looks like you are using a newer version where the command line options did change.

Run the Sidecar as root user to make sure the generated configuration file is accepted by the beat process.


#17

Hi Marius,

I removed the filebeat.yml from generated and rerun the service again.

Now everything is working as of now.

Thanks again for support.


(Jochen) #18

For reference:


(system) #19

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.