Collector side car using filebeat


(EL HIJAZI) #1

Hello to everyone …
hope you all doing great today , well i am confused with the documentation about collector sidecar :
http://docs.graylog.org/en/latest/pages/collector_sidecar.html

  1. i think i don’t need to install filebeat because it’s installed with collector-sidecar_xxxx.deb ??? is it right ? because i can’t find filebeat installed ? :confused:

i can’t find “graylog-collector-sedecar”

any help please
i m using ubuntu 14.04

and this is my conf file :
> >

> server_url: http://192.168.111.132:9000/api/
> update_interval: 30
> tls_skip_verify: true
> send_status: true
> list_log_files:
>         -  /var/log
> node_id: graylog-collector-sidecar
> collector_id: file:/etc/graylog/collector-sidecar/collector-id
> cache_path: /var/cache/graylog/collector-sidecar
> log_path: /var/log/graylog/collector-sidecar
> log_rotation_time: 86400
> log_max_age: 604800
> tags:
>     - linux
>     - apache
> backends:
>     - name: nxlog
>       enabled: false
>       binary_path: /usr/bin/nxlog
>       configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
>     - name: filebeat
>       enabled: true
>       binary_path: /usr/bin/filebeat
>       configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

Graylog backend not starting
(Jan Doberstein) #2

Filebeat is installed with the collector-sidecar - for your reference:

# dpkg-deb -c collector-sidecar_0.1.4-1_amd64.deb
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./var/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./var/spool/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./var/spool/collector-sidecar/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./var/spool/collector-sidecar/nxlog/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./var/run/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./var/run/graylog/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./var/run/graylog/collector-sidecar/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./var/log/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./var/log/graylog/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./var/log/graylog/collector-sidecar/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./usr/
drwxr-xr-x 0/0               0 2017-07-31 15:16 ./usr/share/
drwxr-xr-x 0/0               0 2017-07-31 15:16 ./usr/share/doc/
drwxr-xr-x 0/0               0 2017-07-31 15:16 ./usr/share/doc/collector-sidecar/
-rw-r--r-- 0/0             154 2017-07-31 15:16 ./usr/share/doc/collector-sidecar/changelog.gz
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./usr/bin/
-rwxrwxr-x 0/0         6161120 2017-07-31 15:16 ./usr/bin/graylog-collector-sidecar
-rwxr-xr-x 0/0        15953319 2017-07-31 15:16 ./usr/bin/filebeat
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./etc/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./etc/graylog/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./etc/graylog/collector-sidecar/
drwxrwxr-x 0/0               0 2017-07-31 15:16 ./etc/graylog/collector-sidecar/generated/
-rw-rw-r-- 0/0             703 2017-07-31 15:16 ./etc/graylog/collector-sidecar/collector_sidecar.yml

What step did you refer to when you say

i can’t find “graylog-collector-sedecar”

I can’t follow.

Please elaborate some more on what guide you follow and where you have trouble to follow the guide.

thank you
Jan


(EL HIJAZI) #3

i am following the documentation ,
./graylog-collector-sidecar i can’t find it :confused:

2 i installed collector-sidecar_0.1.5-1_amd64.deb and belive me i coudn’t find
filebeat in the path /usr/bin

well maybe using another version may help can you just how tu uninstall all the collector to reinstall a new one please ?


(Jan Doberstein) #4

So I guess you have followed the following:

http://docs.graylog.org/en/2.4/pages/collector_sidecar.html#ubuntu

What part of the commands isn’t working for you? I see in the description no ./graylog-collector-sidecar that is why I ask what part did you follow.


(EL HIJAZI) #5

yes sir i’m using this part ( ubuntu 16.04 now i give up with 14.04 because i coudn’t uninstall it correctly and now i m trying with this 16.04 i m just in training session so i ve 0 experience please don’t judge me ! )
well !
now after doing every thing in the documentation :

and the configuration :

server_url: http://192.168.111.132:9000/api/
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files:
- /var/log
node_id: graylog-collector-sidecar
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
- linux
- apache
backends:
- name: nxlog
enabled: false
binary_path: /usr/bin/nxlog
configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
- name: filebeat
enabled: true
binary_path: /usr/bin/filebeat
configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

but the result is :frowning: :
image

please can you help more :confused:


(Jan Doberstein) #6

you do not have any tag configured (in the screenshot) and that is the reason why no configuration file is written.

Check the 7th Picture in the step-by-step guide

http://docs.graylog.org/en/2.4/pages/collector_sidecar.html#step-by-step-guide


(EL HIJAZI) #7

yes you right sir !
First thank you for your help but unfortunatly it does not seem to solve my probléme ,
btw where i am supposed to put those lines :

collector_sidecar_cache_time = 2h
collector_sidecar_cache_max_size = 500
it’s the only thing i suspect i m not doing correctly :confused:


(Jan Doberstein) #8

is your Graylog available at http://192.168.111.132:9000/api/ ? The screenhot of your Graylog did not show that.

the server_url in the collector need to be the rest_listen_uri of your Graylog server. Did you see the collector on the above visible page ( system / collectors ) when you hit the overview button?

Why did you think you need to place the both settings somewhere?


(EL HIJAZI) #9

http://graylog.ddns.net == refer to 192.168.111.132
when i go to http://graylog.ddns.net/api or 192.168.111.132/api
i got redirected to http://graylog.ddns.net or 192.168.111.132/

it’s normal ?
and with curl


(EL HIJAZI) #10

no i don’t
and when i hit to include inactive ones


(EL HIJAZI) #11

and here i am confused 192.168.111.134 is the ip of the fist machine that i give up
now i m lanching the collector from 192.168.111.137 :confused:


(Jan Doberstein) #12

when your collector did not show up on the collectos overview - inactive means it does not run and that is saved for historical reasons - do you check if the collector sidecar is actually started?


(EL HIJAZI) #13

yes you right i don’t now why the collector stops working her is the erreur that i found in the log file of the collecteur

i don’t know from why he go for 127.0.0.1:9000/api :confused:

So sorry this probléme will make you creazy
and thank you a lot ! for all your help !


(EL HIJAZI) #14

good morning i’m facing a lot of issues of the ‘collector sidecar’
well now i m instaling
collector-sidecar_0.1.4-1_i386.deb

my config file :

ub1632@ubuntu:~$ cat /etc/graylog/collector-sidecar/collector_sidecar.yml
server_url: http://192.168.111.132:9000/api/
update_interval: 30
tls_skip_verify: true
send_status: true
list_log_files:
- /var/log
node_id: graylog-collector-sidecar
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
- linux
- apache
backends:
- name: nxlog
enabled: false
binary_path: /usr/bin/nxlog
configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
- name: filebeat
enabled: true
binary_path: /usr/bin/filebeat
configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

the result of
systemctl status collector-sidecar

ub1632@ubuntu:~$ systemctl status collector-sidecar
● collector-sidecar.service - Wrapper service for Graylog controlled collector
Loaded: loaded (/etc/systemd/system/collector-sidecar.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) since Wed 2018-03-28 00:57:08 PDT; 22s ago
Process: 1835 ExecStart=/usr/bin/graylog-collector-sidecar (code=exited, status=0/SUCCESS)
Main PID: 1835 (code=exited, status=0/SUCCESS)
the erreur in log file :
time=“2018-03-27T07:07:17-07:00” level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://127.0.0.1:9000/api/plugins/org.graylog.plugins.collector/collectors/68627057-36d5-479a-b32e-f3bc68cc6781: dial tcp 127.0.0.1:9000: getsockopt: connection refused"
time=“2018-03-27T07:07:27-07:00” level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://127.0.0.1:9000/api/plugins/org.graylog.plugins.collector/68627057-36d5-479a-b32e-f3bc68cc6781?tags=[“linux”%2C"apache"]: dial tcp 127.0.0.1:9000: getsockopt: connection refused"
time=“2018-03-27T07:07:27-07:00” level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://127.0.0.1:9000/api/plugins/org.graylog.plugins.collector/collectors/68627057-36d5-479a-b32e-f3bc68cc6781: dial tcp 127.0.0.1:9000: getsockopt: connection refused"
time=“2018-03-27T07:07:31-07:00” level=info msg=“Stopping signal distributor”

thank you for your help !


(EL HIJAZI) #15

any idea please ? i m still strugling with this probléme :confused:


(marius) #16

Hey Amine,
for some reason the Sidecar is using no/or the wrong configuration file. You said that you configured the server_url to http://192.168.111.132:9000/api/ but in the logs you can see that it tries to use the default value: http://127.0.0.1:9000/api

The Sidecar gets the path to configuration file vie the -c parameter, could check in the process list how the Sidecar was started?

Cheers,
Marius


(EL HIJAZI) #17

thank you very much ya you give me a very good hint i ll check right now thank you again Marius :smiley: !


(EL HIJAZI) #18

still it’s not working :confused:

any idea ?
P.S: if you know how to uninstall all the collector to reinstall another it may help to do it in the hardest way :stuck_out_tongue: cauz now i m just learning :confused: and i m just a very begginer…


(Jochen) #19

What’s the complete output of the following commands?

# sudo stat /etc/graylog/collector-sidecar/collector_sidecar.yml
# sudo namei -l /etc/graylog/collector-sidecar/collector_sidecar.yml
# sudo cat /etc/systemd/system/collector-sidecar.service

(EL HIJAZI) #20

thank you for your replie !
the result :

ub1632@ubuntu:~$ sudo stat /etc/graylog/collector-sidecar/collector_sidecar.yml
[sudo] password for ub1632:
File: ‘/etc/graylog/collector-sidecar/collector_sidecar.yml’
Size: 720 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 133014 Links: 1
Access: (0664/-rw-rw-r–) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-03-28 04:01:33.748651647 -0700
Modify: 2018-03-27 03:59:55.682990712 -0700
Change: 2018-03-27 03:59:55.686990954 -0700
Birth: -

AND

ub1632@ubuntu:~$ sudo namei -l /etc/graylog/collector-sidecar/collector_sidecar.yml
f: /etc/graylog/collector-sidecar/collector_sidecar.yml
drwxr-xr-x root root /
drwxr-xr-x root root etc
drwxrwxr-x root root graylog
drwxrwxr-x root root collector-sidecar
-rw-rw-r-- root root collector_sidecar.yml

AND

ub1632@ubuntu:~$ sudo cat /etc/systemd/system/collector-sidecar.service
[Unit]
Description=Wrapper service for Graylog controlled collector
ConditionFileIsExecutable=/usr/bin/graylog-collector-sidecar

[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/bin/graylog-collector-sidecar

Restart=always
RestartSec=120
EnvironmentFile=-/etc/sysconfig/collector-sidecar

[Install]
WantedBy=multi-user.target
ub1632@ubuntu:~$