Collector_sidecar failed to send data to Graylog application


(Ganeshbabu Ramamoorthy) #1

Hi All,

I am currently facing an issue in sending data through collector sidecar using beats with SSL setup.

Consider this as our scenario,

I have two instances in which first instance have only running collector sidecar and the second instance are running Graylog application with SSL setup.

I am trying to collect and send data from first instance collector_sidecar to my second instance graylog application and in which I made the changes in collector_sidecar.yml by giving the server URL as (https://graylogserver.com:9000/api/) and after restarting the collector sidecar in first instance I got an error by saying graylog-certificate.pem & graylogkey.pem files were not found in the directory, so then I copied those two files from second instance and kept in the respective path of the first instance and then file not found issue were resolved, but now I am currently getting the below error something related to certificate format,

time="2017-12-11T18:02:09Z" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put https://graylogserver.com:9000/api/plugins/org.graylog.plugins.collector/collectors/a61675b7-17f3-4b20-8a0c-9d1b929bb53b: x509: certificate signed by unknown authority"
time="2017-12-11T18:02:15Z" level=error msg="[RequestConfiguration] Fetching configuration failed: Get https://graylogserver.com:9000/api/plugins/org.graylog.plugins.collector/a61675b7-17f3-4b20-8a0c-9d1b929bb53b?tags=%5B%22linux%22%2C%22apache%22%2C%22graylogserver%22%5D: x509: certificate signed by unknown authority"

Any help would be really appreciated…

First instance collector_sidecar.yml file,

server_url: https://graylogserver.com:9001/api/
update_interval: 10
tls_skip_verify: false
send_status: true
list_log_files:
node_id: graylog-collector-sidecar_dev
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
    - apache
    - graylogserver
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

Filebeat also got generated in the respective configuration_path,

filebeat:
  prospectors:
  - encoding: plain
    exclude_lines:
    - Ticket
    fields:
      collector_node_id: graylog-collector-sidecar
      data: example.org
      gl2_source_collector: a61675b7-17f3-4b20-8a0c-9d1b929bb53b
      type: log
    ignore_older: 0
    paths:
    - /home/ubuntu/*.csv
    scan_frequency: 10s
    tail_files: false
    type: log
output:
  logstash:
    hosts:
    - graylogserver.com:5044
    ssl:
      certificate: /etc/graylogserver/cluster1/graylog-certificate.pem
      key: /etc/graylogserver/cluster1/graylogkey.pem
      verification_mode: none
path:
  data: /var/cache/graylog/collector-sidecar/filebeat/data
  logs: /var/log/graylog/collector-sidecar
tags:
- linux
- apache
- graylogserver

Please correct me If I am doing anything wrong in the setup.

Regards,
Ganeshbabu R


(Jan Doberstein) #2

as you use a self signed certificate, you should give collector sidecar that Information and skip the verification

tls_skip_verify: true

will be your friend.


(Ganeshbabu Ramamoorthy) #3

@jan

Yes, If I set tls_skip_verify has true and I can able to send data and I could see in the filebeat log file harvest happening,

2017-12-12T08:01:34Z INFO Harvester started for file: /home/ubuntu/graylog.csv
2017-12-12T08:01:34Z WARN SSL/TLS verifications disabled.
2017-12-12T08:02:04Z INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=5 libbeat.logstash.publish.read_bytes=2469 libbeat.logstash.publish.write_bytes=1963971 libbeat.logstash.published_and_acked_events=6747 libbeat.publisher.published_events=6747 publish.events=10002 registrar.states.current=1 registrar.states.update=10002 registrar.writes=5

but the strange thing is I couldn’t find the data in my graylog application.

Is there any other way to debug this?

Thanks,
Ganeshbabu R


(Jan Doberstein) #4

did you see some counts in your Graylog Interface on the Beats Input that is the target of your collector? Did you check “show messages from that input” and choose “all messages” to check if maybe the time is an issue?


(Ganeshbabu Ramamoorthy) #5

No @jan. I couldn’t see any count in the beats input

did you see some counts in your Graylog Interface on the Beats Input that is the target of your collector? Did you check “show messages from that input” and choose “all messages” to check if maybe the time is an issue?

and also I did checked the show messages in that input but the screen has loading for long time and I couldn’t see any messages.

as you use a self signed certificate, you should give collector sidecar that Information and skip the verification

@jan we are not using self signed certificate and we bought the authorized cert & key from third party.

Is the collector_sidecar configuration needs to be changed related to valid certificates?

Thanks,
Ganeshbabu R


(Ganeshbabu Ramamoorthy) #6

@jan

Any thoughts from your end I am trying to resolve it but still I am getting no data in graylog?

Let me know if you need any further informations.

Regards,
Ganeshbabu R


(Matt) #7

Looks like sidecar may not be your problem here at this point but maybe your beat configuration. Make sure that it is correct and shipping as well. If not checking your beat logs may give you insight into where it is failing.

If your beats configuration is making it from graylog to your endpoint server the filebeat configuration should reflect the changes. If not sidecar isn’t communicating with graylog correctly or at all. If the filebeat config is making it from graylog just fine but from there you aren’t seeing logs being shipped then the problem lies within the beat configuration. You have a corresponding beat input setup and listening on the correct port on the graylog side right? Correct input/outputs on the filebeat side? Double check everything. Devil is in the details here.


(Ganeshbabu Ramamoorthy) #8

@Mantil
The error where in beats configuration and I made the corrections and were working fine without issue and able to see the data in graylog.


(Matt) #9

Awesome. Glad you got it all working.


(system) #10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.