Not able to get logs from sidecar

Hello,
I am completely new in this area.
I have followed the 3.3.8 documentation and have installed Graylog server and sidecar on a client machine.

  1. I am able to log on to the server using http://localhost:9000/.
  2. I created a beats input with all the defaults
  3. I created a new sidecar configuration using provided winlogbeat log collector.
  4. I changed hosts ip address to match my server IP address:
    output.logstash:
    hosts: [“192.168.0.107:5044”]
  5. I created a token from Sidecar System User (built-in).
  6. I entered the server IP address and this token in sidecar.yml
  7. On system/sidecars->Administration, my client is displayed as Inactive.
  8. I can ping between the Graylog server and win 10 client (both ways).
  9. On the Win 10 client, if I look at logs, I am getting following error:
    connectex: No connection could be made because the target machine actively refused it."
    time=“2020-11-04T13:53:55-05:00”
    level=error msg="[UpdateRegistration] Failed to report collector status to server:
    Put http://127.0.0.1:9000/api/sidecars/92860565-36c4-4010-93bc-d07d6113e058:
    dial tcp 127.0.0.1:9000:
  10. I really don’t know what mistake I am making.
    Any help is really appreciated. Thank you for your time.
    Thanks and regards,
    Dushyant

Seems your Graylog is listening on localhost only, so another host can’t reach it.
Change Graylog address to another IP from you local network range

Thank you. I will try that when I go back to work on Monday. Have a good weekend.

Hi,
IP address of the machine on which I have installed Graylog server: 192.168.0.107
IP address of Windows 10 client with sidecar installed: 192.168.0.101
Also, I can ping both machines from each other.
You think these IP addresses need to be changed?
Thanks…

check output of sudo lsof -Pni :9000

I entered that command on my Graylog Server and received following output:
smc@smc-B325:~$ sudo lsof -Pni :9000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 978 graylog 54u IPv6 42471 0t0 TCP 127.0.0.1:9000 (LISTEN)
java 978 graylog 72u IPv6 55565 0t0 TCP 127.0.0.1:9000->127.0.0.1:45978 (ESTABLISHED)
java 978 graylog 75u IPv6 55568 0t0 TCP 127.0.0.1:9000->127.0.0.1:45984 (ESTABLISHED)
java 978 graylog 76u IPv6 55569 0t0 TCP 127.0.0.1:9000->127.0.0.1:45986 (ESTABLISHED)
java 978 graylog 77u IPv6 55644 0t0 TCP 127.0.0.1:46056->127.0.0.1:9000 (ESTABLISHED)
java 978 graylog 78u IPv6 56489 0t0 TCP 127.0.0.1:9000->127.0.0.1:46056 (ESTABLISHED)
firefox 2046 smc 194u IPv4 56360 0t0 TCP 127.0.0.1:45978->127.0.0.1:9000 (ESTABLISHED)
firefox 2046 smc 206u IPv4 56363 0t0 TCP 127.0.0.1:45984->127.0.0.1:9000 (ESTABLISHED)
firefox 2046 smc 209u IPv4 56364 0t0 TCP 127.0.0.1:45986->127.0.0.1:9000 (ESTABLISHED)

Also, I am not sure if my http settings are right. If you don’t mind, please let me know if these look ok to you:

###############
# HTTP settings
###############

#### HTTP bind address
#
# The network interface used by the Graylog HTTP interface.
#
# This network interface must be accessible by all Graylog nodes in the cluster and by all clients
# using the Graylog web interface.
#
# If the port is omitted, Graylog will use port 9000 by default.
#
# Default: 127.0.0.1:9000
#http_bind_address = 127.0.0.1:9000
#http_bind_address = [2001:db8::1]:9000

#### HTTP publish URI
#
# The HTTP URI of this Graylog node which is used to communicate with the other Graylog nodes in the cluster and by all
# clients using the Graylog web interface.
#
# The URI will be published in the cluster discovery APIs, so that other Graylog nodes will be able to find and connect to this Graylog node.
#
# This configuration setting has to be used if this Graylog node is available on another network interface than $http_bind_address,
# for example if the machine has multiple network interfaces or is behind a NAT gateway.
#
# If $http_bind_address contains a wildcard IPv4 address (0.0.0.0), the first non-loopback IPv4 address of this machine will be used.
# This configuration setting must not contain a wildcard address!
#
# Default: http://$http_bind_address/
#http_publish_uri = http://192.168.1.1:9000/

#### External Graylog URI
#
# The public URI of Graylog which will be used by the Graylog web interface to communicate with the Graylog REST API.
#
# The external Graylog URI usually has to be specified, if Graylog is running behind a reverse proxy or load-balancer
# and it will be used to generate URLs addressing entities in the Graylog REST API (see $http_bind_address).
#
# When using Graylog Collector, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
#
# This setting can be overriden on a per-request basis with the “X-Graylog-Server-URL” HTTP request header.
#
# Default: $http_publish_uri
#http_external_uri =

thank you…

You need to change it to 192.168.0.107

Yes! I was about to post that exact change I made! and now my side car status is active!
I will continue to read documentation to find out how to get messages…because right now, this is what it is displaying:
While retrieving data for this widget, the following error(s) occurred:

  • Connection refused (Connection refused).
    Thank you for you time and help!

It’s not clear to me where this message is displayed.
Can you show the screenshot?

Hi,
I have attached two screen shots: one showing that sidecar is now active and second one which shows the error.
To see this error, Systems->Sidecars->Show Messages.
Another way:Systems->Inputs->Show Received Messages.
I think, Graylog is capturing messages, because I followed documentation and entered commands to make sure Elasticsearch is running, right after that my entire PC has become slow. I guess, because Graylog is capturing a lot of data. But problem is nothing is displayed in dashboard.
I tried config files from " Ingest Windows eventlog" documentation, but that did not help. Thank you.

Check what’s in /var/log/graylog-server/server.log

Hi,
the log file is huge.
Here are last few lines:
retrying (attempt #314).
2020-11-17T14:16:24.961-05:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #312).
2020-11-17T14:16:37.829-05:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #315).
2020-11-17T14:16:38.079-05:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #314).
2020-11-17T14:16:39.269-05:00 ERROR [Cluster] Couldn’t read cluster health for indices [graylog_, gl-events_, gl-system-events_] (Could not connect to http://127.0.0.1:9200)
2020-11-17T14:16:39.269-05:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2020-11-17T14:16:44.254-05:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #315).
2020-11-17T14:17:01.972-05:00 WARN [IndexFieldTypePollerPeriodical] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2020-11-17T14:17:01.972-05:00 WARN [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2020-11-17T14:17:02.204-05:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #313).
2020-11-17T14:17:08.011-05:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #316).
2020-11-17T14:17:08.314-05:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #315).
2020-11-17T14:17:09.702-05:00 ERROR [Cluster] Couldn’t read cluster health for indices [graylog_
, gl-events_, gl-system-events_] (Could not connect to http://127.0.0.1:9200)
2020-11-17T14:17:09.702-05:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2020-11-17T14:17:14.587-05:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #316).

Thank you.

Graylog can’t connect to Elasticsearch.
Check its status and network settings

Thank you. let me check that.
Thank you.

Hi,
I tried making changes on these two line:
Default: http://127.0.0.1:9200
#elasticsearch_hosts = http://192.168.0.107:9200
But that did not work. I am not sure if I understand this setting correctly. If elasticsearch and Graylog are installed on the same machine, what I should be entering here? Do I need to uncomment elasticsearch_hosts as well and enter 127.0.0.1?
Thank you,

Check lsof -Pni :9200

Hi,
I tried following two configs but did not get any output from the command you provided:
Default: http://127.0.0.1:9200
#elasticsearch_hosts = http://192.168.0.107:9200

and
Default: http://192.168.0.107
#elasticsearch_hosts = http://192.168.0.107:9200.

Each time, I make any change in the server.conf, I am restarting my server and then restarting MongoDB, Elasticsearch and Graylog in that order using the commands provided at:
Ubuntu Installation
For some reason, when I start Elasticsearch service, response of the command is not similar to the ones for MongoDB and Graylog. I have attached the screenshot.
Thank you.

From this output still not clear which ip:port is used by ES
You can check elasticsearch.yml and ES logs to find out what’s the issue

Hi,
network.host and http.port were commented out in elasticsearch.yml. I made following changes:
# ---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 127.0.0.1

Set a custom port for HTTP:

http.port: 9200

For network host I also tried 192.168.0.107. keeping http.port:9200
at var/log/elasticsearch I did not see any log file generated for ES. The most recent one is by the name gc.log.0.current which does not contain any useful information, I think.
I will try different combinations of IP settings, but can you please let me know if:
network.host: field in elasticsearch.yml should match Default: field in server.conf?
and it should be 127.0.0.1 or IP address of the machine (192.168.0.107)?
Also, I have a red cross on the Elasticsearch folder (inside etc and log folder), is that normal?
Thank you.

Yes, they must be the same.
ES is listening on some ip:port, GL is connecting to the same ip:port
If ES and GL located on the same host - you can use 127.0.0.1
You need to find out why ES seems not running.
What does systemctl status elasticsearch show?