New Graylog Server Install - Sidecar Token missing - Loading tokens of user failed

Hope everyone is doing well.

New Graylog Server Install - Sidecar Token missing

I installed and licensed a new Graylog server following the directions. Everything seems to be working except when you go into Sidecars. Nothing is found and if you click on Create or reuse a token for the graylog-sidecar user I get the following error. Not sure if it’s something that can be fixed or if I should start from scratch. Any help would be greatly appreciated.

Could not load tokens of user graylog-sidecar
Loading tokens of user failed with status: Error: cannot GET http://IPAddress:9000/api/users/graylog-sidecar/tokens (404)

[root@GrayLog ~]# tail -f /var/log/graylog-server/server.log
2020-10-09T09:42:38.095-04:00 WARN [IndexFieldTypePollerPeriodical] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2020-10-09T09:42:38.095-04:00 WARN [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2020-10-09T09:43:06.852-04:00 ERROR [Cluster] Couldn’t read cluster health for indices [graylog_*] (Could not connect to http://IPAddress:9200)
2020-10-09T09:43:06.852-04:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.

If I go into Search the Message Count and All Messages state

While retrieving data for this widget, the following error(s) occurred:

  • Connection refused (Connection refused).

Probably your Elastic Search DB is not working, check ES log. Did you follow official installation docs?
Which version did you install?

I used the following documentation: https://docs.graylog.org/en/3.3/pages/installation/os/centos.html#centosguide

[root@GrayLog ~]# tail -f /var/log/elasticsearch/graylog.log
[2020-10-08T15:53:38,623][INFO ][o.e.p.PluginsService ] [53guU1r] no plugins loaded
[2020-10-08T15:53:44,520][INFO ][o.e.d.DiscoveryModule ] [53guU1r] using discovery type [zen] and host providers [settings]
[2020-10-08T15:53:44,992][INFO ][o.e.n.Node ] [53guU1r] initialized
[2020-10-08T15:53:44,993][INFO ][o.e.n.Node ] [53guU1r] starting …
[2020-10-08T15:53:45,639][INFO ][o.e.t.TransportService ] [53guU1r] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2020-10-08T15:53:48,827][INFO ][o.e.c.s.MasterService ] [53guU1r] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {53guU1r}{53guU1rlRtm7uEH_IMhtqQ}{AUti_83gSm6r46Yoe_0D_A}{127.0.0.1}{127.0.0.1:9300}
[2020-10-08T15:53:48,830][INFO ][o.e.c.s.ClusterApplierService] [53guU1r] new_master {53guU1r}{53guU1rlRtm7uEH_IMhtqQ}{AUti_83gSm6r46Yoe_0D_A}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {53guU1r}{53guU1rlRtm7uEH_IMhtqQ}{AUti_83gSm6r46Yoe_0D_A}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2020-10-08T15:53:48,994][INFO ][o.e.g.GatewayService ] [53guU1r] recovered [0] indices into cluster_state
[2020-10-08T15:53:49,039][INFO ][o.e.h.n.Netty4HttpServerTransport] [53guU1r] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-10-08T15:53:49,039][INFO ][o.e.n.Node ] [53guU1r] started

Made an edit to the elesticsearch.yml and now I don’t get the connection refused but I still don’t have anything in the Sidecars Overview.

What is the sidecar configuration like on the client? is it linux or windows? nxlog or beat? If it isn’t pointing properly to the server or if it is blocked by local firewall/ rules then you wouldn’t see it show up in overview. Post your client configuration (nicely formatted with tools) THere are also log files on the client that give good detail about what the sidecar is trying and/or failing to do. Post those if you don’t find anything in there that solves your issue.