New Graylog Server Install - Sidecar Token missing - Loading tokens of user failed

Graylog Central (peer support)
1

Hope everyone is doing well.

New Graylog Server Install - Sidecar Token missing

I installed and licensed a new Graylog server following the directions. Everything seems to be working except when you go into Sidecars. Nothing is found and if you click on Create or reuse a token for the graylog-sidecar user I get the following error. Not sure if it’s something that can be fixed or if I should start from scratch. Any help would be greatly appreciated.

Could not load tokens of user graylog-sidecar
Loading tokens of user failed with status: Error: cannot GET http://IPAddress:9000/api/users/graylog-sidecar/tokens (404)

[root@GrayLog ~]# tail -f /var/log/graylog-server/server.log
2020-10-09T09:42:38.095-04:00 WARN [IndexFieldTypePollerPeriodical] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2020-10-09T09:42:38.095-04:00 WARN [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2020-10-09T09:43:06.852-04:00 ERROR [Cluster] Couldn’t read cluster health for indices [graylog_*] (Could not connect to http://IPAddress:9200)
2020-10-09T09:43:06.852-04:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.

If I go into Search the Message Count and All Messages state

While retrieving data for this widget, the following error(s) occurred:

  • Connection refused (Connection refused).
2

Probably your Elastic Search DB is not working, check ES log. Did you follow official installation docs?
Which version did you install?

3

I used the following documentation: https://docs.graylog.org/en/3.3/pages/installation/os/centos.html#centosguide

[root@GrayLog ~]# tail -f /var/log/elasticsearch/graylog.log
[2020-10-08T15:53:38,623][INFO ][o.e.p.PluginsService ] [53guU1r] no plugins loaded
[2020-10-08T15:53:44,520][INFO ][o.e.d.DiscoveryModule ] [53guU1r] using discovery type [zen] and host providers [settings]
[2020-10-08T15:53:44,992][INFO ][o.e.n.Node ] [53guU1r] initialized
[2020-10-08T15:53:44,993][INFO ][o.e.n.Node ] [53guU1r] starting …
[2020-10-08T15:53:45,639][INFO ][o.e.t.TransportService ] [53guU1r] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2020-10-08T15:53:48,827][INFO ][o.e.c.s.MasterService ] [53guU1r] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {53guU1r}{53guU1rlRtm7uEH_IMhtqQ}{AUti_83gSm6r46Yoe_0D_A}{127.0.0.1}{127.0.0.1:9300}
[2020-10-08T15:53:48,830][INFO ][o.e.c.s.ClusterApplierService] [53guU1r] new_master {53guU1r}{53guU1rlRtm7uEH_IMhtqQ}{AUti_83gSm6r46Yoe_0D_A}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {53guU1r}{53guU1rlRtm7uEH_IMhtqQ}{AUti_83gSm6r46Yoe_0D_A}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2020-10-08T15:53:48,994][INFO ][o.e.g.GatewayService ] [53guU1r] recovered [0] indices into cluster_state
[2020-10-08T15:53:49,039][INFO ][o.e.h.n.Netty4HttpServerTransport] [53guU1r] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-10-08T15:53:49,039][INFO ][o.e.n.Node ] [53guU1r] started

4

Made an edit to the elesticsearch.yml and now I don’t get the connection refused but I still don’t have anything in the Sidecars Overview.

5

What is the sidecar configuration like on the client? is it linux or windows? nxlog or beat? If it isn’t pointing properly to the server or if it is blocked by local firewall/ rules then you wouldn’t see it show up in overview. Post your client configuration (nicely formatted with tools) THere are also log files on the client that give good detail about what the sidecar is trying and/or failing to do. Post those if you don’t find anything in there that solves your issue.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

7

Hi all,

I’m just seeing this post, and have run into the same issue. I raised it a couple of days back with our developers in https://github.com/Graylog2/graylog2-server/issues/9555. The tl;dr is that the link from the sidecar page is a invalid link because of the changes made to how users are managed now. We’ll be pushing out a fix in 4.1, but know that for now, the workaround can be found here: https://github.com/Graylog2/graylog2-server/issues/9555#issuecomment-732940387.