Manually apply configuration and start sidecar

[nick]

We use sidecar to deploy configurations for beats.

We have some larger client deployments coming up and want to configure sidecar within our Windows build so we don’t have to configure each individual sidecar via the Graylog web interface.

We can install Graylog and sidecar and the beats clients as part of the build and we can deploy the configuration files for both Graylog and beats to the client. We then apply our graylog configuration yml and start the Graylog and sidecar service but I can’t find a documented way to then apply the beats configurations to the sidecar services.

We’ve deployed the configuration ymls to the generated folder in the Graylog beats directory but I’m not sure how to start the filebeat service with the config file in a way the sidecar will pick up.

[gsmith]

Hello,

@nick

but I can’t find a documented way to then apply the beats configurations to the sidecar services.

In this documentation below should show everything on Graylog Sidecar. From installing to configuring the log shippers. Just an FYI GL Sidecar is a wrapper for the log shippers (i.e. Nxlog, Winlogbeat, FileBeat, etc…)

https://docs.graylog.org/docs/sidecar

[nick]

Hi @gsmith, the documentation is pretty lightweight and doesn’t cover my question. There is a switch to start sidecar with a configuration file but nothing documented to start a beats instance via sidecar with a configuration switch.

The beats documentation says you can’t run beats as a service with a configuration switch however that is what sidecar seems to do. So I just want to know how to manually do this from command line and not have to do it with the web interface, as that isn’t scalable.

I’m assuming its a sidecar command rather than a beats command due to the above.

[gsmith]

Hello,

@nick

So I just want to know how to manually do this from command line and not have to do it with the web interface, as that isn’t scalable.

Graylog Sidecar is a wrapper for (i.e. Nxlog, Winlogbeat, FileBeat, etc…). In layman’s terms it controls these services from a remote destination. With that been said, Graylog Sidecar is the middle man between the Web UI and the log shipper. I haven’t seen Graylog sidecar command/s to execute these services. If you want to use line commands this would defeat the purpose of convenience through the Web UI. As a suggestion I would just use FileBeat Service instead if that’s the route you want to go.

What I do know is once you install GL Sidecar and enable the preferred log shipper on the Web UI you can execute the systemd command to show the status of that log shipper.

Sum it up I’ll show my lab configuration /setup.

Example of Graylog sidecar on Linux server using FileBeat.

Overview

Collector Configuration

Collector Process

As shown below once you click the tic box then navigate to the upper right under “Process” you would be able to “Stop, Start, Restart” the Beat service.

Now, Ill show the Filebeat service from this node.

root # systemctl status filebeat

So I guess the answer to this post is once you install GL to can use SystemD to “Stop. Start, Restart” the service.

This is not preferred since the Graylog Sidecar was made for convenience, It does resemble Ansible in a way.

Conclusion

Graylog Sidecar is a lightweight configuration management system for different log collectors, also called Backends. The Graylog node(s) act as a centralized hub containing the configurations of log collectors.

Graylog sidecar does control log shippers but its used for the connection between Graylog’s Web UI and the remote node. Graylog sends the commands to GL sidecar and the sidecar executes them, I’m not aware of line commands to execute for Graylog sidecar.

Let’s say you want to use FileBeat /w Graylog sidecar and adjust FileBeat configuration file on this node without using the Web UI ( Not preferred).

Graylog sidecar with FileBeat the log shipper configuration file location shown below. This depends on the what OS is used.

root@keycloak:/var/lib/graylog-sidecar/generated/keycloak-02.conf

FileBeat ( single installation) without Graylog Sidecar Installation, the configuration file is located below.

root@keycloak:/etc/filebeat/filebeat.conf

Hope that helps

[nick]

Thanks for the long reply but I think you’re misunderstanding the problem slightly.

It’s the beats client service on Windows I’m trying to control, not on the server and not on Linux. I understand the architecture and how the shippers are working, we’ve been running Graylog for a couple of years now including sidecar on both servers and clients.

I don’t want to use the Graylog web interface to initially deploy the log shipper configuration files, it’s incredibly inconvenient really, as it’s not scalable to add, for example, 100 clients (which we’re starting
to do). If I can get the client itself to start the beats service with the appropriate configuration file attached the client will automatically begin shipping data to Graylog and should appear in the sidecar list with the assigned configuration files, I can then manage and monitor the beats clients via the web interface on an ongoing basis (which is more ad-hoc, so more scalable).

Going right to the bottom of your post you’ve pointed out the location of the log shipper configuration files, I’m aware of the locations and I have pre-seeded those locations with the correct log files. Now I just need to know what the command is to start beats with those configuration files, the Graylog sidecar service must use this same command when controlled via the web interface to start the beats service with the appropriate configuration file.

I just need to know what this command/powershell is so I can automate this within our build process.

The beats documentation says when beats is run as a service the -c switch that allows the user to assign a configuration file on start isn’t valid (only valid when running in the foreground). However sidecar does control beats when running as a service so it must be doing something differently?

[gsmith]

Hello,

@nick

It’s the beats client service on Windows I’m trying to control,

Not only does my statement above apply for Linux but also windows.

@nick

I don’t want to use the Graylog web interface to initially deploy the log shipper configuration files, it’s incredibly inconvenient really, as it’s not scalable to add, for example, 100 clients (which we’re starting
to do).

Actually I have 141 Virtual machines, ranging from Linux and Windows. I control groups of server pretty quick. Some servers have Winlogbeat and some have FileBeat/Nxlog configuration and this is execute from the Web UI pretty quick.

Can I ask why makes this inconvenient?

@nick

If I can get the client itself to start the beats service with the appropriate configuration file attached the client will automatically begin shipping data to Graylog and should appear in the sidecar list with the assigned configuration files, I can then manage and monitor the beats clients via the web interface on an ongoing basis (which is more ad-hoc, so more scalable).

I actually execute this with Graylog Sidecar via Web UI, so I’m unsure why this is a issue. As I showed a demonstration above with Linux my environment also runs windows

@nick

Now I just need to know what the command is to start beats with those configuration files, the Graylog sidecar

Sorry but I’m not aware of any command/s to control Beat service using Graylog Sidecar remotely, I believe these command/s are internal.

If this doesn’t work for you I would highly suggest just downloading Winlogbeat or another beat service ( Not use Graylog Sidecar) and controlling these remote clients with Ansible which is almost the same as having the GL sidecar. As you know its not recommended to adjust the configuration file that Graylog sidecar is using to configure beat services.

@nick

I just need to know what this command/powershell is so I can automate this within our build process.

No commands that I know of accept what’s in the documentation.

@nick

However sidecar does control beats when running as a service so it must be doing something differently?

Like its stated Graylog Sidecar is a wrapper to these beat services. When installed it has it own environment to control/install these services.

Example: Our deployment we use Ansible, Not only does it install Graylog Sidecar remotely but we have special configuration made during this process that is needed (TCP/TLS).

Sorry @nick I cant be more help. I don’t know of these commands you want, but maybe someone else here has done this before. If I come across this, Ill post it here.

[patrickmann]

Would this be helpful to you?

"Default" configuration for Sidecar Log Collector

[nick]

Yes! This is exactly the problem, as long as when the configuration was auto assigned the services were also auto started this could work.

Unfortunately, looking at the comments it’s unlikely to happen. It’s taken over a year for someone in the dev team to get around to even reviewing it and when they have it looks like its too much work.

Which means I’m back to asking whether anyone knows how the sidecar service in Windows starts the beats service with an assigned configuration file?

[patrickmann]

@nick Just FYI, we haven’t back-burnered this. We are still planning to do it, but just couldn’t fit it into our current sprint.
We understand the importance of sidecar for folks with large deployments; and appreciate your feedback in helping us make it more useful.

[nick]

@patrickmann Thanks Patrick - is there anyone in the dev team who knows how sidecar talks to beats? It’d keep us going if we did have a command we could run just to start beats with a specific config.

[patrickmann]

Sidecar itself does not expose a public API. I think your best bet is to capture the GL API calls that are made by the Graylog UI when assigning a config; and then write automation to iterate over all your sidecar instances and issue those API calls from your script.

[nick]

I’m not sure I need to tell sidecar to do anything. If I can just start beats with the correct configuration file sidecar should pick that up automatically. I don’t want to command sidecar as such but beats. Which is why I’m more interested in how sidecar talks to beats and not how I could talk to sidecar. Ideally this would be documented in the beats documentation but it’s not (unless I’ve missed it).

[patrickmann]

Communication of config files is one way only: from GL server to sidecar to collector. Sidecar is a pretty dumb agent that is simply polling the GL server for new configurations.

• It does not communicate with collectors, other than writing the config files and restarting the collector service in case of a change.
• It does not pick up config changes that are made manually. Configs are managed purely on the server side.

If you want to automate config assignment via GL server and sidecar, then you can use the same APIs that the UI uses, as suggested previously.
If you decide to manage configs yourself, then you are bypassing GL and will need to automate roll-out of config files and starting/restarting collectors yourself.

[nick]

So I’m not sure that’s entirely correct (or I’m missing the key bit of documentation). I realise the config files are pulled down by the sidecar and then the collector is restarted and that’s as far as the sidecar goes in terms of control.

However, the config files aren’t stored in the default beats config file directories. So for beats to pick up the config that sidecar deploys beats must be told where the config files are. Then once that’s done the service restarts when new config files arrive mean it becomes a simple “dumb” process.

If I manually start the beats service with the config files manually placed in the correct sidecar directory then beats either doesn’t start (due to no config) or starts with the default beats (not Graylog/sidecar) config.

[patrickmann]

You can provide filebeat with a custom path and filename for the configuration file. I haven’t double-checked this in the code, but I am pretty certain that is all that sidecar does in order to get filebeat to pick up the custom config file.

Filebeat command reference | Filebeat Reference [8.2] | Elastic

[nick]

So that’s almost answering my original question!

My second reply (Manually apply configuration and start sidecar - #3 by nick) covered this - how does sidecar use that switch? The beats documentation says that switch can only be used with foreground running and not as a service. However beats is running as a service when controlled by sidecar!

(it feels like I’m getting closed to an answer!)

[patrickmann]

Here are instructions on configuring filebeat via systemd:

Filebeat and systemd | Filebeat Reference [8.2] | Elastic

[nick]

These are beats on Windows

[patrickmann]

Sorry, missed that - configure your custom settings via the Windows service properties.