Sidecar config with loadbalancer

[afmin]

Hello,

nginx x.x.x.29
graylog01 x.x.x.30
graylog02 x.x.x.31
graylog03 x.x.x.32

I am not getting the messages in if I set the config over the loadbalancer. If I configure sidecar on windows to one graylog server (x.x.x.31) I get all the logs in.
-> sidecar.yml -> x.x.x.31:9000/api -> see collectors and logs
-> sidecar.yml -> x.x.x.29:9000/api -> see collector but no logs!

Loadbalancer forwards http traffic Port 9000 to 9000
and logs from port 5044 to the graylog servers

Graylog is listening on port 5044 (beats)

Thanks for your help!

[jan (Jan Doberstein)]

he @afmin

you need to notice that you have the server_uri that is for the communication of sidecar and Graylog - basically where sidecar can reach the Graylog server.

The second is the connection from the collector to the Graylog input. From what you have written filebeat/winlogbeat to a Graylog Beats input. How did you balance the beats input? It is very likely that balacing beats is not done right and I would place all Graylog servers in the output configuration and let the beat itself balance between the Graylog servers.

[afmin]

Hi Jan

Thanks for your reply!

nginx x.x.x.29
graylog01 x.x.x.30
graylog02 x.x.x.31
graylog03 x.x.x.32

Case1: works not
following sidecar.yml config is set
server_url: “http://x.x.x.29:9000/api/” -> which is the nginx loadbalancer
and in the sidecar output configuration is set the nginx loadbalancer
output.logstash:
hosts: [“x.x.x.29:5044”]
-> i am not getting logs in

Case2: works
following sidecar.yml config is set
server_url: “http://x.x.x.29:9000/api/” -> which is the nginx loadbalancer
and in the sidecar output configuration is set one Graylog Server
output.logstash:
hosts: [“x.x.x.31:5044”]
-> i am getting logs in

Possible solution?:
following sidecar.yml config is set
server_url: “http://x.x.x.29:9000/api/” -> which is the nginx loadbalancer
and in the sidecar output configuration I set all Graylog Server
output.logstash:
hosts: [“x.x.x.30:5044”]
hosts: [“x.x.x.31:5044”]
hosts: [“x.x.x.32:5044”]
-> is this case working if a note is not working? Or should I set the output.logtash to the master graylog note?

Thanks!

[macko003]

Just a suggestion.
Use your nginx to loadbanace the 5044 port also (you will need a stream).
If you set it, you need to use only the .29 IP. Eg. if you will increase the number of GL servers, you need to set it in your loadbalancer.

///Outside from http part
stream{
        upstream graylog_syslog {
        server 192.168.X.118:1514;
        server 192.168.X.117:1514;
        server 192.168.X.116:1514;
        }
        upstream graylog_beats {
        server 192.168.X.118:5044;
        server 192.168.X.117:5044;
        server 192.168.X.116:5044;
        }

        server {
        listen 1514;
        proxy_pass graylog_syslog;
        }
        server {
        listen 514;
        proxy_pass graylog_syslog;
        }
        server {
        listen 5044;
        proxy_pass graylog_beats;
        }
}

[jan (Jan Doberstein)]

he @afmin

for your case 1, you need to follow the advice of @macko003. Your case 2 is working because you communicate from the collector directly to Graylog. and the possible solution would be the best way to avoid problems if one Graylog server ist not preset/rebooting. Beats are able doing a healthcheck to the target and remove that temporary if not give.

But my personal advice - follow @macko003 recommendation.

[afmin]

hi @macko003 hi @jan
Thanks for your response! This upstream config is already set on my nginx load balancer.

#/etc/nginx/nginx.conf
stream {
include /etc/nginx/conf.d/log_loadbalancing.conf;
proxy_bind $remote_addr transparent;
}

#etc/nginx/conf.d/log_loadbalancing.conf
#windows_sidecar
upstream windows_sidecar {
server x.x.x.31:5044;
server x.x.x.32:5044;
}
server {
listen 5044;
proxy_pass windows_sidecar;
}

http loadbalancing
upstream sidecar {
server x.x.x.31:9000;
server x.x.x.32:9000;
}

server {
listen 9000;
server_name x.x.x.29;

access_log /var/log/nginx/access.log upstreamlog;

location / {
proxy_pass http://sidecar;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL http://$server_name/;
}
}

[macko003]

And I think It doesn’t work…
Please reread my post, and do what I wrote.

@macko003

///Outside from http part

You can use your terminal to verify it.
Eg.
nginx -t
ss -ntlp | grep 5044

[afmin]

@macko003

///Outside from http part

I set it outside the http part → stream {…}
But I also set a port 9000 forwarding in the http section.

on my nginx loadbalancer
ss -ntlp | grep 5044
LISTEN 0 128 *:5044 :

time=“2020-01-30T13:58:35+01:00” level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://x.x.x.29:9000/api/sidecars/10a8exxxxxxxxxxxxxxx: dial tcp x.x.x.29:9000: connectex: No connection could be made because the target machine actively refused it.

[macko003]

5044 is NOT http protocol
so you have to configure it OUTSIDE of the http part of the nginx config.

[afmin]

hi @macko003 i already have it configured OUTSIDE the “http” part -> see nginx.conf below
and those parts refere to the config files “http_loadbalancer.conf” and “log_loadbalancer.conf” as already post on top. In the "log_loadbalancer.conf = STREAM’s i listen on port 5044 and forwarding it to the GL notes

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

access_log  /var/log/nginx/access.log  main;

sendfile        on;
#tcp_nopush     on;

keepalive_timeout  65;

#gzip  on;

include /etc/nginx/conf.d/http_loadbalancing.conf;

}

stream {
include /etc/nginx/conf.d/log_loadbalancing.conf;
proxy_bind $remote_addr transparent;
}

/etc/nginx/conf.d/log_loadbalancing.conf->>>
upstream windows_sidecar {
server x.x.x.31:5044;
server x.x.x.32:5044;
}
server {
listen 5044;
proxy_pass windows_sidecar;
}

[afmin]

Hi @jan

Case3 : is not working as expected -> is the config like this correct?

output.logstash:
hosts: [“x.x.x.31:5044”]
hosts: [“x.x.x.32:5044”]

the beats coming in over the x.x.x.32 GL Server if I take it offline -> stop receiving messages… if i turn it back on -> getting the queued messages in.

[tmacgbay]

filebeat.inputs:
- type: log
  paths:
    - /var/log/*.log
output.logstash:
  hosts: ["localhost:5044", "localhost:5045"]
  loadbalance: true
  worker: 2

The load balancer also supports multiple workers per host. The default is worker: 1 . If you increase the number of workers, additional network connections will be used. The total number of workers participating in load balancing is number of hosts * workers .

https://www.elastic.co/guide/en/beats/filebeat/6.8/load-balancing.html

[afmin]

Hi @macko003
the GL external uri set to isnt the problem you think? -> http_external_uri = http://x.x.x.29/

[afmin]

worked kind of thank you, but now i have to solve the search status code 500 first

[macko003]

Usually I don’t change the http_external_uri, I keep it commented.
I use nginx to modify the X-Graylog-Server-URL header.

Use tcpdump, to check the ip-s, and content of the packages.

[jan (Jan Doberstein)]

you might want to read the documentation of filebeat how that should be written …