Help for Health checking LB (nginx) konfiguration

[Marvin1]

Hi, Im running a multi-node cluster with 3 graylog servers.
I want to ask how nginx can be aware of the health status of a node?
I marked a node as DEAD by hand and it still got send messages from nginx…

For a better understaning of my enviroment:

This is my nginx conf

ser www-data;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {
    ##
    # Basic Settings
    ##
    sendfile on;
    tcp_nopush on;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##
    #ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
    #ssl_prefer_server_ciphers on;

    ##
    # Logging Settings
    ##
    access_log /var/log/nginx/access.log;

    ##
    # Gzip Settings
    ##
    gzip on;

    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;
    # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # Virtual Host Configs
    ##
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;

    ##
    # Load balancing for Graylog web interface
    ##
    upstream graylog_web {
        server gray1.example.intern:9000;
        server gray2.example.intern:9000;
        server gray3.example.intern:9000;
    }

    ##
    # HTTP to HTTPS redirection
    ##

server {
        listen 443 ssl;
        server_name graylog.example.intern;
        ssl_certificate /etc/ssl/certs/graylog.crt;
        ssl_certificate_key /etc/ssl/certs/graylog.key;

        location / {
            proxy_pass http://graylog_web;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

##
# Stream block for UDP and TCP load balancing
##
stream {
    upstream graylog_gelf_udp {
        server gray1.example.intern:12201;
        server gray2.example.intern:12201;
        server gray3.example.intern:12201;
    }

    upstream graylog_raw_udp {
        server gray1.example.intern:12300;
        server gray2.example.intern:12300;
        server gray3.example.intern:12300;


    }

    upstream graylog_syslog_udp {
        server gray1.example.intern:5140;
        server gray2.example.intern:5140;
        server gray3.example.intern:5140;
    }

    upstream graylog_syslog_tcp {
        server gray1.example.intern:5141;
        server gray2.example.intern:5141;
        server gray3.example.intern:5141;
    }

    server {
        listen 12201 udp;
        proxy_pass graylog_gelf_udp;
        proxy_responses 0;
    }

    server {
        listen 12300 udp;
        proxy_pass graylog_raw_udp;
        proxy_responses 0;
    }

    server {
        listen 5140 udp;
        proxy_pass graylog_syslog_udp;
    }
proxy_responses 0;

    server {
        listen 5141;
        proxy_pass graylog_syslog_tcp;
        proxy_responses 1;
    }
}

I need a config for a status checking LB
Thanks for your help Guys

[Wine_Merchant]

Hey @Marvin1,

It might be that the health check function within nginx is a feature reserved for the plus license, which would mean paying.

I believe HA-proxy has a similar function which is free, example below.

option httpchk HEAD /api/system/lbstatus

[RichT]

We are handeling our failover with VRRP (keepalived).
We use a script that calls the API to get the state of inputs on a node.
( VRRP does not do loadbalanceing only failover so not applicable for you)
As mentioned already nginx seems to limit custom checks to the paid license.
There are Plugins for nginx like: GitHub - nginx-modules/nginx_upstream_check_module: Health checks upstreams for nginx
But you would have to compile your own NGINX version which is very much undesirable!
HA Proxy is probably is your best alternative for loadbalanceing.
You could use an “external-check” or ha proxy might even have a builtin check that can parse the json response of the API.