Graylog Nginx Load Balancer

Hi,

As I discovered that Amazon’s ELB’s aren’t fit for UDP I have turned to Nginx as the Load Balancer.

Setup is:

Nginx Load Balancer
3 x Graylog nodes
3 x ES nodes
3 x Mongo nodes

I have been able to setup the Nginx LB with SSL and get to the Graylog web interface. I am now faced with the following issue:

Input - Port 12202 GELF - if I try to send messages to the LB with curl they never reach Graylog i.e.

curl -XPOST https://host.example.org:12202/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'

nginx config:

        upstream graylog_master_interface {
         server 10.1.41.58:12900;
}

        upstream graylog_web_interface {
         server x.x.x.1:12900;
         server x.x.x.1:12202;
         server x.x.x.2:12900;
         server x.x.x.2:12202;
         server x.x.x.3:12900;
         server x.x.x.3:12202;
}

-- Settings for a TLS enabled server.
------USUAL SETTINGS HERE------


        location /api/plugins/
    {
        proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https://host.example.org/api;
        proxy_pass          http://graylog_master_interface;
    }
#
        location / {
         proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https://host.example.org/api;
        proxy_pass          http://graylog_web_interface;
        }

server.conf

rest_listen_uri = http://x.x.x.1:12900/api/
trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128, x.x.x.0/16
web_listen_uri = http://x.x.x.1:12900/

*replicated on second and third node with x.x.x.2 and .3 respectively

I suspect my nginx config is in bad shape but have tried a number of combinations to no avail. The LB seems to be listening on 12202 but nothing is passed through to Graylog.

Query - I am getting a considerable amount of messages sent from the load balancer. Is this normal behaviour as I am requesting the graylog web interface through the LB?

The next step will be to add in UDP.

Thanks in advance for any help/tips. I know you guys get inundated with requests.

Thanks, Matt

Your nginx configuration seems incomplete and you’re mixing up upstreams in graylog_web_interface (port 12900/tcp and 12202/tcp) which you shouldn’t do.

Please post your complete nginx configuration and split up the GELF HTTP input and the Graylog REST API/web interface in the nginx upstreams.

Thanks Jochen. Nginx config as below. I have split out the upstreams but struggling with listening on 12202 and proxy pass.

nginx.conf

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    index   index.html index.htm;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  isat-elb.dev.odyssys.net;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }

        # redirect server error pages to the static page /40x.html
        #
       error_page 404 /404.html;
            location = /40x.html {
        }

        # redirect server error pages to the static page /50x.html
        #
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
 #    deny  all;
        #}
    }

        upstream graylog_master_interface {
         server x.x.x.1:12900;
}

        upstream graylog_web_interface {
         server x.x.x.1:12900;
         server x.x.x.2:12900;
         server x.x.x.3:12900;
}
        upstream graylog_gelf {
        server x.x.x.1:12202;
        server x.x.x.2:12202;
        server x.x.x.3:12202;
}
# Settings for a TLS enabled server.
#
    server {
        listen       443 ssl;
        listen       [::]:443 ssl;
        server_name  graylog.net;
        root         /usr/share/nginx/html;
#
        ssl_certificate "/path/fullchain.pem";
        ssl_certificate_key "/path/privkey.pem";
#        # It is *strongly* recommended to generate unique DH parameters
#        # Generate them with: openssl dhparam -out /etc/pki/nginx/dhparams.pem 2048
#        #ssl_dhparam "/etc/pki/nginx/dhparams.pem";
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout  10m;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP;
        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location /api/plugins/
    {
        proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https://graylog.net/api;
        proxy_pass          http://graylog_master_interface;
    }
#
        location / {
         proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https://graylog.net/api;
        proxy_pass          http://graylog_web_interface;
        }
#
        error_page 404 /404.html;
            location = /40x.html {
        }
#
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
        server {
        listen 12202;
        server_name  graylog.net;
        location / {
        proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https:/graylog.net/api;
        proxy_pass          http://graylog_gelf;
        }
}
}

OK I am starting to get somewhere. I updated the /api within the 12202 listener to /gelf. i.e.

proxy_set_header X-Graylog-Server-URL https://graylog.net/gelf;

I can now echo messages to graylog:

echo -e ‘{“version”: “1.1”,“host”:“example.org”,“short_message”:“Short message”,“full_message”:“Backtrace here\n\nmore stuff”,“level”:1,"_user_id":9001,"_some_info":“foo”,"_some_env_var":“bar”}\0’ | nc -w 1 PUBLICIP 12202

If I try to echo the message to the load balancer’s hostname instead of the IP I get a '400 Bad Request. The host obviously resolves. Is there an issue with echo over https?

Any thoughts?

GELF HTTP is not the same as GELF TCP (which you’re sending in your example with netcat).

GELF HTTP expects a valid HTTP POST request to the /gelf resource with the GELF payload as HTTP request body.

OK. Do you have an example command which I can send GELF TCP with?

*GELF TCP is the input I have configured

Are you sure about that? If so, the configuration of your nginx is wrong. GELF TCP does not use HTTP as transport protocol.

See http://docs.graylog.org/en/2.3/pages/gelf.html#example-payload for further details.

OK so I’ve commented out http_host in my nginx.conf:

 server {
        listen 12202;
        server_name  graylog.net;
        location / {
        #proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https://graylog.net/gelf;
        proxy_pass          http://graylog_gelf;
        }
        }

If I send a message in with curl:

curl -X POST -H 'Content-Type: application/json' -d '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }' 'http://graylog.net:12202/gelf'

I see the following in received messages:

017-07-28 15:03:43.058	x.x.x.10
X-Graylog-Server-URL: https://graylog.net/gelf
2017-07-28 15:03:43.058	x.x.x.10
X-Forwarded-For: PUBLICIP
2017-07-28 15:03:43.058	x.x.x.10
Connection: close
2017-07-28 15:03:43.058	x.x.x.10
User-Agent: curl/7.54.0
2017-07-28 15:03:43.058	x.x.x.10
Content-Type: application/json
2017-07-28 15:03:43.058	x.x.x.10
X-Forwarded-Server: graylog.net
2017-07-28 15:03:43.058	x.x.x.10
X-Forwarded-Host: graylog.net
2017-07-28 15:03:43.058	x.x.x.10
Host: graylog_gelf
2017-07-28 15:03:43.058	x.x.x.10
Content-Length: 112
2017-07-28 15:03:43.058	x.x.x.10
Accept: */*
2017-07-28 15:03:43.058	x.x.x.10
X-Real-IP: PUBLICIP

Message sent isn’t showing and source is private IP of LB.

I presume my nginx.conf still isn’t right?

Thanks, Matt

First of all, please use proper text formatting in your posts. This makes reading them much easier on the eye.

Example:

```
pre formatted text
```

Also see Markdown Reference

These are the lines of a HTTP request. Specifically of the HTTP request sent by curl and nginx.

Take a step back and think about what you want to achieve. Then describe this to us.

  • What type of inputs did you create in Graylog and what’s their configuration?
  • What do you expect nginx to do in your setup?

Currently it looks like you’re simply stabbing in the dark.

Fair enough.

Here are my configured Inputs:

Nginx - I would like the following:

  • Load balance between the 3 Graylog nodes

  • Proxy TCP traffic on Port 12202 to graylog nodes

  • Proxy UDP traffic on Port 12201 to graylog nodes

I am fresh to Nginx (as you can tell) and trying to get my head around using it for Load Balancing.

Thanks for the help so far.

I neither see a GELF UDP, nor a GELF TCP, nor a GELF HTTP input.

Why are you trying to send GELF messages to Syslog or Plaintext inputs?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.