Graylog Nginx Load Balancer


(Matt) #1

Hi,

As I discovered that Amazon’s ELB’s aren’t fit for UDP I have turned to Nginx as the Load Balancer.

Setup is:

Nginx Load Balancer
3 x Graylog nodes
3 x ES nodes
3 x Mongo nodes

I have been able to setup the Nginx LB with SSL and get to the Graylog web interface. I am now faced with the following issue:

Input - Port 12202 GELF - if I try to send messages to the LB with curl they never reach Graylog i.e.

curl -XPOST https://host.example.org:12202/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'

nginx config:

        upstream graylog_master_interface {
         server 10.1.41.58:12900;
}

        upstream graylog_web_interface {
         server x.x.x.1:12900;
         server x.x.x.1:12202;
         server x.x.x.2:12900;
         server x.x.x.2:12202;
         server x.x.x.3:12900;
         server x.x.x.3:12202;
}

-- Settings for a TLS enabled server.
------USUAL SETTINGS HERE------


        location /api/plugins/
    {
        proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https://host.example.org/api;
        proxy_pass          http://graylog_master_interface;
    }
#
        location / {
         proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https://host.example.org/api;
        proxy_pass          http://graylog_web_interface;
        }

server.conf

rest_listen_uri = http://x.x.x.1:12900/api/
trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128, x.x.x.0/16
web_listen_uri = http://x.x.x.1:12900/

*replicated on second and third node with x.x.x.2 and .3 respectively

I suspect my nginx config is in bad shape but have tried a number of combinations to no avail. The LB seems to be listening on 12202 but nothing is passed through to Graylog.

Query - I am getting a considerable amount of messages sent from the load balancer. Is this normal behaviour as I am requesting the graylog web interface through the LB?

The next step will be to add in UDP.

Thanks in advance for any help/tips. I know you guys get inundated with requests.

Thanks, Matt


(Jochen) #2

Your nginx configuration seems incomplete and you’re mixing up upstreams in graylog_web_interface (port 12900/tcp and 12202/tcp) which you shouldn’t do.

Please post your complete nginx configuration and split up the GELF HTTP input and the Graylog REST API/web interface in the nginx upstreams.


(Matt) #3

Thanks Jochen. Nginx config as below. I have split out the upstreams but struggling with listening on 12202 and proxy pass.

nginx.conf

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    index   index.html index.htm;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  isat-elb.dev.odyssys.net;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }

        # redirect server error pages to the static page /40x.html
        #
       error_page 404 /404.html;
            location = /40x.html {
        }

        # redirect server error pages to the static page /50x.html
        #
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
 #    deny  all;
        #}
    }

        upstream graylog_master_interface {
         server x.x.x.1:12900;
}

        upstream graylog_web_interface {
         server x.x.x.1:12900;
         server x.x.x.2:12900;
         server x.x.x.3:12900;
}
        upstream graylog_gelf {
        server x.x.x.1:12202;
        server x.x.x.2:12202;
        server x.x.x.3:12202;
}
# Settings for a TLS enabled server.
#
    server {
        listen       443 ssl;
        listen       [::]:443 ssl;
        server_name  graylog.net;
        root         /usr/share/nginx/html;
#
        ssl_certificate "/path/fullchain.pem";
        ssl_certificate_key "/path/privkey.pem";
#        # It is *strongly* recommended to generate unique DH parameters
#        # Generate them with: openssl dhparam -out /etc/pki/nginx/dhparams.pem 2048
#        #ssl_dhparam "/etc/pki/nginx/dhparams.pem";
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout  10m;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP;
        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location /api/plugins/
    {
        proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https://graylog.net/api;
        proxy_pass          http://graylog_master_interface;
    }
#
        location / {
         proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https://graylog.net/api;
        proxy_pass          http://graylog_web_interface;
        }
#
        error_page 404 /404.html;
            location = /40x.html {
        }
#
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
        server {
        listen 12202;
        server_name  graylog.net;
        location / {
        proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https:/graylog.net/api;
        proxy_pass          http://graylog_gelf;
        }
}
}

(Matt) #4

OK I am starting to get somewhere. I updated the /api within the 12202 listener to /gelf. i.e.

proxy_set_header X-Graylog-Server-URL https://graylog.net/gelf;

I can now echo messages to graylog:

echo -e ‘{“version”: “1.1”,“host”:“example.org”,“short_message”:“Short message”,“full_message”:“Backtrace here\n\nmore stuff”,“level”:1,"_user_id":9001,"_some_info":“foo”,"_some_env_var":“bar”}\0’ | nc -w 1 PUBLICIP 12202

If I try to echo the message to the load balancer’s hostname instead of the IP I get a '400 Bad Request. The host obviously resolves. Is there an issue with echo over https?

Any thoughts?


(Jochen) #5

GELF HTTP is not the same as GELF TCP (which you’re sending in your example with netcat).

GELF HTTP expects a valid HTTP POST request to the /gelf resource with the GELF payload as HTTP request body.


(Matt) #6

OK. Do you have an example command which I can send GELF TCP with?

*GELF TCP is the input I have configured


(Jochen) #7

Are you sure about that? If so, the configuration of your nginx is wrong. GELF TCP does not use HTTP as transport protocol.

See http://docs.graylog.org/en/2.3/pages/gelf.html#example-payload for further details.


(Matt) #8

OK so I’ve commented out http_host in my nginx.conf:

 server {
        listen 12202;
        server_name  graylog.net;
        location / {
        #proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL https://graylog.net/gelf;
        proxy_pass          http://graylog_gelf;
        }
        }

If I send a message in with curl:

curl -X POST -H 'Content-Type: application/json' -d '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }' 'http://graylog.net:12202/gelf'

I see the following in received messages:

017-07-28 15:03:43.058	x.x.x.10
X-Graylog-Server-URL: https://graylog.net/gelf
2017-07-28 15:03:43.058	x.x.x.10
X-Forwarded-For: PUBLICIP
2017-07-28 15:03:43.058	x.x.x.10
Connection: close
2017-07-28 15:03:43.058	x.x.x.10
User-Agent: curl/7.54.0
2017-07-28 15:03:43.058	x.x.x.10
Content-Type: application/json
2017-07-28 15:03:43.058	x.x.x.10
X-Forwarded-Server: graylog.net
2017-07-28 15:03:43.058	x.x.x.10
X-Forwarded-Host: graylog.net
2017-07-28 15:03:43.058	x.x.x.10
Host: graylog_gelf
2017-07-28 15:03:43.058	x.x.x.10
Content-Length: 112
2017-07-28 15:03:43.058	x.x.x.10
Accept: */*
2017-07-28 15:03:43.058	x.x.x.10
X-Real-IP: PUBLICIP

Message sent isn’t showing and source is private IP of LB.

I presume my nginx.conf still isn’t right?

Thanks, Matt


(Jochen) #9

First of all, please use proper text formatting in your posts. This makes reading them much easier on the eye.

Example:

```
pre formatted text
```

Also see http://commonmark.org/help/

These are the lines of a HTTP request. Specifically of the HTTP request sent by curl and nginx.

Take a step back and think about what you want to achieve. Then describe this to us.

  • What type of inputs did you create in Graylog and what’s their configuration?
  • What do you expect nginx to do in your setup?

Currently it looks like you’re simply stabbing in the dark.


(Matt) #10

Fair enough.

Here are my configured Inputs:

Nginx - I would like the following:

  • Load balance between the 3 Graylog nodes

  • Proxy TCP traffic on Port 12202 to graylog nodes

  • Proxy UDP traffic on Port 12201 to graylog nodes

I am fresh to Nginx (as you can tell) and trying to get my head around using it for Load Balancing.

Thanks for the help so far.


(Jochen) #11

I neither see a GELF UDP, nor a GELF TCP, nor a GELF HTTP input.

Why are you trying to send GELF messages to Syslog or Plaintext inputs?


(system) #12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.