Hi,
As I discovered that Amazon’s ELB’s aren’t fit for UDP I have turned to Nginx as the Load Balancer.
Setup is:
Nginx Load Balancer
3 x Graylog nodes
3 x ES nodes
3 x Mongo nodes
I have been able to setup the Nginx LB with SSL and get to the Graylog web interface. I am now faced with the following issue:
Input - Port 12202 GELF - if I try to send messages to the LB with curl they never reach Graylog i.e.
curl -XPOST https://host.example.org:12202/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'
nginx config:
upstream graylog_master_interface {
server 10.1.41.58:12900;
}
upstream graylog_web_interface {
server x.x.x.1:12900;
server x.x.x.1:12202;
server x.x.x.2:12900;
server x.x.x.2:12202;
server x.x.x.3:12900;
server x.x.x.3:12202;
}
-- Settings for a TLS enabled server.
------USUAL SETTINGS HERE------
location /api/plugins/
{
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL https://host.example.org/api;
proxy_pass http://graylog_master_interface;
}
#
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL https://host.example.org/api;
proxy_pass http://graylog_web_interface;
}
server.conf
rest_listen_uri = http://x.x.x.1:12900/api/
trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128, x.x.x.0/16
web_listen_uri = http://x.x.x.1:12900/
*replicated on second and third node with x.x.x.2 and .3 respectively
I suspect my nginx config is in bad shape but have tried a number of combinations to no avail. The LB seems to be listening on 12202 but nothing is passed through to Graylog.
Query - I am getting a considerable amount of messages sent from the load balancer. Is this normal behaviour as I am requesting the graylog web interface through the LB?
The next step will be to add in UDP.
Thanks in advance for any help/tips. I know you guys get inundated with requests.
Thanks, Matt
jochen
(Jochen)
July 27, 2017, 3:43pm
2
Your nginx configuration seems incomplete and you’re mixing up upstreams in graylog_web_interface
(port 12900/tcp and 12202/tcp) which you shouldn’t do.
Please post your complete nginx configuration and split up the GELF HTTP input and the Graylog REST API/web interface in the nginx upstreams.
Thanks Jochen. Nginx config as below. I have split out the upstreams but struggling with listening on 12202 and proxy pass.
nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
index index.html index.htm;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name isat-elb.dev.odyssys.net;
root /usr/share/nginx/html;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
}
# redirect server error pages to the static page /40x.html
#
error_page 404 /404.html;
location = /40x.html {
}
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
upstream graylog_master_interface {
server x.x.x.1:12900;
}
upstream graylog_web_interface {
server x.x.x.1:12900;
server x.x.x.2:12900;
server x.x.x.3:12900;
}
upstream graylog_gelf {
server x.x.x.1:12202;
server x.x.x.2:12202;
server x.x.x.3:12202;
}
# Settings for a TLS enabled server.
#
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name graylog.net;
root /usr/share/nginx/html;
#
ssl_certificate "/path/fullchain.pem";
ssl_certificate_key "/path/privkey.pem";
# # It is *strongly* recommended to generate unique DH parameters
# # Generate them with: openssl dhparam -out /etc/pki/nginx/dhparams.pem 2048
# #ssl_dhparam "/etc/pki/nginx/dhparams.pem";
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP;
ssl_prefer_server_ciphers on;
#
# # Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location /api/plugins/
{
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL https://graylog.net/api;
proxy_pass http://graylog_master_interface;
}
#
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL https://graylog.net/api;
proxy_pass http://graylog_web_interface;
}
#
error_page 404 /404.html;
location = /40x.html {
}
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
server {
listen 12202;
server_name graylog.net;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL https:/graylog.net/api;
proxy_pass http://graylog_gelf;
}
}
}
OK I am starting to get somewhere. I updated the /api within the 12202 listener to /gelf. i.e.
proxy_set_header X-Graylog-Server-URL https://graylog.net/gelf;
I can now echo messages to graylog:
echo -e ‘{“version”: “1.1”,“host”:“example.org ”,“short_message”:“Short message”,“full_message”:“Backtrace here\n\nmore stuff”,“level”:1,"_user_id":9001,"_some_info":“foo”,"_some_env_var":“bar”}\0’ | nc -w 1 PUBLICIP 12202
If I try to echo the message to the load balancer’s hostname instead of the IP I get a '400 Bad Request. The host obviously resolves. Is there an issue with echo over https?
Any thoughts?
jochen
(Jochen)
July 28, 2017, 1:31pm
5
GELF HTTP is not the same as GELF TCP (which you’re sending in your example with netcat
).
GELF HTTP expects a valid HTTP POST request to the /gelf
resource with the GELF payload as HTTP request body.
OK. Do you have an example command which I can send GELF TCP with?
*GELF TCP is the input I have configured
jochen
(Jochen)
July 28, 2017, 1:55pm
7
Are you sure about that? If so, the configuration of your nginx is wrong. GELF TCP does not use HTTP as transport protocol.
See http://docs.graylog.org/en/2.3/pages/gelf.html#example-payload for further details.
OK so I’ve commented out http_host in my nginx.conf:
server {
listen 12202;
server_name graylog.net;
location / {
#proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL https://graylog.net/gelf;
proxy_pass http://graylog_gelf;
}
}
If I send a message in with curl:
curl -X POST -H 'Content-Type: application/json' -d '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }' 'http://graylog.net:12202/gelf'
I see the following in received messages:
017-07-28 15:03:43.058 x.x.x.10
X-Graylog-Server-URL: https://graylog.net/gelf
2017-07-28 15:03:43.058 x.x.x.10
X-Forwarded-For: PUBLICIP
2017-07-28 15:03:43.058 x.x.x.10
Connection: close
2017-07-28 15:03:43.058 x.x.x.10
User-Agent: curl/7.54.0
2017-07-28 15:03:43.058 x.x.x.10
Content-Type: application/json
2017-07-28 15:03:43.058 x.x.x.10
X-Forwarded-Server: graylog.net
2017-07-28 15:03:43.058 x.x.x.10
X-Forwarded-Host: graylog.net
2017-07-28 15:03:43.058 x.x.x.10
Host: graylog_gelf
2017-07-28 15:03:43.058 x.x.x.10
Content-Length: 112
2017-07-28 15:03:43.058 x.x.x.10
Accept: */*
2017-07-28 15:03:43.058 x.x.x.10
X-Real-IP: PUBLICIP
Message sent isn’t showing and source is private IP of LB.
I presume my nginx.conf still isn’t right?
Thanks, Matt
jochen
(Jochen)
July 28, 2017, 3:17pm
9
First of all, please use proper text formatting in your posts. This makes reading them much easier on the eye.
Example:
```
pre formatted text
```
Also see Markdown Reference
These are the lines of a HTTP request. Specifically of the HTTP request sent by curl
and nginx.
Take a step back and think about what you want to achieve. Then describe this to us.
What type of inputs did you create in Graylog and what’s their configuration?
What do you expect nginx to do in your setup?
Currently it looks like you’re simply stabbing in the dark.
Fair enough.
Here are my configured Inputs:
Nginx - I would like the following:
Load balance between the 3 Graylog nodes
Proxy TCP traffic on Port 12202 to graylog nodes
Proxy UDP traffic on Port 12201 to graylog nodes
I am fresh to Nginx (as you can tell) and trying to get my head around using it for Load Balancing.
Thanks for the help so far.
jochen
(Jochen)
July 31, 2017, 3:41pm
11
I neither see a GELF UDP, nor a GELF TCP, nor a GELF HTTP input.
Why are you trying to send GELF messages to Syslog or Plaintext inputs?
system
(system)
Closed
August 14, 2017, 3:42pm
12
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.