Graylog on Azure with Nginx, Gelf TCP. End of File

Graylog Central (peer support)
, ,
1

Hello lovely community!

Having a bit of trouble with a Graylog installation on Azure.
Its running ok - I can log in, specify inputs etc.
I use nginx to proxy the ports on the front interface (due to the way that Azure exposes its Its).
So i have nginx forwarding port 80 to 9000, and then have an input which routes 12200 to 12201.

NXlog is sending messages, but for every message I get the following:

INFO connecting to <myserver>:12200
ERROR om_tcp received data from remote end (got 100 bytes)
ERROR last message repeated 2 times
ERROR om_tcp received data from remote end (got 43 bytes)
INFO reconnecting in 1 seconds
ERROR om_tcp detected a connection error; End of file found

So its getting through the nginx proxy, but the connection is being terminated by the server.

I used curl to try and connect to the 12200 port to see if it was forwarding, and the server.log of graylog had error lines all over it - so its clearly getting to the graylog server.

Any ideas why it would create an end of file??

Heres some config.

<Output 5a66385c7c6291c4ddec3484>
	Module om_tcp
	Host <mysever>
	Port 12200
	OutputType  GELF_TCP
	Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
	Exec $gl2_source_collector = '331bd8e3-435d-42d5-87bc-fb83fbd36fa9';
	Exec $collector_node_id = 'gra-001';
	Exec $Hostname = hostname_fqdn();
</Output>

Any help at all is greatly appreciated!

Regards,
Andrew

2

did you tried to use netcat against your port ?

echo -e '{"version": "1.1","host":"example.org","short_message":"Short message","full_message":"Backtrace here\n\nmore stuff","level":1,"_user_id":9001,"_some_info":"foo","_some_env_var":"bar"}\0' | nc -w 1 my.graylog.server 12200

just to be sure that everything elese is working.

3

This sounds like a botched nginx configuration.

Please post the complete nginx configuration of your setup.

4

If I run it through the proxy, im getting a 400 error.
HTTP/1.1 400 Bad Request
Server: nginx/1.10.3 (Ubuntu)
Date: Tue, 23 Jan 2018 11:00:00 GMT
Content-Type: text/html
Content-Length: 182
Connection: close

400 Bad Request

400 Bad Request

nginx/1.10.3 (Ubuntu)

If I run it on the box and point at at the input, then the errors in the logs complain about the Gelf message length, so at least the input is configured. So all the problems point to nginx. Heres the config - 2 ‘sites’ available:

Main api proxy

server
{
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;
    server_name my.graylog.server

    location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Graylog-Server-URL http://$server_name/api;
      proxy_pass       http://127.0.0.1:9000;
    }
}

Input Proxy
server
{
listen 12200 default_server;
listen [::]:12200 default_server ipv6only=on;
server_name my.graylog.server;

    access_log /var/log/nginx/gelf122.log; 

    error_log /var/log/nginx/gelf122_error.log info;

    location / {
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-Host $host;
     proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_pass       http://127.0.0.1:12201;
    }
}
5

Changed the config to remove headers:

server
{
    listen 12200 default_server;
    listen [::]:12200 default_server ipv6only=on;
    server_name my.graylog.server;

    access_log /var/log/nginx/gelf122.log; 

    error_log /var/log/nginx/gelf122_error.log info;
	ignore_invalid_headers off;

    location / {
      proxy_pass       http://127.0.0.1:12201;
    }

client_max_body_size 1M;
    client_body_buffer_size 1M;

}

Now getting some crazy entries in the nginx log.

client sent invalid method while reading client request line,

6

GELF TCP is not using HTTP as transport protocol.

Either you create a GELF HTTP input or you need to set up a proper TCP (not HTTP) proxy.

Also make sure to read http://docs.graylog.org/en/2.4/pages/gelf.html#example-payload.

1 Like
7

Jochen,

Thank you. I don’t know why I didn’t realise that. For completeness, here is an updated snippet to help anyone dealing with this…

stream
{
  upstream backend{
	  server 127.0.0.1:12201;
  }

server
{
    listen 12200;
    proxy_pass  backend;
}
}

This creates a nginx stream (non-http) to the backend.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.