Gl2_remote_ip always 127.0.0.1 when using nginx reverse proxy

Graylog Central (peer support)
1

Hi there!

I’m using Graylog 2.2.0 and have an HTTP input set up and taking data in.

The problem I have is that the gl2_remote_ip field of documents being stored is always 127.0.0.1 - no matter from which machine I am testing sending data from.

I believe this comes from the fact that I am using nginx as a reverse proxy in front of Graylog, so that I can have folks not worry about setting a port when sending data to graylog.

Here’s the relevant part of my nginx redirect:

  location /gelf {
    proxy_pass http://localhost:12201/gelf;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass_request_headers on;
    proxy_connect_timeout 150;
    proxy_send_timeout 100;
    proxy_read_timeout 100;
    proxy_buffers 4 32k;
  }

However, I was under the impression that, based on this part of the Graylog “Securing” documentation, I would just need to ensure nginx sets the X-Forwarded-For header (which I think I have set correctly in the above config). I have also set the trusted_proxies setting in my graylog.conf, like this:

trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128

I found this relevant topic in the old Google groups but it doesn’t look like it got resolved.

Anyone have any ideas? Any help would be appreciated!

Much obliged,
Fil Maj

2

The GELF HTTP input doesn’t support the X-Forwarded-For header.

You can use the regular host field of GELF messages to specify the source host, though.

3

Thanks for the reply @jochen!

As for setting the host field, in my particular case, not all of our applications sending data to Graylog have access to the client IP address (these are various kinds of JavaScript applications, operating in browser or hybrid mobile application environments). Ideally, I would not have to trouble the different development teams of all of our various applications to have to manually log the IP address of the running client. I think it should be technically possible to infer the IP address at a lower networking level from the graylog server.

Do you know if the GELF HTTP input supports inferring the IP address from any HTTP header at all? The old google groups post I linked to, it seems like the person from that post also tried setting the standard REMOTE_ADDR header manually in nginx to no avail.

If inferring the IP address from HTTP requests is missing from Graylog, can I help in adding it? I am happy to work with the Graylog team on GitHub to get this in place - it is important for me and my employer.

Finally, I think we should call out this behaviour (if it is unique to HTTP inputs) in the Graylog documentation. Once more, I am happy to assist in writing up the documentation. Just point me in the right direction on GitHub or elsewhere, I would be enthused to help out as Graylog is a radical product and I’d be excited to help the team in some way!

Thanks,
Fil

4

It doesn’t, it’s simply using the HTTP clients address.

I agree. You can also contribute to the Graylog documentation at:

5

For those that stumble on this problem as well, I found a workaround.

Credit to Oleg in this issue comment on the idea: have nginx rewrite JSON blobs on the go. https://github.com/Graylog2/graylog2-server/issues/2413#issuecomment-230050841

Additionally, there is an outstanding pull request to add support for pulling out IP info from the X-Forwarded-For header to GELF/HTTP inputs: https://github.com/Graylog2/graylog2-server/pull/3255. Too bad it’s sitting there idle…