The issue is, I can only view the Web Interface behind the LB/firewall on port 9000, however the UI doesn’t show outside of LB: http://<website>.com:9000
I have launched Grafana on the same exact port and server and its was successfully viewable on : http://<website>.com:9000
Is there some sort of specific settings I need to configure for Graylog?
I even following these instructions but didn’t have any luck:
Using a Layer 3 load balancer (forwarding TCP Ports)
Configure your load balancer to forward connections going to 192.168.0.10:80 to 127.0.0.1:9000 (web_listen_uri) and 192.168.0.10:9000/api/ to 127.0.0.1:9000/api/ (rest_listen_uri).
— In my case its VIP to Server IP:Port
Set web_endpoint_uri in your Graylog server config to http://graylog.example.org:9000/api/.
Start the Graylog server as usual.
Access the web interface on http://graylog.example.org.
So the way our network is setup is, if a client hits http://<website>.com:9000, it first goes to the Load Balancer/firewall, then comes to our server
Now if I view Graylog behind Load Balancer/Firewall, I am able to view the Web UI http://<HOST-IP>:9000. Issue is if I try to access it out side Load Balancer/Firewall, Web UI doesn’t show.
I make sure Load Balancer/Firewall are correctly setup by launching another app on port 9000 like Grafana, which showed up without any issues.That made me think that maybe Graylog config needs to be setup in a certain way.
I have set the web endpoint to following, which is the server ip. With this I am able to access it behind the LB/firewall, but not outside still. Not sure why Graylog doesn’t show up outside while other apps do without any issues.
Is http://10.179.84.64:9000 really the correct URI for the Graylog REST API? Are you sure about that?
Just asking because 10.179.84.64 looks a lot like an IP address from the 10.0.0.0/8 IP range defined in RFC 1918. In other words, it’s a private IP which can only be accessed within your network.
I have tried our external IP too which I get when I ping our site URL to test that out. That didn’t work either unfortunately.
Using the external IP, I get “Server currently unavailable” when view Graylog BEHIND the LoadBalancer/firewall
When use server IP, I am able to view and login Graylog
I think you need to set all servers to use the address of the server. You should not use the VIP address in graylog config at all. I.e. if your servers are 10.0.0.1 and 10.0.0.2, you use these IP addresses in the server configs. The VIP address is used only in the load balancer config.
Then, setting up https, depending on how you set it up (whether the load balancer is passing through TLS or not), you might need to set both vip and the server ip addresses and names in the certificate.
After a quick rethinking. If the server works fine, it must be your load balancer config anyway. If you load balancer does not report any error, you could also try using tcpdump on the graylog server to see whether any traffic comes to the graylog port.
I have been messing around a little more, but still didn’t have much luck. @jtkarvo, so my thinking was the same, that the issue is on Load Balancer side. However I deployment Grafana using the same port, and it came up fine, which makes me think that its the way Graylog is showing its UI that needs to be configured. I even have Jenkins running in the same setup.
We actually have two Load Balancer in our network, what I did was, I launched GRAFANA with port 9000 and I was able to reach it following URLS:
Load Balancer VIP 1: IP:9000
Load Balancer VIP 2: IP:9000
HOST: NAME/IP:9000
Then I launched again GRAYLOG on port 9000 and I was ONLY able to reach it on:
HOST: NAME/IP:9000
I tried setting WEB_ENDPOINT_URI to URL, VIP 1 IP and VIP 2 IP, but unfortunately none of those worked.
Another thing I guess Ill try is to get older version of Graylog to see if that works.