I have nginx as reverse proxy and GrayLog on separate networks and servers. Nginx is the only server exposed to the outside. nginx and graylog do communicate properly and I can access the API externally no problem. My sidecar is communicating with GrayLog no problem via nginx. The issue is that it is not sending logs. Do I need to open a port directly to GrayLog exposing it to the internet which it is not ideal or do I have to open a port on nginx? Can nginx forward these logs to GrayLog, if so how? is it through stream or something?
This page covers a lot of ground and while long, is a good read:
https://go2docs.graylog.org/5-1/setting_up_graylog/web_interface.htm
In short, you do need someway for traffic to reach graylog, weather that is exposing the port from graylog directly, or configuring it via the load balancer.
Depending on what load balancer you use you may have to make some tradeoffs. For example some load balancers do not support UDP load balancing.
The guidance we have for load balancers in on the above linked page. Some graylog inputs do support TLS auth (to encrypt the traffic) and some also support mandating TLS client auth so that random people cannot send logs to your exposed input.