1. Describe your incident:
Graylog opens on all our instances a UDP port of a random port number. The port is not in LISTEN state and is sometimes a udp6 port and sometime udp (IPv4).
3. What steps have you already taken to try and solve the problem?
Googled the issue in the context of graylog and java, searched here in the community forum
4. How can the community help?
Since the port is not in LISTEN state connection will be rejected but for the sake of port hygiene I want to know what it is, were it comes from and if I am able to get rid of it or pin it to a specific port to be able to apply firewall rules to it. A not-listening port nobody can connect to is quite useless, I guess.
As well, it is bound to all interfaces. As soon as the LISTEN state changes (I don’t know when or if at all) it would be open to everybody…
Correct me if I’m wrong but you have a INPUT and its not listing to that port, or you have a INPUT and the port changes? is there any way you can show this issue?
sudo lsof -i -P -n | grep LISTEN
Do you have a static IP address on Graylog?
How did you configure your firewall to allow these ports through? Was it something like this below?
iptables -A INPUT -i eth0 -p tcp --dport 3000 -m state --state NEW,ESTABLISHED -j ACCEPT
If your using a loop back IP Address (127.0.0.1) how do other people connect to Graylog?
I have an INPUT with a fixed port. It is not the port I am talking about. I do not use any iptable fw functionality (neither filtering nor forwarding rules). On the public machines I have a public IP bound to it. Of course the node is behind a firewall allowing specific ports to pass through. Nothing special there.
A GREP on LISTEN is way too long. Additionally, the port in question would not show up since it is not in LISTEN state. I can filter it down to graylog and exclude the INPUT port:
I just wanted to say that the datacenter setup is definitely not relevant. I should have mentioned that it happens in local environments, too.
I digged a little bit deeper and found my own plugin the source of the port. Without the plugin port in question is not opened. Is that a general behavior of a graylog plugin? If not, it has to be the Java StatsD or MongoDB library. The plugin provides pipeline methods which store data in graylogs MongoDB and reports metrics via StatsD.
My apologies, I’m kind of confused with this post. What I understand so far is that there is a random port that opens BUT you don’t have a firewall configured or enabled.
But your input, is set to 0.0.0.0- Port 55944, this correct?
So you have another INPUT with an IP Address? Is this correct?
Can I ask what plugin are you referring too? Was this plugin from Graylog Market or did you create one?
So I guess what I’m asking is you have a random port opening on graylog? is this correct?
You stated that firewall is not enabled. Is this correct? If so that tells me you have ALL ports are opened.
In your first statement you stated.
Since your using UDP (I hope my data gets to its destination) protocol. This is normally a one way trip. UDP doesn’t have a three way handshake like TCP and I haven’t seen Graylog opening different port on other instances.
Never had a plugin open a different port, I DROP ALL connections unless its is specified in my Firewall configuration.
Any connection outside my environment. My firewall has specific policy’s to only allow those ports needed on Graylog that is connecting to the internet.
If you could enlighten me a little more it would be appreciated.
Yes. I do not use iptables. Any service which opens a port the port is reachable by anyone in the network. This is fine since this is a network with trusted nodes. Additionally, I configure the services in such a whay that I close ports or bind them to localhost to prevent any connection from the network if possible and where necessary. Like this I get a small list with services with open ports bound to the interfaces they have to listen to. That’s the reason why I want to close the port or at least bind it to the localhost interface. There is a hardware firewall, of course, in the network infrastructure which limits traffic from “outside”.
Of course, I could just add a rule to iptables blocking the port. But this is not my way of doing things. Basically, this would be the way of “fixing things with a workaround”. There is an open port and I do not configure the service to bind it to localhost or close it. So I deploy another tool which prevents any traffic to that port. Though, in case where the port needs to be accessable only by a few source IPs it makes totally sense to use iptables to limit it.
I was just wondering where the port comes from and wanted to verify that this is not a standard graylog or graylog plugin behavior, which can be configured in some way. Since the source of all evil is my own plugin and I can’t seem to configure in such a way that the port is not bound to all interfaces I have to leave it like that
First, my apologies for not understanding this earlier. When I reply to post/s I’m at work, so I get sidetracked.
Thank you for the added details. What you just posted I now completely understand. As for plugin I guess this would be what resources its using. After re-reading your earlier statement I see java. Since most componites on Graylog use JAVA and you create your own plugin (Which is pretty Kool ) something funky is going on. That would be the first place I would look. I assume you looked up the PID to find out what is opening the port?
root # ls /proc/some_process_id
Since this is custom plugin I’m not going to be much help. What caught my eye was “Graylog open random UDP Port”. Now realizing this issue is from a custom plugin I’m not 100% sure if this is a standard graylog or graylog plugin behavior, but I would take a guess that it is in which all depends on what the plugin is used for and how it works within your environment. And as you can see Graylog plugin can do a lot of work.
Inputs: Accept/write any messages into Graylog
Outputs: Forward ingested messages to other systems as they are processed
Services: Run at startup and able to implement any functionality
Event Notifications: Called when an event alert has been triggered
Processors: Transform/drop incoming messages (can create multiple new messages)
Filters: (Deprecated) Transform/drop incoming messages during processing
REST API Resources: An HTTP resource exposed as part of the Graylog REST API
Periodical: Called at periodical intervals during server runtime
Decorators: Used during search time to modify the presentation of messages
Authentication Realms: Allowing to implement different authentication mechanisms (like single sign-on or 2FA)
So judging what it capable of It very well that it can open a random port within your Graylog Server.
That’s all I have for you, sorry I don’t have a direct answer but maybe someone else here can jump in that may know more about custom plugin/s.