Graylog open random UDP port

1. Describe your incident:
Graylog opens on all our instances a UDP port of a random port number. The port is not in LISTEN state and is sometimes a udp6 port and sometime udp (IPv4).

Example:
udp 0 0 0.0.0.0:55944 0.0.0.0:* 32535/java
or
udp6 0 0 :::47218 :::* 130470/java

Because of the PID in front of the “/java” i know that it is graylog since the same PID opens graylogs input ports as well for receiving logs.

2. Describe your environment:

  • OS Information:
    Debian 10.11
  • Package Version:
    Graylog 4.2.4+b643d2b
    Same on Graylog 3.2.5+b0d3334
  • Service logs, configurations, and environment variables:
#/etc/default/graylog 

# Path to the java executable.
JAVA=/usr/bin/java

GRAYLOG_SERVER_JAVA_OPTS="-Xms24g -Xmx24g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow"
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true"
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true"
GRAYLOG_SERVER_ARGS=""
GRAYLOG_COMMAND_WRAPPER=""
# /etc/graylog/server/server.conf

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = ...
root_username = ...
root_password_sha2 = ...
root_email = "..."
root_timezone = CET
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin

http_bind_address = 127.0.0.1:9000
http_enable_cors = true
http_enable_gzip = true

trusted_proxies = 127.0.0.1/32
ring_size = 65536
allow_leading_wildcard_searches = true
allow_highlighting = false

elasticsearch_hosts = http://127.0.0.1:9200
elasticsearch_max_total_connections = 200
elasticsearch_max_total_connections_per_route = 200
elasticsearch_discovery_enabled = false
elasticsearch_shards = 1
elasticsearch_replicas = 0
elasticsearch_analyzer = standard

rotation_strategy = time
retention_strategy = delete
#index_ranges_cleanup_interval = 1h
elasticsearch_max_number_of_indices = 7
elasticsearch_index_prefix = graylog

elasticsearch_max_time_per_index = 1d

inputbuffer_ring_size = 65536
inputbuffer_processors = 8
inputbuffer_wait_strategy = blocking

processbuffer_processors = 14

output_batch_size = 10000
output_flush_interval = 1
outputbuffer_processors = 50
outputbuffer_processor_threads_core_pool_size = 30
outputbuffer_processor_threads_max_pool_size = 50
outputbuffer_processor_keep_alive_time = 10000
processor_wait_strategy = blocking
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30

message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 24h
message_journal_max_size = 128gb
message_journal_flush_age = 1s
message_journal_flush_interval = 1000

lb_recognition_period_seconds = 3

mongodb_uri = mongodb://127.0.0.1:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5

http_connect_timeout = 10s
http_read_timeout = 20s
http_write_timeout = 20s

content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json

proxied_requests_thread_pool_size = 32

enabled_tls_protocols = TLSv1.1,TLSv1.2

#
# Custom Plugin
#
custom_statsd_host = 127.0.0.1
custom_statsd_port = 8125
custom_mongodb_uri = mongodb://127.0.0.1:27017/custom

3. What steps have you already taken to try and solve the problem?
Googled the issue in the context of graylog and java, searched here in the community forum

4. How can the community help?
Since the port is not in LISTEN state connection will be rejected but for the sake of port hygiene I want to know what it is, were it comes from and if I am able to get rid of it or pin it to a specific port to be able to apply firewall rules to it. A not-listening port nobody can connect to is quite useless, I guess.

As well, it is bound to all interfaces. As soon as the LISTEN state changes (I don’t know when or if at all) it would be open to everybody…

Cheers
Thomas

Hello,

I might be able to help.

Correct me if I’m wrong but you have a INPUT and its not listing to that port, or you have a INPUT and the port changes? is there any way you can show this issue?

sudo lsof -i -P -n | grep LISTEN

Do you have a static IP address on Graylog?
How did you configure your firewall to allow these ports through? Was it something like this below?

iptables -A INPUT -i eth0 -p tcp --dport 3000 -m state --state NEW,ESTABLISHED -j ACCEPT

If your using a loop back IP Address (127.0.0.1) how do other people connect to Graylog?

Thanks

Hey,

thanks for your help!

I have an INPUT with a fixed port. It is not the port I am talking about. I do not use any iptable fw functionality (neither filtering nor forwarding rules). On the public machines I have a public IP bound to it. Of course the node is behind a firewall allowing specific ports to pass through. Nothing special there.

A GREP on LISTEN is way too long. Additionally, the port in question would not show up since it is not in LISTEN state. I can filter it down to graylog and exclude the INPUT port:

sudo lsof -i -P -n | grep graylog | grep -v 5044
java       56613        graylog   34u  IPv4 1033912034      0t0  TCP 127.0.0.1:16608->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog   35u  IPv4 1033915769      0t0  TCP 127.0.0.1:16610->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog   39u  IPv4 1033912035      0t0  UDP *:33074
java       56613        graylog   40u  IPv4 1033910165      0t0  TCP 127.0.0.1:16612->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog   41u  IPv4 1033912036      0t0  TCP 127.0.0.1:16614->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog   42u  IPv4 1033918919      0t0  TCP 127.0.0.1:16616->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog   91u  IPv4 1033921610      0t0  TCP 127.0.0.1:9000 (LISTEN)
java       56613        graylog   92u  IPv4 1033910170      0t0  TCP 127.0.0.1:16618->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog   93u  IPv4 1033912038      0t0  TCP 127.0.0.1:16620->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog   94u  IPv4 1033913682      0t0  TCP 127.0.0.1:16622->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog   95u  IPv4 1033910175      0t0  TCP 127.0.0.1:16624->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog   96u  IPv4 1033918924      0t0  TCP 127.0.0.1:41900->127.0.0.1:9200 (ESTABLISHED)
java       56613        graylog   97u  IPv4 1033916655      0t0  TCP 127.0.0.1:41902->127.0.0.1:9200 (ESTABLISHED)
java       56613        graylog   98u  IPv4 1033916656      0t0  TCP 127.0.0.1:41906->127.0.0.1:9200 (ESTABLISHED)
java       56613        graylog   99u  IPv4 1033919500      0t0  TCP 127.0.0.1:41904->127.0.0.1:9200 (ESTABLISHED)
java       56613        graylog  129u  IPv4 1033921612      0t0  TCP 127.0.0.1:16660->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog  159u  IPv4 1034449475      0t0  TCP 127.0.0.1:34752->127.0.0.1:9200 (ESTABLISHED)
java       56613        graylog  174u  IPv4 1034610712      0t0  TCP 127.0.0.1:49570->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog  175u  IPv4 1034605333      0t0  TCP 127.0.0.1:49572->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog  176u  IPv4 1034609809      0t0  TCP 127.0.0.1:49574->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog  177u  IPv4 1034608792      0t0  TCP 127.0.0.1:49576->127.0.0.1:27017 (ESTABLISHED)
java       56613        graylog  259u  IPv4 1141627387      0t0  TCP 127.0.0.1:9000->127.0.0.1:57374 (ESTABLISHED)
java       56613        graylog  374u  IPv4 1141624557      0t0  TCP 127.0.0.1:9000->127.0.0.1:57378 (ESTABLISHED)
java       56613        graylog  387u  IPv4 1141976241      0t0  TCP 172.xx.17:19382->172.xx.17:443 (ESTABLISHED)
java       56613        graylog  389u  IPv4 1141968421      0t0  TCP 127.0.0.1:31614->127.0.0.1:9000 (CLOSE_WAIT)

The UDP line is the one. Port 33074. The port is always a different after restarting graylog. Bound to all interfaces.

Regards
Thomas

Hello

Can you show your INPUT?

I’m not sure what’s going on in your environment.
Are you using the same port as the remote device and Graylog INPUT?

You may need to try this command.

sudo lsof -i -P -n | grep 5044

Which gives me this output

[root@graylog graylog_user]# sudo lsof -i -P -n | grep 5044
java      1564       graylog  247u  IPv6   35034      0t0  TCP *:5044 (LISTEN)
java      1564       graylog  360u  IPv6   32579      0t0  TCP 8.8.8.70:5044->8.8.8.20:54158 (ESTABLISHED)
java      1564       graylog  370u  IPv6   35046      0t0  TCP 8.8.8.70:5044->8.8.8.28:50928 (ESTABLISHED)
java      1564       graylog  416u  IPv6   46137      0t0  TCP 8.8.8.70:5044->8.8.8.157:49700 (ESTABLISHED)

My input for that port.

Could you explain in greater details what you mean by that statement?

Hey gsmith,

thanks for your help.

It is definitely not related to our input.

I just wanted to say that the datacenter setup is definitely not relevant. I should have mentioned that it happens in local environments, too.

I digged a little bit deeper and found my own plugin the source of the port. Without the plugin port in question is not opened. Is that a general behavior of a graylog plugin? If not, it has to be the Java StatsD or MongoDB library. The plugin provides pipeline methods which store data in graylogs MongoDB and reports metrics via StatsD.

Hello,

My apologies, I’m kind of confused with this post. What I understand so far is that there is a random port that opens BUT you don’t have a firewall configured or enabled.

But your input, is set to 0.0.0.0- Port 55944, this correct?

So you have another INPUT with an IP Address? Is this correct?

Can I ask what plugin are you referring too? Was this plugin from Graylog Market or did you create one?

So I guess what I’m asking is you have a random port opening on graylog? is this correct?
You stated that firewall is not enabled. Is this correct? If so that tells me you have ALL ports are opened.

In your first statement you stated.

Since your using UDP (I hope my data gets to its destination) protocol. This is normally a one way trip. UDP doesn’t have a three way handshake like TCP and I haven’t seen Graylog opening different port on other instances.

Never had a plugin open a different port, I DROP ALL connections unless its is specified in my Firewall configuration.
Any connection outside my environment. My firewall has specific policy’s to only allow those ports needed on Graylog that is connecting to the internet.

If you could enlighten me a little more it would be appreciated.

Yes. I do not use iptables. Any service which opens a port the port is reachable by anyone in the network. This is fine since this is a network with trusted nodes. Additionally, I configure the services in such a whay that I close ports or bind them to localhost to prevent any connection from the network if possible and where necessary. Like this I get a small list with services with open ports bound to the interfaces they have to listen to. That’s the reason why I want to close the port or at least bind it to the localhost interface. There is a hardware firewall, of course, in the network infrastructure which limits traffic from “outside”.

My Input, the only Graylog input:

No other input.

I wrote it myself

Yes

Of course, I could just add a rule to iptables blocking the port. But this is not my way of doing things. Basically, this would be the way of “fixing things with a workaround”. There is an open port and I do not configure the service to bind it to localhost or close it. So I deploy another tool which prevents any traffic to that port. Though, in case where the port needs to be accessable only by a few source IPs it makes totally sense to use iptables to limit it.
I was just wondering where the port comes from and wanted to verify that this is not a standard graylog or graylog plugin behavior, which can be configured in some way. Since the source of all evil is my own plugin and I can’t seem to configure in such a way that the port is not bound to all interfaces I have to leave it like that :confused:

1 Like

Hello @staelche

First, my apologies for not understanding this earlier. When I reply to post/s I’m at work, so I get sidetracked.
Thank you for the added details. What you just posted I now completely understand. As for plugin I guess this would be what resources its using. After re-reading your earlier statement I see java. Since most componites on Graylog use JAVA and you create your own plugin (Which is pretty Kool :+1: ) something funky is going on. That would be the first place I would look. I assume you looked up the PID to find out what is opening the port?

root # ls /proc/some_process_id

Since this is custom plugin I’m not going to be much help. What caught my eye was “Graylog open random UDP Port”. Now realizing this issue is from a custom plugin I’m not 100% sure if this is a standard graylog or graylog plugin behavior, but I would take a guess that it is in which all depends on what the plugin is used for and how it works within your environment. And as you can see Graylog plugin can do a lot of work.

  • Inputs: Accept/write any messages into Graylog
  • Outputs: Forward ingested messages to other systems as they are processed
  • Services: Run at startup and able to implement any functionality
  • Event Notifications: Called when an event alert has been triggered
  • Processors: Transform/drop incoming messages (can create multiple new messages)
  • Filters: (Deprecated) Transform/drop incoming messages during processing
  • REST API Resources: An HTTP resource exposed as part of the Graylog REST API
  • Periodical: Called at periodical intervals during server runtime
  • Decorators: Used during search time to modify the presentation of messages
  • Authentication Realms: Allowing to implement different authentication mechanisms (like single sign-on or 2FA)

https://docs.graylog.org/v1/docs/plugins

So judging what it capable of It very well that it can open a random port within your Graylog Server.
That’s all I have for you, sorry I don’t have a direct answer but maybe someone else here can jump in that may know more about custom plugin/s.