I found a simmilar problem, but that has no end, and before I report as a bug, I would like to understand the full picture.
https://github.com/Graylog2/graylog2-server/issues/4375
Environment:
CentOS 7.5 3.10.0-862.11.6.el7.x86_64
4 Graylog severs v2.4.6+ceaa7e4
10 Elasticsearch servers v5.6.10
54 opened ports (TCP+UDP) (one for every server role)
~5k log/s
GL config:
node_id_file = /etc/graylog/server/node-id
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = true
elasticsearch_cluster_name = XX
elasticsearch_discovery_zen_ping_multicast_enabled = false
elasticsearch_hosts = http://IP:9200, ...
elasticsearch_max_total_connections = 50
elasticsearch_max_total_connections_per_route = 4
elasticsearch_connect_timeout = 20s
elasticsearch_analyzer = standard
elasticsearch_request_timeout = 2m
elasticsearch_index_optimization_timeout = 1h
output_batch_size = 1000
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 48h
message_journal_max_size = 95gb
message_journal_flush_age = 1m
message_journal_flush_interval = 25000
message_journal_segment_age = 15m
message_journal_segment_size = 100mb
lb_recognition_period_seconds = 3
lb_throttle_threshold_percentage = 50
mongodb_uri = mongodb://XX:XX@IP:27017,IP:27017,IP:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
Problem/Interesting thing:
# systemctl stop graylog-server
# lsof | wc -l
2782
# systemctl start graylog-server
# lsof | wc -l
3291106
Little slice of the lsof
...
java 8569 996 2416u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2417r FIFO 0,9 0t0 2755180 pipe
java 8569 996 2418w FIFO 0,9 0t0 2755180 pipe
java 8569 996 2419u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2420u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2421r FIFO 0,9 0t0 2755181 pipe
java 8569 996 2422w FIFO 0,9 0t0 2755181 pipe
java 8569 996 2423u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2424r FIFO 0,9 0t0 2755182 pipe
java 8569 996 2425w FIFO 0,9 0t0 2755182 pipe
java 8569 996 2426u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2427r FIFO 0,9 0t0 2755183 pipe
java 8569 996 2428w FIFO 0,9 0t0 2755183 pipe
java 8569 996 2429u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2430r FIFO 0,9 0t0 2755184 pipe
java 8569 996 2431w FIFO 0,9 0t0 2755184 pipe
java 8569 996 2432u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2433r FIFO 0,9 0t0 2779476 pipe
java 8569 996 2434w FIFO 0,9 0t0 2779476 pipe
java 8569 996 2435u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2436r FIFO 0,9 0t0 2779477 pipe
java 8569 996 2437w FIFO 0,9 0t0 2779477 pipe
java 8569 996 2438u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2439r FIFO 0,9 0t0 2779478 pipe
java 8569 996 2440w FIFO 0,9 0t0 2779478 pipe
java 8569 996 2441u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2442r FIFO 0,9 0t0 2779479 pipe
java 8569 996 2443w FIFO 0,9 0t0 2779479 pipe
java 8569 996 2444u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2445r FIFO 0,9 0t0 2779480 pipe
java 8569 996 2446w FIFO 0,9 0t0 2779480 pipe
java 8569 996 2447u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2448r FIFO 0,9 0t0 2779481 pipe
java 8569 996 2449w FIFO 0,9 0t0 2779481 pipe
java 8569 996 2450u a_inode 0,10 0 7150 [eventpoll]
java 8569 996 2451r FIFO 0,9 0t0 2779482 pipe
...
Statistic about the 10th column
# cat /tmp/lsof.txt | awk '{print $10}' | sort | uniq -c | sort -n -r
2078954 pipe
1039478 [eventpoll]
4456 socket
3325 /dev/urandom
3321 /dev/random
2240 /
2214 /usr/share/graylog-server/plugin/graylog-plugin-threatintel-2.4.6.jar
2214 /usr/share/graylog-server/plugin/graylog-plugin-pipeline-processor-2.4.6.jar
2214 /usr/share/graylog-server/plugin/graylog-plugin-netflow-2.4.6.jar
2214 /usr/share/graylog-server/plugin/graylog-plugin-map-widget-2.4.6.jar
2214 /usr/share/graylog-server/plugin/graylog-plugin-enterprise-integration-2.4.6.jar
2214 /usr/share/graylog-server/plugin/graylog-plugin-collector-2.4.6.jar
2214 /usr/share/graylog-server/plugin/graylog-plugin-cef-2.4.6.jar
2214 /usr/share/graylog-server/plugin/graylog-plugin-beats-2.4.6.jar
2214 /usr/share/graylog-server/plugin/graylog-plugin-aws-2.4.6.jar
2214 /usr/share/graylog-server/plugin/graylog-output-syslog-2.1.1.jar
2214 /usr/share/graylog-server/plugin/graylog-output-euroone-syslog-2.3.1.jar.OLD
2214 /usr/share/graylog-server/plugin/graylog-output-euroone-syslog-2.3.1.jar.bck
2214 /usr/share/graylog-server/plugin/graylog-output-euroone-syslog-2.3.1.jar
2214 /usr/share/graylog-server/graylog.jar
2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/rt.jar
2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/resources.jar
2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/jsse.jar
2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/jce.jar
2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/ext/sunjce_provider.jar
2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/ext/sunec.jar
2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/ext/nashorn.jar
2214 *:3024
2214 *:3014
2214 *:2051
2214 *:2019
2214 *:2018
2214 *:2016
2214 *:2015
2214 *:2010
2214 *:2005
So the things what I would like to understand,
Is it normal? Or it is a bug?
Why graylog opens files multiple times?
Is it cause any problem if we will open more ports? (we plan about 500 more)
How can I decrease the number of opened files?