The file descriptor is unnormal


#1

I found a simmilar problem, but that has no end, and before I report as a bug, I would like to understand the full picture.
https://github.com/Graylog2/graylog2-server/issues/4375

Environment:
CentOS 7.5 3.10.0-862.11.6.el7.x86_64
4 Graylog severs v2.4.6+ceaa7e4
10 Elasticsearch servers v5.6.10
54 opened ports (TCP+UDP) (one for every server role)
~5k log/s

GL config:

node_id_file = /etc/graylog/server/node-id
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = true
elasticsearch_cluster_name = XX
elasticsearch_discovery_zen_ping_multicast_enabled = false
elasticsearch_hosts = http://IP:9200, ...
elasticsearch_max_total_connections = 50
elasticsearch_max_total_connections_per_route = 4
elasticsearch_connect_timeout = 20s
elasticsearch_analyzer = standard
elasticsearch_request_timeout = 2m
elasticsearch_index_optimization_timeout = 1h
output_batch_size = 1000
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 48h
message_journal_max_size = 95gb
message_journal_flush_age = 1m
message_journal_flush_interval = 25000
message_journal_segment_age = 15m
message_journal_segment_size = 100mb
lb_recognition_period_seconds = 3
lb_throttle_threshold_percentage = 50
mongodb_uri = mongodb://XX:XX@IP:27017,IP:27017,IP:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

Problem/Interesting thing:

# systemctl stop graylog-server
# lsof | wc -l
2782
# systemctl start graylog-server
# lsof | wc -l
3291106

Little slice of the lsof

...
java       8569            996 2416u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2417r     FIFO                0,9       0t0    2755180 pipe
java       8569            996 2418w     FIFO                0,9       0t0    2755180 pipe
java       8569            996 2419u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2420u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2421r     FIFO                0,9       0t0    2755181 pipe
java       8569            996 2422w     FIFO                0,9       0t0    2755181 pipe
java       8569            996 2423u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2424r     FIFO                0,9       0t0    2755182 pipe
java       8569            996 2425w     FIFO                0,9       0t0    2755182 pipe
java       8569            996 2426u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2427r     FIFO                0,9       0t0    2755183 pipe
java       8569            996 2428w     FIFO                0,9       0t0    2755183 pipe
java       8569            996 2429u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2430r     FIFO                0,9       0t0    2755184 pipe
java       8569            996 2431w     FIFO                0,9       0t0    2755184 pipe
java       8569            996 2432u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2433r     FIFO                0,9       0t0    2779476 pipe
java       8569            996 2434w     FIFO                0,9       0t0    2779476 pipe
java       8569            996 2435u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2436r     FIFO                0,9       0t0    2779477 pipe
java       8569            996 2437w     FIFO                0,9       0t0    2779477 pipe
java       8569            996 2438u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2439r     FIFO                0,9       0t0    2779478 pipe
java       8569            996 2440w     FIFO                0,9       0t0    2779478 pipe
java       8569            996 2441u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2442r     FIFO                0,9       0t0    2779479 pipe
java       8569            996 2443w     FIFO                0,9       0t0    2779479 pipe
java       8569            996 2444u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2445r     FIFO                0,9       0t0    2779480 pipe
java       8569            996 2446w     FIFO                0,9       0t0    2779480 pipe
java       8569            996 2447u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2448r     FIFO                0,9       0t0    2779481 pipe
java       8569            996 2449w     FIFO                0,9       0t0    2779481 pipe
java       8569            996 2450u  a_inode               0,10         0       7150 [eventpoll]
java       8569            996 2451r     FIFO                0,9       0t0    2779482 pipe
...

Statistic about the 10th column

# cat /tmp/lsof.txt | awk '{print $10}' | sort | uniq -c | sort -n -r
2078954 pipe
1039478 [eventpoll]
   4456 socket
   3325 /dev/urandom
   3321 /dev/random
   2240 /
   2214 /usr/share/graylog-server/plugin/graylog-plugin-threatintel-2.4.6.jar
   2214 /usr/share/graylog-server/plugin/graylog-plugin-pipeline-processor-2.4.6.jar
   2214 /usr/share/graylog-server/plugin/graylog-plugin-netflow-2.4.6.jar
   2214 /usr/share/graylog-server/plugin/graylog-plugin-map-widget-2.4.6.jar
   2214 /usr/share/graylog-server/plugin/graylog-plugin-enterprise-integration-2.4.6.jar
   2214 /usr/share/graylog-server/plugin/graylog-plugin-collector-2.4.6.jar
   2214 /usr/share/graylog-server/plugin/graylog-plugin-cef-2.4.6.jar
   2214 /usr/share/graylog-server/plugin/graylog-plugin-beats-2.4.6.jar
   2214 /usr/share/graylog-server/plugin/graylog-plugin-aws-2.4.6.jar
   2214 /usr/share/graylog-server/plugin/graylog-output-syslog-2.1.1.jar
   2214 /usr/share/graylog-server/plugin/graylog-output-euroone-syslog-2.3.1.jar.OLD
   2214 /usr/share/graylog-server/plugin/graylog-output-euroone-syslog-2.3.1.jar.bck
   2214 /usr/share/graylog-server/plugin/graylog-output-euroone-syslog-2.3.1.jar
   2214 /usr/share/graylog-server/graylog.jar
   2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/rt.jar
   2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/resources.jar
   2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/jsse.jar
   2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/jce.jar
   2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/ext/sunjce_provider.jar
   2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/ext/sunec.jar
   2214 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/ext/nashorn.jar
   2214 *:3024
   2214 *:3014
   2214 *:2051
   2214 *:2019
   2214 *:2018
   2214 *:2016
   2214 *:2015
   2214 *:2010
   2214 *:2005

So the things what I would like to understand,
Is it normal? Or it is a bug?
Why graylog opens files multiple times?
Is it cause any problem if we will open more ports? (we plan about 500 more)
How can I decrease the number of opened files?


#2

Someone any idea? No one faced the same problem?


(system) #3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.