“Show Receiver messages”, the Loading never stop


#1

Hello everyone.

I have a classical problem, when I do “Show Receiver messages”, the Loading never stop.

Let me explain my context,

I have 4 Virtual Machine:

Graylog : 192.168.159.163  (Graylog 2.4.6)
MongoDB :192.168.159.165 (db version v2.6.12)
Elasticsearch : 192.168.159.159 ("lucene_version" : "5.5.4" , "number" : "2.4.6")
Rsyslog : 192.168.159.166

This is my configurations:

Graylog :

[root@graylogv2 ~]# netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1071/master
tcp        0      0 0.0.0.0:1025            0.0.0.0:*               LISTEN      1373/java
tcp        0      0 192.168.159.163:9000    0.0.0.0:*               LISTEN      1373/java
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      899/sshd

serveur.conf

is_master = true
node_id_file = /etc/graylog/server/node-id
elasticsearch_max_docs_per_index = 20000000
password_secret = 6HEHVhdIYNEzao1xDSLGW456789V8qj5cU3mBDPdcRp2YJNMdfCJgWQgOFzTqNZFtMJjcyWTxVmDmBVXPZMoY1mwny9
root_password_sha2 = bb4cd01aa7c719c31234567115e95970af73d3a2c0d6202ceaf3d1183928b
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://192.168.159.163:9000/api/
#rest_listen_uri = http://127.0.0.1:9000/api/
web_listen_uri = http://192.168.159.163:9000/
#web_listen_uri = http://127.0.0.1:9000/
rotation_strategy = count
elasticsearch_hosts = http://192.168.159.159:9200
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 2
outputbuffer_processors = 2
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://192.168.159.165:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
root_timezone = Europe/Paris

Mongodb

vi /etc/mongod.conf
bind_ip = 192.168.159.165

Netstat:

[root@mongov2 ~]# netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1137/master     
tcp        0      0 192.168.159.165:27017   0.0.0.0:*               LISTEN      22457/mongod  

ElasticSearch

vi /etc/elasticsearch/elasticsearch.yml

cluster.name: graylog
network.host: ["127.0.0.1","192.168.159.159"]

netstat

[root@elastic ~]# netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1149/master
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1020/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      1149/master
tcp6       0      0 192.168.159.159:9200    :::*                    LISTEN      1026/java
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      1026/java
tcp6       0      0 192.168.159.159:9300    :::*                    LISTEN      1026/java
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      1026/java
tcp6       0      0 :::22                   :::*                    LISTEN      1020/sshd

Rsyslog

On rsyslog.conf

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 1025

$template RemoteLogsTesting,"/var/log/remotehosts/%HOSTNAME%/%$now%.log"
if $fromhost-ip != '127.0.0.1' then -?RemoteLogsTesting
& stop

*.* @@192.168.159.163:1025

My all log coming on rsyslog

[root@rsyslog remotehosts]# tree
.
├── elastic
│   ├── 2018-10-01.log
│   ├── 2018-10-02.log
│   ├── 2018-10-03.log
│   └── 2018-10-04.log
├── graylogv2
│   ├── 2018-10-01.log
│   ├── 2018-10-02.log
│   ├── 2018-10-03.log
│   └── 2018-10-04.log
└── mongov2
    ├── 2018-10-01.log
    ├── 2018-10-02.log
    ├── 2018-10-03.log
    └── 2018-10-04.log

This is my imput:

I supose i do a mistake or I forgot somthing !

Thank you for reading.


#2

Edit : I just try to disable IPv6 on all Vms.

And upgrade Mongodb:

[root@mongoV2 ~]# mongod --version
db version v4.0.3
git version: 7ea530946fa7880364d88c8d8b6026bbc9ffa48c
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: none
build environment:
    distmod: rhel70
    distarch: x86_64
    target_arch: x86_64 

The problem is the same.

Thank you.


#3

Hello,

When I do

[root@graylogv2 ~]# tcpdump -i ens33 host 192.168.159.166 and tcp port 1025

I have :

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes
10:36:16.095306 IP elastic.38956 > rsyslog.blackjack: Flags [P.], seq 1521063633:1521063831, ack 834329731, win 229, options [nop,nop,TS val 4845135 ecr 4259073], length 198
10:36:16.095344 IP rsyslog.blackjack > elastic.38956: Flags [.], ack 198, win 463, options [nop,nop,TS val 4434358 ecr 4845135], length 0
10:36:16.098380 IP elastic.38956 > rsyslog.blackjack: Flags [P.], seq 198:291, ack 1, win 229, options [nop,nop,TS val 4845138 ecr 4434358], length 93
10:36:16.098468 IP rsyslog.blackjack > elastic.38956: Flags [.], ack 291, win 463, options [nop,nop,TS val 4434362 ecr 4845138], length 0
10:36:16.100600 IP elastic.38956 > rsyslog.blackjack: Flags [P.], seq 291:354, ack 1, win 229, options [nop,nop,TS val 4845140 ecr 4434362], length 63
10:36:16.100690 IP rsyslog.blackjack > elastic.38956: Flags [.], ack 354, win 463, options [nop,nop,TS val 4434364 ecr 4845140], length 0
10:36:16.101724 IP elastic.38956 > rsyslog.blackjack: Flags [P.], seq 354:425, ack 1, win 229, options [nop,nop,TS val 4845141 ecr 4434364], length 71
10:36:16.101830 IP rsyslog.blackjack > elastic.38956: Flags [.], ack 425, win 463, options [nop,nop,TS val 4434365 ecr 4845141], length 0
10:36:16.102949 IP elastic.38956 > rsyslog.blackjack: Flags [P.], seq 425:491, ack 1, win 229, options [nop,nop,TS val 4845142 ecr 4434365], length 66

So the connexion is good I think

Thank you


(Jan Doberstein) #4

on the right of the input, you have metrics displayed. Did you see any stats counting?


#5

Hello Jan,
Thank you for your time.

I supose I have not metrics.

I dont understant why because the tcpdump on port 1025 works.

For information, i have upgrade Elasticseach to 5.6 from 2.x. The problem is the same.

Thank you.


#6

I try other configuration, the “Total” move a bit:

But the “Loading” status stay the same :frowning:

Thank you.


#7

Hello,

I solve the probleme.

It come from Elasticsearch, I have beging with version 2, and upgrade for version 5.

I do the same configuration from scratch, but I gegin by the version 5, and now that works.
Normaly I could deleted and recreate node, but now all is ok.

Thank you for your advisement.