Graylog doesn't show all message in any INPUT

[S1L3NC3]

Hi all guys:
I’ve the following environment: 2 graylog-server , 3 Elastic-Search , 1 MongoDB. I’ve a strange behaviour: I created some inputs but in some of them I loose messages.

I did some checks:
1/ take with tcpdump the traffic coming from external devices, all of them appears (seeing the source).
2/ from Graylog UI these messages are not shown.
3/ from elastic I did;

curl -H 'Content-Type: application/json' -X GET http://localhost:9200/ads__0/_search?pretty |grep source

and it shows just from one source.

I don’t know what the next to check but I will be crazy with this issue

Some help?
Thanks in advance.

Note: Graylog 3.0.2

[macko003]

just some things
check the input - started on all GL node?
you send all log to GL or you put it in the ES database (eg. beats with 9200 port)
check your firewall
check graylog logs
check system overview - index failures (I can’t remember, it’s RED if you have problem…)

some possible reason:
you send wrong format of syslog, GL can’t recognize some field, so it will drop message
ES can’t store message, because you use another format. eg. level declarated as intiger, and you send string.
etc…

[S1L3NC3]

Hi macko003 :
Thanks for your reply.
Inputs are on all GL nodes. Issue is detected in one gelf input udp listening on graylog in port 5141 udp. Devices are configured with nxlog (4 windows) to send gelf udp.
Firewall is clean (iptables).
Graylog logs doesn’t reports errors and the system overview seems ok.
From tcpdump in graylog master I can see some packets coming to this port from all source devices, but for elastic I cant see some of them into index.

Same input config and nxlog config in other environment and with the same messages appears.

This is the first time i see this… so i cannot understand why ES cant storage all these messages.

Thanks in advange.

[macko003]

First check with tcpdump you got UDP or TCP packages.
what do you see under GL system - overview - index failures?

Honestly I can’t recomend use UDP, it is not garanteed the log processing (eg. known issue, at GL’s java’s GC process all udp dropped out…)

[S1L3NC3]

Many thanks for you reply @macko003 :
The check was made taking just udp packets for any interface and grep TCP and UDP:

tcpdump -vvv -i any port 5414 |grep TCP -> 50 seconds -> 0 packets
tcpdump -vvv -i any port 5414|grep UDP -> 50 seconds -> more than 900 messages

The UDP tcpdump check shows every “source” withouth problems (5 sources).

What I can see over GL system - overview - index failures --> “No failed indexing attemps in the last 24 hours”. No errors here.

So… next step will be the change from UDP to TCP port.

This is a strange behaviour becouse it is the first time I put UDP GELF or UDP Syslog and I’ve got “problems”.

Again many thanks

Best Regards,

[macko003]

yes… maybe I got it…
You have 2 GL servers!
ONE GL server needs to get ALL UDP message from one host. If the sender divide the message for more UDP packages it is possible the different servers got the message, so all servers got only a part of your message.
Maybe the working host sends shotr messages, or it sends only for one GL server.

I don’t know how nxlog sends gelf, you can check it with tcpdump … -AAA param.
If it’s not compressed, you will see the content.

https://docs.graylog.org/en/3.1/pages/gelf.html

All chunks **MUST** arrive within 5 seconds or the server will discard all already arrived and still arriving chunks. A message **MUST NOT** consist of more than 128 chunks.

[S1L3NC3]

Hi there again @macko003 :
Yes, it is. I’ve 2 GL server, so… I did some checks before open this issue here, one of these checks was using just one server and the behaviour is the same (have no all messages).

Anyway… I will check the last response of you about TCP and not UDP method.

I’ll come back with news , many thnks.