Logs vanish in black hole?

Graylog Central (peer support)
1

Hello,
in our Graylog setup mysteriously logs seem to vanish.

Graylog Version 2.4.4, Cluster Setup with 2 nodes, both nodes show the same behaviour.
Graylog runs on SLES 12 SP2.
Port forwarding from 514 to 10514 is defined like described in your FAQ:
iptables -t nat -A PREROUTING -p UDP -m udp --dport 514 -j REDIRECT --to-ports 10514
iptables -t nat -A PREROUTING -p TCP -m tcp --dport 514 -j REDIRECT --to-ports 10514

Now we receive Logs via Port 514 from different systems without problems.
Some other systems send logs as well, but those logs don’t show up in Graylog, they are vanished.
Now, if I check wiht tcpdump the input from a specific system (one of the missing logs system) I can see logs coming:
graylog1:/ # tcpdump -i eth0 port 514 and src 10.148.22.1 -v
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
09:06:00.898935 IP (tos 0x0, ttl 26, id 35740, offset 0, flags [none], proto UDP (17), length 145)
axvxx123.unix.it.example.38557 > graylog1.int.it.example.syslog: SYSLOG, length: 117
Facility auth (4), Severity debug (7)
Msg: Jun 7 09:06:00 Message forwarded from axvxx123: sapuxuserchk PAM: pam_authenticate: error Authentication failed
^C
1 packet captured
9 packets received by filter
0 packets dropped by kernel

Why goes this log message missing, i.e. I can not find it in Graylog?
The input in Graylog on Port 10514 is defined as RAW/Plaintext UDP.

The strange thing is, why can I receive logs from some systems, and not from others.
Any idea?

Thanks in advance,

Dietmar

2

What’s the complete configuration of the Graylog inputs?

Try using the correct protocol names /etc/protocols, i. e. “udp” and “tcp” (lower case).

3

Hello,
the iptables rule is now in lower case.

The input configuration is below (extracted fro REST API browser):

{
  "inputs": [
    {
      "title": "appliance-gelf-udp",
      "global": true,
      "name": "GELF UDP",
      "content_pack": "5a14262cf8e0854c54b496c1",
      "created_at": "2017-11-21T13:15:00.721Z",
      "type": "org.graylog2.inputs.gelf.udp.GELFUDPInput",
      "creator_user_id": "admin",
      "attributes": {
        "override_source": null,
        "recv_buffer_size": 1048576,
        "bind_address": "0.0.0.0",
        "port": 12201,
        "decompress_size_limit": 8388608
      },
      "static_fields": {},
      "node": null,
      "id": "573335b6055ef70381c00bc6"
    },
    {
      "title": "Syslog TCP",
      "global": true,
      "name": "Syslog TCP",
      "content_pack": null,
      "created_at": "2017-11-21T13:15:38.560Z",
      "type": "org.graylog2.inputs.syslog.tcp.SyslogTCPInput",
      "creator_user_id": "admin",
      "attributes": {
        "recv_buffer_size": 1048576,
        "tcp_keepalive": false,
        "use_null_delimiter": false,
        "tls_client_auth_cert_file": "",
        "force_rdns": false,
        "bind_address": "0.0.0.0",
        "tls_cert_file": "",
        "store_full_message": false,
        "expand_structured_data": false,
        "port": 10514,
        "tls_key_file": "",
        "tls_enable": false,
        "tls_key_password": "",
        "max_message_size": 2097152,
        "tls_client_auth": "disabled",
        "override_source": null,
        "allow_override_date": true
      },
      "static_fields": {},
      "node": null,
      "id": "57333583055ef70381c00b90"
    },
    {
      "title": "appliance-gelf-tcp",
      "global": true,
      "name": "GELF TCP",
      "content_pack": "5a14262cf8e0854c54b496c1",
      "created_at": "2017-11-21T13:15:00.698Z",
      "type": "org.graylog2.inputs.gelf.tcp.GELFTCPInput",
      "creator_user_id": "admin",
      "attributes": {
        "recv_buffer_size": 4194304,
        "tcp_keepalive": false,
        "use_null_delimiter": true,
        "tls_client_auth_cert_file": "",
        "bind_address": "0.0.0.0",
        "tls_cert_file": "",
        "decompress_size_limit": 8388608,
        "port": 12201,
        "tls_key_file": "admin",
        "tls_enable": true,
        "tls_key_password": "gl4gkvi",
        "max_message_size": 2097152,
        "tls_client_auth": "disabled",
        "override_source": null
      },
      "static_fields": {},
      "node": null,
      "id": "575a680d055ef72001e86360"
    },
    {
      "title": "Beats_Input",
      "global": true,
      "name": "Beats",
      "content_pack": null,
      "created_at": "2018-02-09T13:11:15.651Z",
      "type": "org.graylog.plugins.beats.BeatsInput",
      "creator_user_id": "gki10090",
      "attributes": {
        "recv_buffer_size": 1048576,
        "port": 5044,
        "tls_key_file": "",
        "tls_enable": false,
        "tls_key_password": "",
        "tcp_keepalive": false,
        "tls_client_auth_cert_file": "",
        "tls_client_auth": "disabled",
        "override_source": null,
        "bind_address": "0.0.0.0",
        "tls_cert_file": ""
      },
      "static_fields": {},
      "node": null,
      "id": "58248056055ef7035ab5c96c"
    },
    {
      "title": "appliance-syslog-udp",
      "global": true,
      "name": "Syslog UDP",
      "content_pack": "5a14262cf8e0854c54b496c1",
      "created_at": "2017-11-21T13:15:00.744Z",
      "type": "org.graylog2.inputs.syslog.udp.SyslogUDPInput",
      "creator_user_id": "admin",
      "attributes": {
        "expand_structured_data": false,
        "recv_buffer_size": 33554432,
        "port": 1514,
        "override_source": null,
        "force_rdns": false,
        "allow_override_date": true,
        "bind_address": "0.0.0.0",
        "store_full_message": true
      },
      "static_fields": {},
      "node": null,
      "id": "573335eb055ef70381c00c02"
    },
    {
      "title": "RAW Syslog UDP",
      "global": true,
      "name": "Raw/Plaintext UDP",
      "content_pack": null,
      "created_at": "2017-11-21T13:15:24.818Z",
      "type": "org.graylog2.inputs.raw.udp.RawUDPInput",
      "creator_user_id": "admin",
      "attributes": {
        "override_source": null,
        "recv_buffer_size": 33554432,
        "bind_address": "0.0.0.0",
        "port": 10514
      },
      "static_fields": {},
      "node": null,
      "id": "573d6f23055ef73a3eae80d6"
    },
    {
      "title": "Syslog UDP 11514 (VMware)",
      "global": true,
      "name": "Syslog UDP",
      "content_pack": null,
      "created_at": "2018-02-13T07:55:18.783Z",
      "type": "org.graylog2.inputs.syslog.udp.SyslogUDPInput",
      "creator_user_id": "gki10090",
      "attributes": {
        "expand_structured_data": false,
        "recv_buffer_size": 262144,
        "port": 11514,
        "override_source": null,
        "force_rdns": false,
        "allow_override_date": true,
        "bind_address": "0.0.0.0",
        "store_full_message": false
      },
      "static_fields": {},
      "node": null,
      "id": "5a8299e6f8e0855817281ab1"
    },
    {
      "title": "ZISLog_GELF_UDP",
      "global": true,
      "name": "GELF UDP",
      "content_pack": "5a14262cf8e0854c54b496c1",
      "created_at": "2017-11-21T13:15:00.715Z",
      "type": "org.graylog2.inputs.gelf.udp.GELFUDPInput",
      "creator_user_id": "admin",
      "attributes": {
        "override_source": null,
        "recv_buffer_size": 1048576,
        "bind_address": "0.0.0.0",
        "port": 12211,
        "decompress_size_limit": 8388608
      },
      "static_fields": {},
      "node": null,
      "id": "5a0c0e41055ef7566cad1f5a"
    }
  ],
  "total": 8
}

Thanks in advance.

Dietmar

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.