Input Messages are not received any more!

mbahaa · December 4, 2017, 8:11pm

i faced a problem that suddenly the input message stopped being received since days!

what shall i do please??

[root@Syslog_Trial ~]# tcpdump -i ens160 -n |  grep 10.10
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
22:10:10.315022 IP 10.39.224.10.ssh > 10.175.14.75.54479: Flags [P.], seq 690048666:690048862, ack 2489872760, win 258, length 196
22:10:10.318979 IP 10.175.14.75.54479 > 10.39.224.10.ssh: Flags [.], ack 196, win 254, length 0
22:10:10.687340 IP 10.10.20.2.63195 > 10.39.224.10.syslog: SYSLOG local3.debug, length: 75
22:10:10.687517 IP 10.10.20.2.63195 > 10.39.224.10.syslog: SYSLOG local3.debug, length: 80
22:10:10.687832 IP 10.10.20.2.63195 > 10.39.224.10.syslog: SYSLOG local3.debug, length: 117
22:10:11.478588 IP 10.10.10.1.38514 > 10.39.224.10.syslog: SYSLOG local7.info, length: 212
22:10:11.478768 IP 10.10.10.1.38514 > 10.39.224.10.syslog: SYSLOG local7.info, length: 211
22:10:11.479351 IP 10.10.10.1.38514 > 10.39.224.10.syslog: SYSLOG local7.info, length: 213
22:10:11.480095 IP 10.10.10.1.38514 > 10.39.224.10.syslog: SYSLOG local7.info, length: 212
22:10:11.583309 IP 10.10.10.1.38514 > 10.39.224.10.syslog: SYSLOG local7.info, length: 210

[root@Syslog_Trial ~]# tail -f /var/log/messages
Dec  4 20:11:15 UTO-NPE-NE40E-01 %%01BGP/6/SEND_NOTIFY(l):The router sent a NOTIFICATION message to peer 10.5.219.19. (ErrorCode=2, SubErrorCode=2, BgpAddressFamily=BNP-ATM:001, ErrorData=41040000fc00)
Dec  4 20:11:15 UTO-NPE-NE40E-01 %%01BGP/6/SEND_NOTIFY(l):The router sent a NOTIFICATION message to peer 10.5.219.39. (ErrorCode=2, SubErrorCode=2, BgpAddressFamily=BNP-ATM:001, ErrorData=41040000fc00)
Dec  4 20:11:15 UTO-NPE-NE40E-01 %%01BGP/6/SEND_NOTIFY(l):The router sent a NOTIFICATION message to peer 10.3.219.75. (ErrorCode=2, SubErrorCode=2, BgpAddressFamily=BNP-ATM:001, ErrorData=41040000fc00)
Dec  4 20:11:15 UTO-NPE-NE40E-01 %%01BGP/6/SEND_NOTIFY(l):The router sent a NOTIFICATION message to peer 10.5.219.35. (ErrorCode=2, SubErrorCode=2, BgpAddressFamily=BNP-ATM:001, ErrorData=41040000fc00)
Dec  4 20:11:15 UTO-NPE-NE40E-01 %%01BGP/6/SEND_NOTIFY(l):The router sent a NOTIFICATION message to peer 10.6.219.175. (ErrorCode=2, SubErrorCode=2, BgpAddressFamily=BNP-ATM:001, ErrorData=41040000fc00)
Dec  4 20:11:15 UTO-NPE-NE40E-01 %%01BGP/6/SEND_NOTIFY(l):The router sent a NOTIFICATION message to peer 10.3.219.191. (ErrorCode=2, SubErrorCode=2, BgpAddressFamily=BNP-ATM:001, ErrorData=41040000fc00)
Dec  4 20:11:15 UTO-NPE-NE40E-01 LSPM/4/MPLSXCDOWN:OID 1.3.6.1.2.1.10.166.2.0.2 LSP went Down. (BeginLspIndex=56447.56447.-1, EndLspIndex=56447.56447.-1)
Dec  4 20:11:16 UTO-NPE-NE40E-01 %%01BGP/3/STATE_CHG_UPDOWN(l):The status of the peer 10.7.219.122 changed from OPENCONFIRM to ESTABLISHED. (InstanceName=Ma3mal-Borg:001, StateChangeReason=Up) 
Dec  4 20:11:16 UTO-NPE-NE40E-01 BGP/4/ESTABLISHED:OID 1.3.6.1.2.1.15.7.1 The BGP FSM enters the Established stat

[root@Syslog_Trial ~]# tcpdump -i ens160 -n -vvv | grep 514
tcpdump: listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
    10.10.10.1.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 195
    10.10.10.1.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 203
    10.10.10.2.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 154
        0x0020:  452d 4e45 3430 452d 3031 204e 5141 2f34
        0x0050:  352e 3235 2e31 3131 2e36 2e31 204e 5141
    10.10.10.2.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 152
        0x0020:  452d 4e45 3430 452d 3031 204e 5141 2f34
    10.10.10.1.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 207
    10.10.10.1.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 218
    10.10.10.1.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 362
    10.10.10.1.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 208
    10.10.10.1.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 195
    10.10.10.1.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 203
    10.10.10.2.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 165
    10.10.10.2.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 150
    10.10.10.1.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 195
    10.10.10.1.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 195
    10.10.10.1.38514 > 10.39.224.10.syslog: [udp sum ok] SYSLOG, length: 163

[root@Syslog_Trial ~]# tail -f -vv /var/log/boot.log
==> /var/log/boot.log <==
Dec  4 20:24:27 UTO-NPE-NE40E-01 LSPM/4/MPLSXCDOWN:OID 1.3.6.1.2.1.10.166.2.0.2 LSP went Down. (BeginLspIndex=54277.54277.-1, EndLspIndex=54277.54277.-1)
Dec  4 20:24:28 UTO-NPE-NE40E-01 %%01BGP/6/RECV_NOTIFY(l):The router received NOTIFICATION message from peer 10.40.115.22. (ErrorCode=4, SubErrorCode=0, BgpAddressFamily=ARAMEX:001, ErrorData=NULL)
Dec  4 20:24:28 UTO-NPE-NE40E-01 %%01BGP/3/STATE_CHG_UPDOWN(l):The status of the peer 10.40.115.22 changed from ESTABLISHED to IDLE. (InstanceName=ARAMEX:001, StateChangeReason=Notification Message Received) 
Dec  4 20:24:28 UTO-NPE-NE40E-01 BGP/2/BACKWARD:OID 1.3.6.1.2.1.15.7.2 The BGP FSM moves from a higher numbered state to a lower numbered state. (BgpPeerRemoteAddr=10.40.115.22, InstanceId=66, Afi=1, Safi=1, PeerType=1, PeerRemoteAddr=10.40.115.22, InterfaceIndex=0, BgpPeerLastError=67, BgpPeerState=1, BgpPeerUnavaiReason=2, InterfaceName=null)
Dec  4 20:24:28 UTO-NPE-NE40E-01 LSPM/4/MPLSXCDOWN:OID 1.3.6.1.2.1.10.166.2.0.2 LSP went Down. (BeginLspIndex=51440.51440.-1, EndLspIndex=51440.51440.-1)
Dec  4 20:24:29 UTO-NPE-NE40E-01 %%01BGP/6/RECV_NOTIFY(l):The router received NOTIFICATION message from peer 10.11.219.146. (ErrorCode=4, SubErrorCode=0, BgpAddressFamily=Ma3mal-Borg:001, ErrorData=NULL)
Dec  4 20:24:29 UTO-NPE-NE40E-01 %%01BGP/3/STATE_CHG_UPDOWN(l):The status of the peer 10.11.219.146 changed from ESTABLISHED to IDLE. (InstanceName=Ma3mal-Borg:001, StateChangeReason=Notification Message Received) 
Dec  4 20:24:29 UTO-NPE-NE40E-01 BGP/2/BACKWARD:OID 1.3.6.1.2.1.15.7.2 The BGP FSM moves from a higher numbered state to a lower numbered state. (BgpPeerRemoteAddr=10.11.219.146, InstanceId=31, Afi=1, Safi=1, PeerType=1, PeerRemoteAddr=10.11.219.146, InterfaceIndex=0, BgpPeerLastError=67, BgpPeerState=1, BgpPeerUnavaiReason=2, InterfaceName=null)
Dec  4 20:24:29 UTO-NPE-NE40E-01 LSPM/4/MPLSXCDOWN:OID 1.3.6.1.2.1.10.166.2.0.2 LSP went Down. (BeginLspIndex=56391.56391.-1, EndLspIndex=56391.56391.-1)
Dec  4 22:24:29 10.10.12.5 12/04/2017 20:23:41 security: vty access denied (Denied due to access-list check for all available vtys) src address 88.248.84.182
Dec  4 20:24:31 RMS-NPE-NE40E-01 NQA/4/PROBEFAIL:OID 1.3.6.1.4.1.2011.5.25.111.6.1 NQA entry probe failed. (OwnerIndex=admin, TestName=vctest_2)
Dec  4 20:24:31 RMS-NPE-NE40E-01 NQA/4/TESTFAIL:OID 1.3.6.1.4.1.2011.5.25.111.6.2 NQA entry test failed. (OwnerIndex=admin, TestName=vctest_2)

jochen · December 5, 2017, 8:53am

The input is running on port 1514/udp but the packet dumps clearly show that clients send their messages to port 514/udp.

mbahaa · December 5, 2017, 10:55am

Shall I change input to run on port 514?

jochen · December 5, 2017, 11:02am

Do whatever fits your requirements but make sure that the clients are sending messages to the actual input.

mbahaa · December 5, 2017, 2:59pm

Clients are sending towards port number 514

mbahaa · December 5, 2017, 7:46pm

the below commands were re-applied again & input seems start receiving messages afterwards But i noticed that there is error in the search tab & i can see the logs as before [i will open a new case for this point]

iptables -t nat -A PREROUTING -i ens160 -p udp --dport 514 -j REDIRECT --to-port 1514

semanage port -a -t syslogd_port_t -p tcp 1514

system · December 19, 2017, 7:46pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Input is always failed! Graylog Central (peer support)	6	23435	December 4, 2017
Not receiving TCP messages Graylog Central (peer support)	4	737	May 27, 2022
Collecting Syslog Graylog Central (peer support)	7	1115	January 29, 2020
Don't receive syslog on graylog Graylog Central (peer support)	12	7813	August 9, 2018
Log are not display in my input Graylog Central (peer support) basic-configuration	14	1861	May 18, 2022

Input Messages are not received any more!

Related Topics