GELF HTTP does not process messages


(Roman Havrilyuk) #1

Hello!
I sending events from minio to endpoint graylog input, tcp gelf http. Count of connections change, but no processes message at all. I catch some tcpdump from it.
its first http packet, 10.10.10.10 its ip graylog node

Summary

Frame 4: 1262 bytes on wire (10096 bits), 1262 bytes captured (10096 bits)
Ethernet II, Src: HuaweiTe_18:bf:91 (fa:fa:fa:fa:fa:fa:fa), Dst: Vmware_a4:77:e4 (fb:fb:fb:fb:fb:fb)
Internet Protocol Version 4, Src: 172.20.6.103, Dst: 10.10.10.10
0100 … = Version: 4
… 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 1248
Identification: 0xb7c2 (47042)
Flags: 0x4000, Don’t fragment
Time to live: 62
Protocol: TCP (6)
Header checksum: 0x87b2 [validation disabled]
[Header checksum status: Unverified]
Source: 172.20.6.103
Destination: 10.10.10.10
Transmission Control Protocol, Src Port: 44898, Dst Port: 11010, Seq: 1, Ack: 1, Len: 1208
Source Port: 44898
Destination Port: 11010
[Stream index: 0]
[TCP Segment Len: 1208]
Sequence number: 1 (relative sequence number)
[Next sequence number: 1209 (relative sequence number)]
Acknowledgment number: 1 (relative ack number)
0101 … = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window size value: 229
[Calculated window size: 29312]
[Window size scaling factor: 128]
Checksum: 0x7db2 [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
[SEQ/ACK analysis]
[iRTT: 0.003999000 seconds]
[Bytes in flight: 1208]
[Bytes sent since last PSH flag: 1208]
[Timestamps]
TCP payload (1208 bytes)
Hypertext Transfer Protocol
POST /gelf HTTP/1.1\r\n
[Expert Info (Chat/Sequence): POST /gelf HTTP/1.1\r\n]
[POST /gelf HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: POST
Request URI: /gelf
Request Version: HTTP/1.1
Host: 10.10.10.10:11010\r\n
User-Agent: Go-http-client/1.1\r\n
Content-Length: 1051\r\n
[Content length: 1051]
Content-Type: application/json\r\n
Accept-Encoding: gzip\r\n
\r\n
[Full request URI: http://10.10.10.10:11010/gelf]
[HTTP request 1/1]
[Response in frame: 6]
File Data: 1051 bytes
JavaScript Object Notation: application/json
JSON compact form: {…}
Object
Member Key: EventName
Member Key: Key
String value: bucket/5845e608fb0b0755fa99d7e7.png
Key: Key
Member Key: Records

then node answer 202

Summary

Frame 6: 122 bytes on wire (976 bits), 122 bytes captured (976 bits)
Ethernet II, Src: Vmware_a4:77:e4 (fb:fb:fb:fb:fb:fb), Dst: HuaweiTe_18:bf:91 (fa:fa:fa:fa:fa:fa:fa)
Internet Protocol Version 4, Src: 10.10.10.10, Dst: 172.20.6.103
0100 … = Version: 4
… 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 108
Identification: 0xc497 (50327)
Flags: 0x4000, Don’t fragment
Time to live: 64
Protocol: TCP (6)
Header checksum: 0x7d51 [validation disabled]
[Header checksum status: Unverified]
Source: 10.10.10.10
Destination: 172.20.6.103
Transmission Control Protocol, Src Port: 11010, Dst Port: 44898, Seq: 1, Ack: 1209, Len: 68
Source Port: 11010
Destination Port: 44898
[Stream index: 0]
[TCP Segment Len: 68]
Sequence number: 1 (relative sequence number)
[Next sequence number: 69 (relative sequence number)]
Acknowledgment number: 1209 (relative ack number)
0101 … = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window size value: 8030
[Calculated window size: 32120]
[Window size scaling factor: 4]
Checksum: 0xf901 [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
[SEQ/ACK analysis]
[iRTT: 0.003999000 seconds]
[Bytes in flight: 68]
[Bytes sent since last PSH flag: 68]
[Timestamps]
TCP payload (68 bytes)
Hypertext Transfer Protocol
HTTP/1.1 202 Accepted\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 202 Accepted\r\n]
[HTTP/1.1 202 Accepted\r\n]
[Severity level: Chat]
[Group: Sequence]
Response Version: HTTP/1.1
Status Code: 202
[Status Code Description: Accepted]
Response Phrase: Accepted
Content-Length: 0\r\n
[Content length: 0]
Connection: keep-alive\r\n
\r\n
[HTTP response 1/1]
[Time since request: 0.001567000 seconds]
[Request in frame: 4]

I see that i have json there.


(Tess) #2

When you browse the System > Inputs section and go to the GELF input, there should be a button on the right saying something like “Show received messages”… What does that show you?

  • Is this the only input running on the host?
  • If there are other inputs, are their messages showing up?

(Roman Havrilyuk) #3

Ony node only, input started and message in this input is 0.


(Tess) #4

Then it’s not a matter of the input not processing messages. Then it’s a matter of messages not arriving at the input. Now you know where to troubleshoot :slight_smile:


(Roman Havrilyuk) #5

have no idea. the message is comes to node and graylog answer with 202, that mean that message proceed to processing ( http://docs.graylog.org/en/2.4/pages/sending_data.html#gelf-via-http ). And there is 9 my attempt
https://prnt.sc/loicff


(Tess) #6

Hey that’s great troubleshooting so far! Nice!

So… that leaves the question where those messages have gone :smiley:

EDIT:
Your screenshot, which I’ve included here for easy viewing, indicates that a few kB of data was indeed received.

So what happens when you bang on that “Show received messages” button?


(Roman Havrilyuk) #7

only one test message, generated manually from same host, i think thats it


(Tess) #8

Hmmm, that’s pretty odd. No way does 9.4kB add up to one message…


(system) #9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.