GELF HTTP does not process messages

Hello!
I sending events from minio to endpoint graylog input, tcp gelf http. Count of connections change, but no processes message at all. I catch some tcpdump from it.
its first http packet, 10.10.10.10 its ip graylog node

Summary

Frame 4: 1262 bytes on wire (10096 bits), 1262 bytes captured (10096 bits)
Ethernet II, Src: HuaweiTe_18:bf:91 (fa:fa:fa:fa:fa:fa:fa), Dst: Vmware_a4:77:e4 (fb:fb:fb:fb:fb:fb)
Internet Protocol Version 4, Src: 172.20.6.103, Dst: 10.10.10.10
0100 … = Version: 4
… 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 1248
Identification: 0xb7c2 (47042)
Flags: 0x4000, Don’t fragment
Time to live: 62
Protocol: TCP (6)
Header checksum: 0x87b2 [validation disabled]
[Header checksum status: Unverified]
Source: 172.20.6.103
Destination: 10.10.10.10
Transmission Control Protocol, Src Port: 44898, Dst Port: 11010, Seq: 1, Ack: 1, Len: 1208
Source Port: 44898
Destination Port: 11010
[Stream index: 0]
[TCP Segment Len: 1208]
Sequence number: 1 (relative sequence number)
[Next sequence number: 1209 (relative sequence number)]
Acknowledgment number: 1 (relative ack number)
0101 … = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window size value: 229
[Calculated window size: 29312]
[Window size scaling factor: 128]
Checksum: 0x7db2 [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
[SEQ/ACK analysis]
[iRTT: 0.003999000 seconds]
[Bytes in flight: 1208]
[Bytes sent since last PSH flag: 1208]
[Timestamps]
TCP payload (1208 bytes)
Hypertext Transfer Protocol
POST /gelf HTTP/1.1\r\n
[Expert Info (Chat/Sequence): POST /gelf HTTP/1.1\r\n]
[POST /gelf HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: POST
Request URI: /gelf
Request Version: HTTP/1.1
Host: 10.10.10.10:11010\r\n
User-Agent: Go-http-client/1.1\r\n
Content-Length: 1051\r\n
[Content length: 1051]
Content-Type: application/json\r\n
Accept-Encoding: gzip\r\n
\r\n
[Full request URI: http://10.10.10.10:11010/gelf]
[HTTP request 1/1]
[Response in frame: 6]
File Data: 1051 bytes
JavaScript Object Notation: application/json
JSON compact form: {…}
Object
Member Key: EventName
Member Key: Key
String value: bucket/5845e608fb0b0755fa99d7e7.png
Key: Key
Member Key: Records

then node answer 202

Summary

Frame 6: 122 bytes on wire (976 bits), 122 bytes captured (976 bits)
Ethernet II, Src: Vmware_a4:77:e4 (fb:fb:fb:fb:fb:fb), Dst: HuaweiTe_18:bf:91 (fa:fa:fa:fa:fa:fa:fa)
Internet Protocol Version 4, Src: 10.10.10.10, Dst: 172.20.6.103
0100 … = Version: 4
… 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 108
Identification: 0xc497 (50327)
Flags: 0x4000, Don’t fragment
Time to live: 64
Protocol: TCP (6)
Header checksum: 0x7d51 [validation disabled]
[Header checksum status: Unverified]
Source: 10.10.10.10
Destination: 172.20.6.103
Transmission Control Protocol, Src Port: 11010, Dst Port: 44898, Seq: 1, Ack: 1209, Len: 68
Source Port: 11010
Destination Port: 44898
[Stream index: 0]
[TCP Segment Len: 68]
Sequence number: 1 (relative sequence number)
[Next sequence number: 69 (relative sequence number)]
Acknowledgment number: 1209 (relative ack number)
0101 … = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window size value: 8030
[Calculated window size: 32120]
[Window size scaling factor: 4]
Checksum: 0xf901 [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
[SEQ/ACK analysis]
[iRTT: 0.003999000 seconds]
[Bytes in flight: 68]
[Bytes sent since last PSH flag: 68]
[Timestamps]
TCP payload (68 bytes)
Hypertext Transfer Protocol
HTTP/1.1 202 Accepted\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 202 Accepted\r\n]
[HTTP/1.1 202 Accepted\r\n]
[Severity level: Chat]
[Group: Sequence]
Response Version: HTTP/1.1
Status Code: 202
[Status Code Description: Accepted]
Response Phrase: Accepted
Content-Length: 0\r\n
[Content length: 0]
Connection: keep-alive\r\n
\r\n
[HTTP response 1/1]
[Time since request: 0.001567000 seconds]
[Request in frame: 4]

I see that i have json there.

When you browse the System > Inputs section and go to the GELF input, there should be a button on the right saying something like “Show received messages”… What does that show you?

  • Is this the only input running on the host?
  • If there are other inputs, are their messages showing up?

Ony node only, input started and message in this input is 0.

Then it’s not a matter of the input not processing messages. Then it’s a matter of messages not arriving at the input. Now you know where to troubleshoot :slight_smile:

have no idea. the message is comes to node and graylog answer with 202, that mean that message proceed to processing ( http://docs.graylog.org/en/2.4/pages/sending_data.html#gelf-via-http ). And there is 9 my attempt
https://prnt.sc/loicff

1 Like

Hey that’s great troubleshooting so far! Nice!

So… that leaves the question where those messages have gone :smiley:

EDIT:
Your screenshot, which I’ve included here for easy viewing, indicates that a few kB of data was indeed received.

So what happens when you bang on that “Show received messages” button?

only one test message, generated manually from same host, i think thats it

Hmmm, that’s pretty odd. No way does 9.4kB add up to one message…

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.