GELF HTTP curl example for chunk off messages

Graylog Central (peer support)
1

Hi,

I am trying to debug, why GELF HTTP shows only first message from chunk of messages send by fluentbit.

If example for one message in curl looks like this
curl -X POST -H ‘Content-Type: application/json’ -d ‘{ “version”: “1.1”, “host”: “example.org”, “short_message”: “A short message”, “level”: 5, “_some_info”: “foo” }’ ‘http://graylog.example.com:12201/gelf’

How should look curl for multiple messages?
Can somebody help me?

2

Try to debug fluentbit data by tcpdump on graylog server.
tcpdump -i INTERFACE -A ‘tcp port 12201’

Or save to pcap file and analyze using wireshark:
tcpdump -i INTERFACE -w output.pcap ‘tcp port 12201’

3

After setting gelf.conf to

[OUTPUT]
    Name  http
    Match *
    Host  mydomain.ltd
    Port  12233
    URI   /gelf
    Format gelf
    Gelf_Short_Message_Key  log
    Gelf_Timestamp_Key  timestamp

and TCP dump:

Host: mydomain.ltd:12233
Content-Length: 3581
Content-Type: application/json
User-Agent: Fluent-Bit


14:19:17.577932 IP (tos 0x0, ttl 59, id 53804, offset 0, flags [DF], proto TCP (6), length 1500)
    mytestingIP.35470 > graylog.12233: Flags [.], cksum 0x70d6 (correct), seq 138:1586, ack 1, win 502, options [nop,nop,TS val 2944472793 ecr 1584844588], length 1448
E....,@.;.j......e.(../.H.!G...I....p......
....^v.,{"version":"1.1", "short_message":"2020-09-08T12:19:17+0000 INFO This is less important than debug log and is often used to provide context in the current task.", "_container_id":"626acce3777cda8d8888fbf003033a57925937bc2de5e8494c1149aea4e8aa7b", "_container_name":"/random_generator", "_source":"stdout", "_version":"1.1", "host":"host_name", "timestamp":1599567557.345713}
{"version":"1.1", "_container_id":"626acce3777cda8d8888fbf003033a57925937bc2de5e8494c1149aea4e8aa7b", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:19:17+0000 INFO This is less important than debug log and is often used to provide context in the current task.", "_version":"1.1", "host":"host_name", "timestamp":1599567557.352637}
{"version":"1.1", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:19:17+0000 ERROR An error is usually an exception that has been caught and not handled.", "_container_id":"626acce3777cda8d8888fbf003033a57925937bc2de5e8494c1149aea4e8aa7b", "_version":"1.1", "host":"host_name", "timestamp":1599567557.356370}
{"version":"1.1", "_container_id":"626acce3777cda8d8888fbf003033a57925937bc2de5e8494c1149aea4e8aa7b", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:19:17+0000 ERROR An error is usually an exception that has been caught and not handled.", "_version":"1.1", "host":"host_name", "timestamp":1599567557.3610
14:19:17.577936 IP (tos 0x0, ttl 59, id 53805, offset 0, flags [DF], proto TCP (6), length 2185)
    mytestingIP.35470 > graylog.12233: Flags [P.], cksum 0x0575 (incorrect -> 0xc3d0), seq 1586:3719, ack 1, win 502, options [nop,nop,TS val 2944472793 ecr 1584844588], length 2133
E....-@.;.hH.....e.(../.H.&....I.....u.....
....^v.,42}
{"version":"1.1", "_container_id":"626acce3777cda8d8888fbf003033a57925937bc2de5e8494c1149aea4e8aa7b", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:19:17+0000 INFO This is less important than debug log and is often used to provide context in the current task.", "_version":"1.1", "host":"host_name", "timestamp":1599567557.365717}
{"version":"1.1", "_container_id":"626acce3777cda8d8888fbf003033a57925937bc2de5e8494c1149aea4e8aa7b", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:19:17+0000 WARN A warning that should be ignored is usually at this level and should be actionable.", "_version":"1.1", "host":"host_name", "timestamp":1599567557.370437}
{"version":"1.1", "_container_id":"626acce3777cda8d8888fbf003033a57925937bc2de5e8494c1149aea4e8aa7b", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:19:17+0000 ERROR An error is usually an exception that has been caught and not handled.", "_version":"1.1", "host":"host_name", "timestamp":1599567557.374913}
{"version":"1.1", "_container_id":"626acce3777cda8d8888fbf003033a57925937bc2de5e8494c1149aea4e8aa7b", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:19:17+0000 DEBUG This is a debug log that shows a log that can be ignored.", "_version":"1.1", "host":"host_name", "timestamp":1599567557.379377}
{"version":"1.1", "short_message":"2020-09-08T12:19:17+0000 ERROR An error is usually an exception that has been caught and not handled.", "_container_id":"626acce3777cda8d8888fbf003033a57925937bc2de5e8494c1149aea4e8aa7b", "_container_name":"/random_generator", "_source":"stdout", "_version":"1.1", "host":"host_name", "timestamp":1599567557.384026}
{"version":"1.1", "_source":"stdout", "short_message":"2020-09-08T12:19:17+0000 ERROR An error is usually an exception that has been caught and not handled.", "_container_id":"626acce3777cda8d8888fbf003033a57925937bc2de5e8494c1149aea4e8aa7b", "_container_name":"/random_generator", "_version":"1.1", "host":"host_name", "timestamp":1599567557.388561}

14:19:17.578383 IP (tos 0x0, ttl 63, id 4568, offset 0, flags [DF], proto TCP (6), length 52)
    graylog.12233 > mytestingIP.35470: Flags [.], cksum 0xfd1f (incorrect -> 0xa445), ack 138, win 4163, options [nop,nop,TS val 1584844590 ecr 2944472793], length 0
E..4..@.?.,..e.(..../......IH.!G...C.......
^v......
14:19:17.578400 IP (tos 0x0, ttl 63, id 4569, offset 0, flags [DF], proto TCP (6), length 52)
    graylog.12233 > mytestingIP.35470: Flags [.], cksum 0xfd1f (incorrect -> 0x9de8), ack 1586, win 4344, options [nop,nop,TS val 1584844590 ecr 2944472793], length 0
E..4..@.?.,..e.(..../......IH.&............
^v......
14:19:17.578404 IP (tos 0x0, ttl 63, id 4570, offset 0, flags [DF], proto TCP (6), length 52)
    graylog.12233 > mytestingIP.35470: Flags [.], cksum 0xfd1f (incorrect -> 0x9488), ack 3719, win 4611, options [nop,nop,TS val 1584844590 ecr 2944472793], length 0
E..4..@.?.,..e.(..../......IH./D...........
^v......
14:19:17.581051 IP (tos 0x0, ttl 63, id 4571, offset 0, flags [DF], proto TCP (6), length 120)
    graylog.12233 > mytestingIP.35470: Flags [P.], cksum 0xfd63 (incorrect -> 0x4b84), seq 1:69, ack 3719, win 4611, options [nop,nop,TS val 1584844593 ecr 2944472793], length 68
E..x..@.?.,..e.(..../......IH./D.....c.....
^v.1....HTTP/1.1 202 Accepted
content-length: 0
connection: keep-alive


14:19:17.581354 IP (tos 0x0, ttl 59, id 53807, offset 0, flags [DF], proto TCP (6), length 52)
    mytestingIP.35470 > graylog.12233: Flags [.], cksum 0xa44b (correct), ack 69, win 502, options [nop,nop,TS val 2944472796 ecr 1584844593], length 0
E..4./@.;.p......e.(../.H./D.........K.....
....^v.1
14:19:47.574758 IP (tos 0x0, ttl 59, id 53808, offset 0, flags [DF], proto TCP (6), length 52)
    mytestingIP.35470 > graylog.12233: Flags [F.], cksum 0x2f22 (correct), seq 3719, ack 69, win 502, options [nop,nop,TS val 2944502788 ecr 1584844593], length 0
E..4.0@.;.p......e.(../.H./D......../".....
....^v.1
14:19:47.575309 IP (tos 0x0, ttl 63, id 4572, offset 0, flags [DF], proto TCP (6), length 52)
    graylog.12233 > mytestingIP.35470: Flags [F.], cksum 0xfd1f (incorrect -> 0xa9e9), seq 69, ack 3720, win 4611, options [nop,nop,TS val 1584874587 ecr 2944502788], length 0
E..4..@.?.,..e.(..../.......H./E...........
^wD[....
14:19:47.576279 IP (tos 0x0, ttl 59, id 53809, offset 0, flags [DF], proto TCP (6), length 52)
    mytestingIP.35470 > graylog.12233: Flags [.], cksum 0xb9f3 (correct), ack 70, win 502, options [nop,nop,TS val 2944502791 ecr 1584874587], length 0
E..4.1@.;.p......e.(../.H./E...............
....^wD[

and only one message was recorded by graylog:

2020-09-08 14_28_41-Graylog
2020-09-08 14_28_41-Graylog1317×450 24.7 KB

As you can see, Timestamp is taken from timestamp files in GELF message: “Timestamp
2020-09-08 14:35:04.050”

Whet messages are sent by GELF TCP and gelf.conf looks like this:

[OUTPUT]
    Name  gelf
    Match *
    Host  graylog.dogadamycie.pl
    Port  12233
    Mode  tcp
    Gelf_Short_Message_Key  log
    Gelf_Timestamp_Key  timestamp

And TCP dump:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:35:04.585323 IP (tos 0x0, ttl 59, id 47446, offset 0, flags [DF], proto TCP (6), length 60)
    mytestingIP.35572 > graylog.12233: Flags [S], cksum 0xf556 (correct), seq 2253709329, win 64240, options [mss 1460,sackOK,TS val 2945419787 ecr 0,nop,wscale 7], length 0
E..<.V@.;..l.....e.(../..T...........V.........
............
14:35:04.585445 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    graylog.12233 > mytestingIP.35572: Flags [S.], cksum 0xfd27 (incorrect -> 0x0c60), seq 2579409196, ack 2253709330, win 65535, options [mss 1460,sackOK,TS val 1585791593 ecr 2945419787,nop,wscale 4], length 0
E..<..@.?.>..e.(..../......,.T.......'.........
^.Bi........
14:35:04.586567 IP (tos 0x0, ttl 59, id 47447, offset 0, flags [DF], proto TCP (6), length 52)
    mytestingIP.35572 > graylog.12233: Flags [.], cksum 0x3931 (correct), ack 1, win 502, options [nop,nop,TS val 2945419789 ecr 1585791593], length 0
E..4.W@.;..s.....e.(../..T.....-....91.....
....^.Bi
14:35:04.587655 IP (tos 0x0, ttl 59, id 47448, offset 0, flags [DF], proto TCP (6), length 415)
    mytestingIP.35572 > graylog.12233: Flags [P.], cksum 0x2616 (correct), seq 1:364, ack 1, win 502, options [nop,nop,TS val 2945419790 ecr 1585791593], length 363
E....X@.;........e.(../..T.....-....&......
....^.Bi{"version":"1.1", "_container_id":"d05502fb85b413fd3e51eb6bd1107286a77c2dfdee27fb721f65a879fc5acd6e", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:35:04+0000 WARN A warning that should be ignored is usually at this level and should be actionable.", "_version":"1.1", "host":"host_name", "timestamp":1599568504.049533}.
14:35:04.587682 IP (tos 0x0, ttl 59, id 47449, offset 0, flags [DF], proto TCP (6), length 390)
    mytestingIP.35572 > graylog.12233: Flags [P.], cksum 0x9088 (correct), seq 364:702, ack 1, win 502, options [nop,nop,TS val 2945419791 ecr 1585791593], length 338
E....Y@.;........e.(../..T.}...-...........
....^.Bi{"version":"1.1", "_source":"stdout", "short_message":"2020-09-08T12:35:04+0000 DEBUG This is a debug log that shows a log that can be ignored.", "_container_id":"d05502fb85b413fd3e51eb6bd1107286a77c2dfdee27fb721f65a879fc5acd6e", "_container_name":"/random_generator", "_version":"1.1", "host":"host_name", "timestamp":1599568504.052848}.
14:35:04.587687 IP (tos 0x0, ttl 59, id 47450, offset 0, flags [DF], proto TCP (6), length 427)
    mytestingIP.35572 > graylog.12233: Flags [P.], cksum 0x3f91 (correct), seq 702:1077, ack 1, win 502, options [nop,nop,TS val 2945419791 ecr 1585791593], length 375
E....Z@.;........e.(../..T.....-....?......
....^.Bi{"version":"1.1", "_container_id":"d05502fb85b413fd3e51eb6bd1107286a77c2dfdee27fb721f65a879fc5acd6e", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:35:04+0000 INFO This is less important than debug log and is often used to provide context in the current task.", "_version":"1.1", "host":"host_name", "timestamp":1599568504.067998}.
14:35:04.587735 IP (tos 0x0, ttl 63, id 31920, offset 0, flags [DF], proto TCP (6), length 52)
    graylog.12233 > mytestingIP.35572: Flags [.], cksum 0xfd1f (incorrect -> 0x2975), ack 364, win 4163, options [nop,nop,TS val 1585791596 ecr 2945419790], length 0
E..4|.@.?....e.(..../......-.T.}...C.......
^.Bl....
14:35:04.587751 IP (tos 0x0, ttl 63, id 31921, offset 0, flags [DF], proto TCP (6), length 52)
    graylog.12233 > mytestingIP.35572: Flags [.], cksum 0xfd1f (incorrect -> 0x27df), ack 702, win 4230, options [nop,nop,TS val 1585791596 ecr 2945419791], length 0
E..4|.@.?....e.(..../......-.T.............
^.Bl....
14:35:04.587754 IP (tos 0x0, ttl 63, id 31922, offset 0, flags [DF], proto TCP (6), length 52)
    graylog.12233 > mytestingIP.35572: Flags [.], cksum 0xfd1f (incorrect -> 0x2625), ack 1077, win 4297, options [nop,nop,TS val 1585791596 ecr 2945419791], length 0
E..4|.@.?....e.(..../......-.T.F...........
^.Bl....
14:35:04.587767 IP (tos 0x0, ttl 59, id 47451, offset 0, flags [DF], proto TCP (6), length 427)
    mytestingIP.35572 > graylog.12233: Flags [P.], cksum 0x3f22 (correct), seq 1077:1452, ack 1, win 502, options [nop,nop,TS val 2945419791 ecr 1585791593], length 375
E....[@.;........e.(../..T.F...-....?".....
....^.Bi{"version":"1.1", "_container_id":"d05502fb85b413fd3e51eb6bd1107286a77c2dfdee27fb721f65a879fc5acd6e", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:35:04+0000 INFO This is less important than debug log and is often used to provide context in the current task.", "_version":"1.1", "host":"host_name", "timestamp":1599568504.075936}.
14:35:04.587787 IP (tos 0x0, ttl 63, id 31923, offset 0, flags [DF], proto TCP (6), length 52)
    graylog.12233 > mytestingIP.35572: Flags [.], cksum 0xfd1f (incorrect -> 0x246b), ack 1452, win 4364, options [nop,nop,TS val 1585791596 ecr 2945419791], length 0
E..4|.@.?....e.(..../......-.T.............
^.Bl....
14:35:04.587983 IP (tos 0x0, ttl 59, id 47453, offset 0, flags [DF], proto TCP (6), length 403)
    mytestingIP.35572 > graylog.12233: Flags [P.], cksum 0x7750 (correct), seq 1790:2141, ack 1, win 502, options [nop,nop,TS val 2945419791 ecr 1585791593], length 351
E....]@.;........e.(../..T.....-....wP.....
....^.Bi{"version":"1.1", "_container_id":"d05502fb85b413fd3e51eb6bd1107286a77c2dfdee27fb721f65a879fc5acd6e", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:35:04+0000 ERROR An error is usually an exception that has been caught and not handled.", "_version":"1.1", "host":"host_name", "timestamp":1599568504.091659}.
14:35:04.588000 IP (tos 0x0, ttl 59, id 47454, offset 0, flags [DF], proto TCP (6), length 415)
    mytestingIP.35572 > graylog.12233: Flags [P.], cksum 0x17b4 (correct), seq 2141:2504, ack 1, win 502, options [nop,nop,TS val 2945419791 ecr 1585791593], length 363
E....^@.;........e.(../..T.n...-...........
....^.Bi{"version":"1.1", "_container_id":"d05502fb85b413fd3e51eb6bd1107286a77c2dfdee27fb721f65a879fc5acd6e", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:35:04+0000 WARN A warning that should be ignored is usually at this level and should be actionable.", "_version":"1.1", "host":"host_name", "timestamp":1599568504.099485}.
14:35:04.588004 IP (tos 0x0, ttl 59, id 47455, offset 0, flags [DF], proto TCP (6), length 415)
    mytestingIP.35572 > graylog.12233: Flags [P.], cksum 0xc4a7 (correct), seq 2504:2867, ack 1, win 502, options [nop,nop,TS val 2945419791 ecr 1585791593], length 363
E...._@.;........e.(../..T.....-...........
....^.Bi{"version":"1.1", "_source":"stdout", "short_message":"2020-09-08T12:35:04+0000 WARN A warning that should be ignored is usually at this level and should be actionable.", "_container_id":"d05502fb85b413fd3e51eb6bd1107286a77c2dfdee27fb721f65a879fc5acd6e", "_container_name":"/random_generator", "_version":"1.1", "host":"host_name", "timestamp":1599568504.102739}.
14:35:04.588008 IP (tos 0x0, ttl 59, id 47456, offset 0, flags [DF], proto TCP (6), length 390)
    mytestingIP.35572 > graylog.12233: Flags [P.], cksum 0x49ff (correct), seq 2867:3205, ack 1, win 502, options [nop,nop,TS val 2945419791 ecr 1585791593], length 338
E....`@.;........e.(../..T.D...-....I......
....^.Bi{"version":"1.1", "short_message":"2020-09-08T12:35:04+0000 DEBUG This is a debug log that shows a log that can be ignored.", "_container_id":"d05502fb85b413fd3e51eb6bd1107286a77c2dfdee27fb721f65a879fc5acd6e", "_container_name":"/random_generator", "_source":"stdout", "_version":"1.1", "host":"host_name", "timestamp":1599568504.111788}.
14:35:04.588011 IP (tos 0x0, ttl 59, id 47457, offset 0, flags [DF], proto TCP (6), length 403)
    mytestingIP.35572 > graylog.12233: Flags [P.], cksum 0x7ecd (correct), seq 3205:3556, ack 1, win 502, options [nop,nop,TS val 2945419791 ecr 1585791593], length 351
E....a@.;..
.....e.(../..T.....-....~......
....^.Bi{"version":"1.1", "_container_name":"/random_generator", "_source":"stdout", "short_message":"2020-09-08T12:35:04+0000 ERROR An error is usually an exception that has been caught and not handled.", "_container_id":"d05502fb85b413fd3e51eb6bd1107286a77c2dfdee27fb721f65a879fc5acd6e", "_version":"1.1", "host":"host_name", "timestamp":1599568504.120613}.
14:35:04.588071 IP (tos 0x0, ttl 63, id 31928, offset 0, flags [DF], proto TCP (6), length 52)
    graylog.12233 > mytestingIP.35572: Flags [.], cksum 0xfd1f (incorrect -> 0x1c43), ack 3205, win 4699, options [nop,nop,TS val 1585791596 ecr 2945419791], length 0
E..4|.@.?....e.(..../......-.T.....[.......
^.Bl....
14:35:35.075345 IP (tos 0x0, ttl 59, id 47458, offset 0, flags [DF], proto TCP (6), length 52)
    mytestingIP.35572 > graylog.12233: Flags [F.], cksum 0xb431 (correct), seq 3556, ack 1, win 502, options [nop,nop,TS val 2945450277 ecr 1585791596], length 0
E..4.b@.;..h.....e.(../..T.....-.....1.....
...%^.Bl
14:35:35.075995 IP (tos 0x0, ttl 63, id 31930, offset 0, flags [DF], proto TCP (6), length 52)
    graylog.12233 > mytestingIP.35572: Flags [F.], cksum 0xfd1f (incorrect -> 0x2c70), seq 1, ack 3557, win 4766, options [nop,nop,TS val 1585822084 ecr 2945450277], length 0
E..4|.@.?....e.(..../......-.T.............
^......%
14:35:35.077232 IP (tos 0x0, ttl 59, id 47459, offset 0, flags [DF], proto TCP (6), length 52)
    mytestingIP.35572 > graylog.12233: Flags [.], cksum 0x3d15 (correct), ack 2, win 502, options [nop,nop,TS val 2945450280 ecr 1585822084], length 0
E..4.c@.;..g.....e.(../..T..........=......
...(^...

20 packets captured
26 packets received by filter
6 packets dropped by kernel

In GELF TCP i can see all ten messages generated by random-logger:

docker run --rm --name random_generator chentex/random-logger:latest 1 1 10

It looks that, all messages are sent to fluentbit, but GELF HTTP doesn’t decode more than one message per chunk

4

I’ve checked fluitbit docs, and I don’t see support for http output, only tcp, udp and tls. TLS is probably tcp with TLS encryption. So it’s not working for you, because it’s not supported. It’s lacking support on fluentbit side, not graylog.

5

It is:

https://docs.fluentbit.io/manual/pipeline/outputs/http

format:

Specify the data format to be used in the HTTP request body, by default it uses *msgpack* . Other supported formats are *json* , *json_stream* and *json_lines* and *gelf* .

So it should work :slight_smile: but it does not…

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.