GELF message is too short. Not even the type header would fit

Graylog Central (peer support)
1

Trying to configuring a GELF TCP input, and for every message received, the following ERROR message appears in server.log:

2020-02-01T09:56:08.671Z ERROR [AbstractTcpTransport] Error in Input [GELF TCP/5e3466e59046521f06170f45] (channel [id: 0x55bb5741, L:/127.0.0.1:12201 ! R:/127.0.0.1:36195]) (cause java.lang.IllegalStateException: GELF message is too short. Not even the type header would fit.)

Logging host sends syslog to the graylog host on UDP/514, where syslog-ng captures it then outputs to 127.0.0.1:12201 to the graylog server GELF TCP Input. I’m using syslog-ng v3.25.1’s built-in “graylog2” driver, using the following config:

source s_514udp {
network(
ip(0.0.0.0)
port(514)
transport(“udp”)
);
};

destination d_graylog {
graylog2(
host(“127.0.0.1”)
transport(tcp)
);
};

log {
source(s_514udp);
destination(d_graylog);
};

Below is the tcpdump for a given log, from when it arrives on the box via UDP/514 to when it arrives to the GELF TCP input:

Incoming syslog (UDP/514):

20:21:16.637811 IP 10.0.0.1.39608 > 10.0.0.5.514: SYSLOG daemon.info, length: 391
E…@.@.B.
…
…z.<30>2020:02:01-15:21:16 willstechblog ulogd[5320]: id=“2002” severity=“info” sys=“SecureNet” sub=“packetfilter” name=“Packet accepted” action=“accept” fwrule=“22” initf=“eth0” outitf=“eth1” srcmac=“04:d4:c4:38:26:90” dstmac=“00:0c:29:4b:67:fd” srcip=“192.168.1.50” dstip=“72.52.80.23” proto=“6” length=“153” tos=“0x00” prec=“0x00” ttl=“126” srcport=“51100” dstport=“443” tcpflags=“ACK PSH”

GELF TCP (localhost TCP/12201):

20:21:30.103858 IP 127.0.0.1.34710 > 127.0.0.1.12201: Flags [S], seq 1341813785, win 43690, options [mss 65495,sackOK,TS val 142863155 ecr 0,nop,wscale 7], length 0
E…<.*@.@…/.O.t…0…
…3…

As you can see, when the log reaches the GELF Input, it has a length of 0. What am I doing wrong?

Thanks in advance,
William

2

according to the syslog-ng docs you did it right:

https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.25/administration-guide/35#TOPIC-1349421

but the sender did not send a message at all - so you might need to ask the syslog ng community if they have actually knowledge what is wrong. I personla do not know what the problem might be here.

3

Well I showed proof that the sender in indeed sending logs (see the output below "Incoming syslog (UDP/514) section above). However, I have raised a bug report on syslog-ng’s Github. We’ll see what they say about it:

If you can’t think of any other reason why it doesn’t work, it sounds like a syslog-ng issue.

4

Jan, I can confirm with syslog-ng debug output that the outgoing message from syslog-ng is correct:

[2020-02-04T13:59:43.411386] Outgoing message; message=‘{“version”:“1.1”,“timestamp”:1580824783,“short_message”:“02:04-08:59:43 willstechblog ulogd[5379]: id="2002" severity="info" sys="SecureNet" sub="packetfilter" name="Packet accepted" action="accept" fwrule="22" initf="eth0" outitf="eth1" srcmac="04:d4:c4:38:26:90" dstmac="00:0c:29:4b:67:fd" srcip="192.168.1.50" dstip="52.179.224.121" proto="6" length="41" tos="0x00" prec="0x00" ttl="126" srcport="61755" dstport="443" tcpflags="ACK" “,“level”:6,“host”:“10.0.0.1”,”_program”:“2020”,“_pid”:-1,“_facility”:“daemon”,“_class”:“”}’

However, Graylog server.log still shows the message coming in incorrectly:

2020-02-04T14:00:47.688Z ERROR [AbstractTcpTransport] Error in Input [GELF TCP/5e3466e59046521f06170f45] (channel [id: 0x6fe83a51, L:/127.0.0.1:12201 ! R:/127.0.0.1:43189]) (cause java.lang.IllegalStateException: GELF message is too short. Not even the type header would fit.)

I know I could probably just use a Raw TCP Input instead, but this solution should work so I want to make it work.

5

he @wtrelawny

if you use a RAW input, does it work? That is not clear from your posting …

6

Jan, I meant that I have a GELF TCP Input and I’d like to use that because it should work. But Raw TCP doesn’t work either- it appears to not know where the end of a message is:

image
image958×502 55.3 KB

7

And to further clarify, the GELF Input does seem to be working better now that I implemented a patch from the syslog-ng developers, but the Syslog id field is not making it through to GL, which is parsing the year (“2020”) as the id instead:

image
image892×397 16.8 KB

I’m still pursuing this with the bug report I filed for syslog-ng.

8

he @wtrelawny

do you have multiple Graylog servers or just a single one? It looks like syslog-ng has some problems to chunk the messages correct.

for the not correct parsing - can you show how the message looks before ingest? GELF is a json message with defined fields - so that json message might be a little different than expected if such happens.

9

Jan, I just have one node. 10.0.0.1 sends logs to 10.0.0.5:514/udp, syslog-ng captures it and forwards to 127.0.0.1:12201 where GELF Input is listening.

This is a sample log from the syslog-ng debug output, so it what syslog-ng sends and is before the GELF Input ingest:

Syslog-ng devs told me to explicitly set the _pid field as -1 with a rewrite rule because apparently it wasn’t getting set properly. But as you can see, that’s causing other issues too.

10

he @wtrelawny

I might be unclear - to find the true problem I would look what is the output of syslog-ng to Graylog - means what has left syslog-ng exactly.
What is ingested or outputted on other outputs might be different.

11

Jan, yes that is the output from syslog-ng. Here is a tcpdump of the same output:

16:51:06.786126 IP 127.0.0.1.45797 > 127.0.0.1.12201: Flags [P.], seq 572:1139, ack 1, win 342, options [nop,nop,TS val 48510860 ecr 48510860], length 567
E…kW$@.@…f…/…f.N…V.`…
…7…7.{“version”:“1.1”,“timestamp”:1580921466,“short_message”:“02:05-16:51:06 willstechblog ulogd[6045]: id="2002" severity="info" sys="SecureNet" sub="packetfilter" name="Packet accepted" action="accept" fwrule="9" initf="eth0" outitf="eth0" srcmac="00:0c:29:e5:1c:d1" dstmac="00:0c:29:4b:67:fd" srcip="10.0.0.5" dstip="192.168.1.100" proto="6" length="52" tos="0x00" prec="0x00" ttl="63" srcport="443" dstport="53181" tcpflags="ACK SYN" “,“level”:6,“host”:“10.0.0.1”,”_program”:“2020”,“_pid”:-1,“_facility”:“daemon”,“_class”:“”}.

And in Graylog, the data looks like it does in the above screenshot. It appears to grab the year “2020” and put it in a “program” field, and the “pid” field that syslog-ng devs told me to add is coming through in search when it is supposed to be hidden.

12

But in your GELF Message you have:

_program”:“2020”,

as the _ a forbidden in the field names that is removed and the field program is created with the value 2020.

So nothing wrong in Graylog. It is in your data.

13

Yeah I agree- it appears to be a syslog-ng issue with their graylog2() driver.

Thanks for your help Jan.

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.