java.lang.IllegalStateException: GELF message is too short

I built a simple TCP Server in Delphi, and made a test there, from the same Oracle pl/sql code.
Logs were collected like:

Status:Connected
{ "version": "1.1", "host": "dataplus05", "short_message": "A short message", "level": 5, "_some_info": "foo" }"\0"
{ "version": "1.1", "host": "dataplus05", "short_message": "A short message", "level": 5, "_some_info": "foo" }"\0"
{ "version": "1.1", "host": "dataplus05", "short_message": "A short message", "level": 5, "_some_info": "foo" }"\0"
Status:Dissconnected

please see attached image too. the first and last lines are my buttonstcpserver events.

and also my simple Oracle PL/SQL code is as below:

create or replace procedure AAA_GRAYLOG_TCP_05
is

conn                utl_tcp.connection;
v_GL_record         clob;
ret_val             pls_integer;

v_lms_host          varchar2(4000) := '192.168.1.94';
v_lms_port          number := 41000;

begin

v_GL_record := 

'{ "version": "1.1", "host": "dataplus05", "short_message": "A short message", "level": 5, "_some_info": "foo" }' 
 || '"\0"'
;

-- Output 
dbms_output.put_line(v_GL_record);

conn := utl_tcp.open_connection(
v_lms_host, 
v_lms_port
);
ret_val := UTL_TCP.WRITE_LINE(conn, v_GL_record);
utl_tcp.close_connection(conn);

end;

am I doing anything wrong?

thank you very much for you efforts,
altin

1 Like

Well, it does say that “each message needs to be delimited with a null byte”. So even if you’re just sending one :slight_smile:

You’re doing some great work here! And it sounds like you’re getting very close to the answer.

I would actually suggest running a Netcat listener to do the other test I suggested. Send it both the test message through Netcat and the test message from your Oracle app.

Ok for that.

But from Oracle I am already putting:

\"0"

and still I have the "“GELF message is too short…” error and the 45 records.
I gave the same even when double quotes are removed, the way of the documentation.
why should the same string from Oracle generate error and debug?

thank you very much @Totally_Not_A_Robot
Altin

I’m not sure, hence my suggestion to send both the manual and the Oracle message to a Netcat listener, so you can compare them to eachother.

Just for shits and giggles, does Oracle treat \"0" the same as it would, say a bare \0 or even"\0" - good chance it may be that it figures you want to escape the single quote and not add a null byte.

1 Like

Thank you very much to both @benvanstaveren and @Totally_Not_A_Robot for trying to help.

But I have seen in thew Oracle docs for UTL_TCP, the WRITE_LINE Function,
https://docs.oracle.com/cd/E11882_01/appdev.112/e40758/u_tcp.htm#ARPLS71576

quoted:
“The newline character sequence is appended to the message before it is transmitted.”

which means the newline is added as a last (!) action.

While in your doc, on GELF TCP, paragraph “Example payload”, it says:

Note

Newlines must be denoted with the `\n` escape sequence to ensure the payload is valid JSON as per [RFC 7159]

which tells me it is looking for some escape of newlines in the original text, which cannot be done since Oracle’s package by default adds it as a last action before delivery.

Best regards,
Altin

1 Like

I also changed my UTL_TCP connection’s default newline CRLF when opening, setting it to NULL.

conn := utl_tcp.open_connection(
remote_host => v_lms_host, 
remote_port => v_lms_port, 
newline => NULL
);

I have the same problem again.

best regards,
Altin

I think the newlines it’s talking about for the payload is if you are embedding newlines in your JSON message, and not the separator between messages. You want to set “newline” to the null byte (\0 ) and see if that makes the difference.

As far as I can tell from the docs, a GELF message is in JSON, any newlines inside the JSON payload must be escaped properly, but the separator between GELF messages is a null byte.

1 Like

I have tried with both:

 (\"0") and (\0).

The result is the same.

but I did opened the Oracle’s TCP connection with the extra parameter newline, as below:

conn := utl_tcp.open_connection(
remote_host => v_lms_host, 
remote_port => v_lms_port,
newline => Chr(0)

and it did work, I don’t have the cosmetic :slight_smile: messages again.
thank you very much for suggesting Chr(0).
I had tried that before, but not inside UTL_TCP, but as an added string in the end.
A guy at stackoverflow recalled me that.

Important:
what troubles me most is the inability to identify the problem by Graylog own logs. I cannot find anything on the 45 messages to indicate me what is the error and how to be corrected.

Is there anything more to debug on Graylog? have more messages in hope of identifying the problem?

best regards
Altin

1 Like

The thing is that you are trying to send GELF formatted output, using a raw TCP library - while it’s possible, nobody ever said it’d be easy :wink:

Set up a raw TCP input, have Oracle send the JSON data that way, at that point it won’t care what the format is, as long as each message is separated by a newline, after that you can attach processor/pipeline to parse the JSOn and sort it out from there - at this point this seems to be the better option :slight_smile:

1 Like

Yup, from that point out you could run a JSON extractor against it…

I have the data uploaded correctly without any JSON Extractor.

As long as I am sending data in the JSON format as in the Payload Example,
is the JSON Extractor necessary ?

best regards
altin

Only if you want things to be split into separate fields automagically.

I didn’t make any extractor. My GELF TCP Input has:

Configured extractors
This input has no configured extractors.

I just send the JSON from Oracle PL/SQL and it gets written to Graylog.

This has attracted my attention, because in Splunk I had to set some KV_MODE=XML when
sending data by such (XML). While at Graylog I have none anywhere for the JSON.

May be the JSON extractor is default - and works even unconfined?

best regards,
Altin

What you see is the raw json payload in the “message” field - if that works for you, then leave it as is. If you want to parse the JSON and convert the keys in the payload to actual fields in an event, you’ll have to run an extractor over it :slight_smile:

1 Like

Thank you very much @benvanstaveren

but it would be best if you told me why I can get my JSON loaded without “declaring” it as a type somewhere. Is it a Graylog default? A GELF TCP Input default? Or a Graylog’s default index default?

thanks and best regards,
Altin

You have the JSON “loaded” because it’s treated as a raw bit of text in the ‘message’ field of an event - but that’s all it is, right now. Graylog has not assigned meaning to any of it, as far as Graylog is concerned, it’s a text message (that just happens to be formatted in JSON). If you want to actually parse it as JSON and do useful things with the individual fields, that’s where the extractor comes in.

Also, the raw TCP input takes anything it receives and creates an event consisting of a timestamp (when the input was received), and a message field which is whatever content was sent.

But altink told us altink did not use raw tcp input, but GELF input. And GELF input does not necessarily need extractors, if the data sent to it is formatted properly.

1 Like

GELF input still doesn’t parse the JSON - what I said about the raw tcp one also goes for GELF, if the json data is part of the “message” field then it gets stored, but not parsed.

1 Like

Yes.

I am using GELF TCP input. With data formatted as JSON. The later I have not declared anywhere.
Indeed, the GELF TCP Input was the only thing I needed to do inside Graylog to get the data in.

And I also do have all my custom _foo1, _foo2… _fooX fields.

best regards
Altin

1 Like