java.lang.IllegalStateException: GELF message is too short

(Altin Karaulli) #1

Dear All,

I am uploading data via GELF TCP Input, exactly the “Example Payload” as in the:
GELF - Graylog 2.5.0 documentation

  "version": "1.1",
  "host": "",
  "short_message": "A short message that helps you identify what is going on",
  "full_message": "Backtrace here\n\nmore stuff",
  "timestamp": 1385053862.3072,
  "level": 1,
  "_user_id": 9001,
  "_some_info": "foo",
  "_some_env_var": "bar"

The event does load successfully, but what I found for source:graylog-server is:

    java.lang.IllegalStateException: GELF message is too short. Not even the type header would fit.

and also

    ERROR [NettyTransport] Error in Input [GELF TCP/5c119b1eb358cb056b417b89] (channel [id: 0xc6ed250a, / :> /])

and when searched (source:graylog-server ), there are 45 messages in general.

Why the ERROR and the Exception ?

best regards,

(Konrad Merz) #2

Hi Altin,

Can you be a bit more specific about your setup. Can you make screenshots of your GELF Input configuration and explain, how you set up the log sender?


(Tess) #3

So does that mean the message are being received correctly? And that the GELF input is working and that you’re not missing any data? That’s our first priority. After that we can focus on cosmetics :slight_smile:

(Altin Karaulli) #4

the log comes correctly.
But I have too much cosmectics (some 45 records) for each one uploaded, where the ERROR and java.lang.Illegal… are present too.

I will upload images and details asap

thank you very much,
best regards

(Tess) #5

Sorry, I had misunderstood your previous post. I thought you said you received 45 correct messages, but what you means is that for every 1 correct message, you get 45 errors as well? Yikes! That’s not pretty.

(Altin Karaulli) #6

No Sir.

What happens is that for every 1 record uploaded, I see some 45 generated as (it looks) internal graylog messages, where between them, I can find the highlighted:
ERROR and java.lang.IllegalStateException - related to my single upload

I cannot find a place here to upload them all.

thank you very much Totally_not_A_Robot

(Tess) #7

The errors can be copy-pasted over here in a codeblock, or you could post them on PasteBin and share the link here.

If it’s a screenshot, you can simply upload the image by using the “insert image” button.

(Altin Karaulli) #8

Thank you Tess,

you are right now. I have one good record and 45 cosmetics, two of them, One ERROR and one Java Illegal

below is my Omega CA GELF TCP Input


best regards

ps. I can find no “insert image” link in my profile,
the records
TCP Input Messages 001

(Tess) #9

I can’t make a screenshot right now, so I’ll describe it.

In the editor where you write your replies, there’s a menu bar:

balloon bold italic | insertlink insertquote insertcode insertimage | bullets numberedlist smiley options

There’s an insert image option :slight_smile: Seventh button from the left.

(Altin Karaulli) #10

thank you very much for the image link, found where you pointed.
but now that I uploaded the TCP Input config as text, I guess you don’t need it anymore?

best regards

(Altin Karaulli) #11

Hi Konrad,

sorry for missing the point and answering to Tess someplace instead.

For your question: “how you set up the log sender?”:
I am sending logs directly from Oracle PL/SQ to GELF TCP Input by json.
Using the UTL_TCP package, and the way I successfully do with Splunk.

best regards

(Ben van Staveren) #12

GELF requires a specific format though as far as I remember, you’re probably better off sending to a raw TCP input.

(Tess) #13

Yeah, absolutely.

Looking at the UTL_TCP documentation, it’s clearly a plain-text “raw” connection.

GELF expects a specific formatting of the data. You’ll need to use a “RAW” input instead @altink. What kind of input/receiver did you use back on Splunk?

(Altin Karaulli) #14

Thank yopu very much @Totally_Not_A_Robot and @benvanstaveren

In Splunk I am using /Data Inputs/TCP - alias the Splunk’s TCP Data Input.

In Graylog I am using GELF TCP, the way described in:

The test record I am loading - displayed as above - is identical to what is given in the Documentation’s example.

As far as I have seen it, its looked to me that GrayLog’s and Splunk’s TCP Data Input mechanisms are the same. The only difference being in that in Splunk I have used XML for formatting the record, while JSON for Graylog

Am I doing anything wrong?

best regards

(Tess) #15

Cool! So that sounds good! I wasn’t sure whether you were just sending ASCII strings at the input, or more structured data. It seems that you are, which is awesome.

The full GELF spec can be found here:

As long as your JSON fits that format, you’ll be fine. Making mistakes in the format could explain why Graylog says the message is invalid!

(Altin Karaulli) #16

Thank you very much @Totally_Not_A_Robot for your efforts.

Yes - I want to prepare as much as possible right into Oracle before I send to Graylog.

But it seems you have missed something:
please refer the link I have mentioned at the body of this ticket - the “GELF - Graylog 2.5.0 documentation”. It the version 2.5 of the link you lastly suggested.
As for the data I am sending - they are exactly those presented in the end of that page under the paragraph named “Example Payload”.
These lines too are displayed by me in this ticket’s body, please refer to 1/15. Here they are as they are send by Oracle’s dbms_output.put_line(graylog_record) - and as you can see they do match 100% the example.

I don’t guess the Documentation is wrong, so do I have something wrong in my stuff?

best regards,

(Altin Karaulli) #17

I am using GELF TCP Input with JSON formatted.
As documented in

The record/event loaded, is the one of the “Payload Example”.

best regards

(Tess) #18

Remarkable… So let’s try two extra troubleshooting steps! This is a fun problem :slight_smile:

  1. Can you start a Netcat listener somewhere and direct your Oracle app to send the logs to that listener? That will show you exactly what the outcome is that Graylog would receive.

  2. Vice versa, could you use Netcat to send the exact JSON output that you are sending directly into the Graylog input? That way you can tell whether it does make its way into Graylog.

(Altin Karaulli) #19

Thank you @Totally_Not_A_Robot.

I went for the second option, tested:

echo -n -e '{ "version": "1.1", "host": "dataplus05", "short_message": "A short message", "level": 5, "_some_info": "foo" }'"\0" | nc -w0 41000

This did upload correctly and without (!) extra internal messages.

I tested again by removing the:

 "\0" option

and I noticed that it was the “\0” option that made the difference. Without it - there are the ERROR, the java.Ilegal and the other 45 msg (in all). If it is present - it is OK.

But I still have my problem when I send it from Oracle’s UTL_TCP. I have the ERROR, java.Illegal and the messaged.
The string is identical to the one with the echo:

{ "version": "1.1", "host": "dataplus05", "short_message": "A short message", "level": 5, "_some_info": "foo" }"\0"

taken (dbms_output-ed) just before record send.

is there something that can be seen in the graylog internal events to identify the problem?

please advise,
best regards

(Altin Karaulli) #20

Why should the backslash-zero cause an impact on the result?

I mean when testing with echo. The docs, part “GELF via TCP”, notes:
“Each message needs to be delimited with a null byte ( \0 ) when sent in the same TCP connection.”

So it looks that its effect should happen only for multirecords sent in the same connection?

So why does it has an impact for a single record.

And my main problem still stands. echo does work, but in the real life, other things (like UTL_TCP) are supposeed to work too, cause it is with them the data comes, and not the echo.

best regards