java.lang.IllegalStateException: GELF message is too short


(Altin Karaulli) #1

Dear All,

I am uploading data via GELF TCP Input, exactly the “Example Payload” as in the:
GELF - Graylog 2.5.0 documentation

{
  "version": "1.1",
  "host": "example.org",
  "short_message": "A short message that helps you identify what is going on",
  "full_message": "Backtrace here\n\nmore stuff",
  "timestamp": 1385053862.3072,
  "level": 1,
  "_user_id": 9001,
  "_some_info": "foo",
  "_some_env_var": "bar"
}

The event does load successfully, but what I found for source:graylog-server is:

facility
    runit-service
from_gelf
    true
level
    6
message
    java.lang.IllegalStateException: GELF message is too short. Not even the type header would fit.
source
    graylog-server
timestamp
2018-12-17T22:52:22.914Z

and also

facility
    runit-service
from_gelf
    true
level
    6
message
    ERROR [NettyTransport] Error in Input [GELF TCP/5c119b1eb358cb056b417b89] (channel [id: 0xc6ed250a, /192.168.1.22:52136 :> /192.168.1.74:41000])
source
    graylog-server
timestamp
2018-12-17T23:12:18.241Z

and when searched (source:graylog-server ), there are 45 messages in general.

Why the ERROR and the Exception ?

best regards,
Altin


(Konrad Merz) #2

Hi Altin,

Can you be a bit more specific about your setup. Can you make screenshots of your GELF Input configuration and explain, how you set up the log sender?

Konrad


(Tess) #3

So does that mean the message are being received correctly? And that the GELF input is working and that you’re not missing any data? That’s our first priority. After that we can focus on cosmetics :slight_smile:


(Altin Karaulli) #4

YES
the log comes correctly.
But I have too much cosmectics (some 45 records) for each one uploaded, where the ERROR and java.lang.Illegal… are present too.

I will upload images and details asap

thank you very much,
best regards
Altin


(Tess) #5

Sorry, I had misunderstood your previous post. I thought you said you received 45 correct messages, but what you means is that for every 1 correct message, you get 45 errors as well? Yikes! That’s not pretty.


(Altin Karaulli) #6

No Sir.

What happens is that for every 1 record uploaded, I see some 45 generated as (it looks) internal graylog messages, where between them, I can find the highlighted:
ERROR and java.lang.IllegalStateException - related to my single upload

I cannot find a place here to upload them all.

thank you very much Totally_not_A_Robot
Altin


(Tess) #7

The errors can be copy-pasted over here in a codeblock, or you could post them on PasteBin and share the link here.

If it’s a screenshot, you can simply upload the image by using the “insert image” button.


(Altin Karaulli) #8

Thank you Tess,

you are right now. I have one good record and 45 cosmetics, two of them, One ERROR and one Java Illegal

below is my Omega CA GELF TCP Input

bind_address:
 0.0.0.0
decompress_size_limit:
 8388608
max_message_size:
 2097152
override_source:
 <empty>
port:
 41000
recv_buffer_size:
 1048576
tcp_keepalive:
 false
tls_cert_file:
 <empty>
tls_client_auth:
 disabled
tls_client_auth_cert_file:
 <empty>
tls_enable:
 false
tls_key_file:
 <empty>
tls_key_password:
 ********
use_null_delimiter:
 true

best regards
Altin

ps. I can find no “insert image” link in my profile,
the records
TCP Input Messages 001


(Tess) #9

I can’t make a screenshot right now, so I’ll describe it.

In the editor where you write your replies, there’s a menu bar:

balloon bold italic | insertlink insertquote insertcode insertimage | bullets numberedlist smiley options

There’s an insert image option :slight_smile: Seventh button from the left.


(Altin Karaulli) #10

thank you very much for the image link, found where you pointed.
but now that I uploaded the TCP Input config as text, I guess you don’t need it anymore?

best regards
Altin


(Altin Karaulli) #11

Hi Konrad,

sorry for missing the point and answering to Tess someplace instead.

For your question: “how you set up the log sender?”:
I am sending logs directly from Oracle PL/SQ to GELF TCP Input by json.
Using the UTL_TCP package, and the way I successfully do with Splunk.

best regards
Altin


(Ben van Staveren) #12

GELF requires a specific format though as far as I remember, you’re probably better off sending to a raw TCP input.


(Tess) #13

Yeah, absolutely.

Looking at the UTL_TCP documentation, it’s clearly a plain-text “raw” connection.

https://docs.oracle.com/cd/B28359_01/appdev.111/b28419/u_tcp.htm#i1003167

GELF expects a specific formatting of the data. You’ll need to use a “RAW” input instead @altink. What kind of input/receiver did you use back on Splunk?


(Altin Karaulli) #14

Thank yopu very much @Totally_Not_A_Robot and @benvanstaveren

In Splunk I am using /Data Inputs/TCP - alias the Splunk’s TCP Data Input.

In Graylog I am using GELF TCP, the way described in:
http://docs.graylog.org/en/2.5/pages/gelf.html#gelf-via-tcp

The test record I am loading - displayed as above - is identical to what is given in the Documentation’s example.

As far as I have seen it, its looked to me that GrayLog’s and Splunk’s TCP Data Input mechanisms are the same. The only difference being in that in Splunk I have used XML for formatting the record, while JSON for Graylog

Am I doing anything wrong?

best regards
Altin


(Tess) #15

Cool! So that sounds good! I wasn’t sure whether you were just sending ASCII strings at the input, or more structured data. It seems that you are, which is awesome.

The full GELF spec can be found here:

http://docs.graylog.org/en/2.4/pages/gelf.html

As long as your JSON fits that format, you’ll be fine. Making mistakes in the format could explain why Graylog says the message is invalid!


(Altin Karaulli) #16

Thank you very much @Totally_Not_A_Robot for your efforts.

Yes - I want to prepare as much as possible right into Oracle before I send to Graylog.

But it seems you have missed something:
please refer the link I have mentioned at the body of this ticket - the “GELF - Graylog 2.5.0 documentation”. It the version 2.5 of the link you lastly suggested.
As for the data I am sending - they are exactly those presented in the end of that page under the paragraph named “Example Payload”.
These lines too are displayed by me in this ticket’s body, please refer to 1/15. Here they are as they are send by Oracle’s dbms_output.put_line(graylog_record) - and as you can see they do match 100% the example.

I don’t guess the Documentation is wrong, so do I have something wrong in my stuff?

best regards,
Altin


(Altin Karaulli) #17

I am using GELF TCP Input with JSON formatted.
As documented in
http://docs.graylog.org/en/2.5/pages/gelf.html#gelf-via-tcp

The record/event loaded, is the one of the “Payload Example”.

best regards
Altin


(Tess) #18

Remarkable… So let’s try two extra troubleshooting steps! This is a fun problem :slight_smile:

  1. Can you start a Netcat listener somewhere and direct your Oracle app to send the logs to that listener? That will show you exactly what the outcome is that Graylog would receive.

  2. Vice versa, could you use Netcat to send the exact JSON output that you are sending directly into the Graylog input? That way you can tell whether it does make its way into Graylog.


(Altin Karaulli) #19

Thank you @Totally_Not_A_Robot.

I went for the second option, tested:

echo -n -e '{ "version": "1.1", "host": "dataplus05", "short_message": "A short message", "level": 5, "_some_info": "foo" }'"\0" | nc -w0 192.168.1.74 41000

This did upload correctly and without (!) extra internal messages.

I tested again by removing the:

 "\0" option

and I noticed that it was the “\0” option that made the difference. Without it - there are the ERROR, the java.Ilegal and the other 45 msg (in all). If it is present - it is OK.

But I still have my problem when I send it from Oracle’s UTL_TCP. I have the ERROR, java.Illegal and the messaged.
The string is identical to the one with the echo:

{ "version": "1.1", "host": "dataplus05", "short_message": "A short message", "level": 5, "_some_info": "foo" }"\0"

taken (dbms_output-ed) just before record send.

is there something that can be seen in the graylog internal events to identify the problem?

please advise,
best regards
Altin


(Altin Karaulli) #20

Why should the backslash-zero cause an impact on the result?

I mean when testing with echo. The docs, part “GELF via TCP”, notes:
“Each message needs to be delimited with a null byte ( \0 ) when sent in the same TCP connection.”

So it looks that its effect should happen only for multirecords sent in the same connection?

So why does it has an impact for a single record.

And my main problem still stands. echo does work, but in the real life, other things (like UTL_TCP) are supposeed to work too, cause it is with them the data comes, and not the echo.

best regards
Altin