Graylog Output Working but with Errors

janc · September 11, 2017, 7:17am

Hello,

Currently I have two Graylog Deployments.
One that receives a lot of messages from various sources (let’s call it BIG) and another one mainly for receiving logs from first graylog site (let’s call it SMALL).

I am trying to output a stream from BIG to SMALL and it seem to be working through GelfOutput and GelfInput TCP.
However I’ve been receiving a lot of errors like the following:

2017-09-11T07:01:54.428Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput]. java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String
at org.graylog2.outputs.GelfOutput.toGELFMessage(GelfOutput.java:245) ~[graylog.jar:?]
at org.graylog2.outputs.GelfOutput.write(GelfOutput.java:184) ~[graylog.jar:?]
at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:194) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

AND a lot of

ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput]. java.lang.ClassCastException: null

I’ve tried using UDP output and INPUT however results are still the same:

Currently I’m running with TCP and still keep on receiving errors.
Has anyone encountered this before?

Also, when TCP output is running, sometimes indexing will suddenly stop working, with the outputbuffer empty and processbuffer full. I would have to restart BIG graylog-servers in order to get it back to normal.

Thanks

jochen · September 11, 2017, 7:51am

janc:

2017-09-11T07:01:54.428Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput]. java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String
at org.graylog2.outputs.GelfOutput.toGELFMessage(GelfOutput.java:245) ~[graylog.jar:?]

Graylog (or more specifically the GELF output) expects the “facility” message field to be a string, but it was a number (of type Long) in this case.

This is a bug and should be fixed. Please create a bug report with the complete error message (and possibly a link to this topic) at Issues · Graylog2/graylog2-server · GitHub

Please post the following line(s) of this error message. Otherwise we cannot identify the root of the problem.

janc · September 12, 2017, 2:49am

Thanks for the response.
Although, would you recommend any workaround to output logs to another graylog through gelf without having to force facility to string?

I have tried using another syslog output plugin, which is not showing any errors but still no messages are being received on the other end.

As for the second error, I believe both the first one and this are related.
Here’s an excerpt from the logfile:

2017-09-12T02:05:18.622Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String
        at org.graylog2.outputs.GelfOutput.toGELFMessage(GelfOutput.java:245) ~[graylog.jar:?]
        at org.graylog2.outputs.GelfOutput.write(GelfOutput.java:184) ~[graylog.jar:?]
        at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:194) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2017-09-12T02:05:18.934Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String
        at org.graylog2.outputs.GelfOutput.toGELFMessage(GelfOutput.java:245) ~[graylog.jar:?]
        at org.graylog2.outputs.GelfOutput.write(GelfOutput.java:184) ~[graylog.jar:?]
        at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:194) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2017-09-12T02:05:18.934Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String
        at org.graylog2.outputs.GelfOutput.toGELFMessage(GelfOutput.java:245) ~[graylog.jar:?]
        at org.graylog2.outputs.GelfOutput.write(GelfOutput.java:184) ~[graylog.jar:?]
        at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:194) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2017-09-12T02:05:19.034Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String
        at org.graylog2.outputs.GelfOutput.toGELFMessage(GelfOutput.java:245) ~[graylog.jar:?]
        at org.graylog2.outputs.GelfOutput.write(GelfOutput.java:184) ~[graylog.jar:?]
        at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:194) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2017-09-12T02:05:23.572Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:24.733Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:24.955Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:26.689Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:26.934Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:27.351Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:28.377Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:28.462Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:29.367Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:29.708Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:29.803Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:29.879Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:30.562Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:31.880Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:32.666Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:34.863Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].

Thanks

jochen · September 12, 2017, 3:10pm

Follow up:

janc · September 14, 2017, 2:04am

Thanks for filing the bug report.
As a workaround, we used pipelines to ensure facility fields are strings, the logs are now being forwarded.

system · September 28, 2017, 2:05am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylof syslog output plugin cef format not working for nginx logs stream Graylog Add-ons	3	1166	June 1, 2021
[ProcessBufferProcessor] Unable to process message java.lang.NullPointerException Graylog Central (peer support)	8	785	October 20, 2020
Graylog outputs not working Graylog Central (peer support)	4	1361	November 22, 2017
Graylog Output Driver Disconnection Behavior Graylog Central (peer support)	3	460	December 28, 2017
Problem with gelf tcp input Graylog Central (peer support)	2	637	August 20, 2019

Graylog Output Working but with Errors

Related Topics