Graylog Output Working but with Errors


(Jan Caster) #1

Hello,

Currently I have two Graylog Deployments.
One that receives a lot of messages from various sources (let’s call it BIG) and another one mainly for receiving logs from first graylog site (let’s call it SMALL).

I am trying to output a stream from BIG to SMALL and it seem to be working through GelfOutput and GelfInput TCP.
However I’ve been receiving a lot of errors like the following:

2017-09-11T07:01:54.428Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput]. java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String
at org.graylog2.outputs.GelfOutput.toGELFMessage(GelfOutput.java:245) ~[graylog.jar:?]
at org.graylog2.outputs.GelfOutput.write(GelfOutput.java:184) ~[graylog.jar:?]
at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:194) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

AND a lot of

ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput]. java.lang.ClassCastException: null

I’ve tried using UDP output and INPUT however results are still the same:

Currently I’m running with TCP and still keep on receiving errors.
Has anyone encountered this before?

Also, when TCP output is running, sometimes indexing will suddenly stop working, with the outputbuffer empty and processbuffer full. I would have to restart BIG graylog-servers in order to get it back to normal.

Thanks


(Jochen) #2

Graylog (or more specifically the GELF output) expects the “facility” message field to be a string, but it was a number (of type Long) in this case.

This is a bug and should be fixed. Please create a bug report with the complete error message (and possibly a link to this topic) at https://github.com/Graylog2/graylog2-server/issues

Please post the following line(s) of this error message. Otherwise we cannot identify the root of the problem.


(Jan Caster) #3

Thanks for the response.
Although, would you recommend any workaround to output logs to another graylog through gelf without having to force facility to string?

I have tried using another syslog output plugin, which is not showing any errors but still no messages are being received on the other end.

As for the second error, I believe both the first one and this are related.
Here’s an excerpt from the logfile:

2017-09-12T02:05:18.622Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String
        at org.graylog2.outputs.GelfOutput.toGELFMessage(GelfOutput.java:245) ~[graylog.jar:?]
        at org.graylog2.outputs.GelfOutput.write(GelfOutput.java:184) ~[graylog.jar:?]
        at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:194) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2017-09-12T02:05:18.934Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String
        at org.graylog2.outputs.GelfOutput.toGELFMessage(GelfOutput.java:245) ~[graylog.jar:?]
        at org.graylog2.outputs.GelfOutput.write(GelfOutput.java:184) ~[graylog.jar:?]
        at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:194) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2017-09-12T02:05:18.934Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String
        at org.graylog2.outputs.GelfOutput.toGELFMessage(GelfOutput.java:245) ~[graylog.jar:?]
        at org.graylog2.outputs.GelfOutput.write(GelfOutput.java:184) ~[graylog.jar:?]
        at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:194) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2017-09-12T02:05:19.034Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String
        at org.graylog2.outputs.GelfOutput.toGELFMessage(GelfOutput.java:245) ~[graylog.jar:?]
        at org.graylog2.outputs.GelfOutput.write(GelfOutput.java:184) ~[graylog.jar:?]
        at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:194) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2017-09-12T02:05:23.572Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:24.733Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:24.955Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:26.689Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:26.934Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:27.351Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:28.377Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:28.462Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:29.367Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:29.708Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:29.803Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:29.879Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:30.562Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:31.880Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:32.666Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].
java.lang.ClassCastException: null
2017-09-12T02:05:34.863Z ERROR [OutputBufferProcessor] Error in output [class org.graylog2.outputs.GelfOutput].

Thanks


(Jochen) #4

Follow up:


(Jan Caster) #5

Thanks for filing the bug report.
As a workaround, we used pipelines to ensure facility fields are strings, the logs are now being forwarded.


(system) #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.