Graylog error when trying to create widget

wazlecracker · February 4, 2019, 8:24pm

Getting an error in the web interface of:

Loading quick values failed with status: Error: cannot GET http://10.100.11.80:9000/api/search/universal/relative/terms?query=gl2_source_input%3A5aa6d25d650c87209251451d%20AND%20sudo_commands%3A*&range=86400&field=source&order=source%3Adesc&size=50&stacked_fields=sudo_commands%2Csudo_username (500)

This error has only started happening since the server was reset after a read-only issue. Here are the errors that pop up in server.log when I try to create the widget:

at org.graylog2.syslog4j.server.impl.event.SyslogServerEvent.<init>(SyslogServerEvent.java:50) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.SyslogCodec.parse(SyslogCodec.java:132) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.SyslogCodec.decode(SyslogCodec.java:96) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2019-02-04T15:15:37.251-05:00 ERROR [GelfCodec] Could not parse JSON, first 400 characters: <174>Feb  4 15:15:37 BTF-QTS-FX2s-600MRD2 Severity: Informational, Category: Audit, MessageID: LOG007, Message: The previous log entry was repeated 4 times.
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: <174>Feb  4 15:15:37 BTF-QTS-FX2s-600MRD2 Severity: Informational, Category: Audit, MessageID: LOG007, Message: The previous log entry was repeated 4 times.; line: 1, column: 2]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:456) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1906) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2397) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:127) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2019-02-04T15:15:37.253-05:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=a289c730-28b9-11e9-8c42-0050568cfd45, journalOffset=40536621796, codec=gelf, payloadSize=156, timestamp=2019-02-04T20:15:37.251Z, remoteAddress=/10.100.10.21:57618} on input <5be06a94f3b47363eb30fa5d>.
2019-02-04T15:15:37.253-05:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=a289c730-28b9-11e9-8c42-0050568cfd45, journalOffset=40536621796, codec=gelf, payloadSize=156, timestamp=2019-02-04T20:15:37.251Z, remoteAddress=/10.100.10.21:57618}
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: <174>Feb  4 15:15:37 BTF-QTS-FX2s-600MRD2 Severity: Informational, Category: Audit, MessageID: LOG007, Message: The previous log entry was repeated 4 times.; line: 1, column: 2]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:456) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1906) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2397) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:127) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2019-02-04T15:15:37.323-05:00 ERROR [GelfCodec] Could not parse JSON, first 400 characters: <172>Feb  4 15:15:37 BTF-QTS-FX2s-600MRD2 Severity: Warning, Category: Audit, MessageID: USR8500, Message: Excessive login failures from 10.100.22.90; blocked for 300 seconds.
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: <172>Feb  4 15:15:37 BTF-QTS-FX2s-600MRD2 Severity: Warning, Category: Audit, MessageID: USR8500, Message: Excessive login failures from 10.100.22.90; blocked for 300 seconds.; line: 1, column: 2]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:456) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1906) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2397) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:127) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2019-02-04T15:15:37.324-05:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=a294c3b0-28b9-11e9-8c42-0050568cfd45, journalOffset=40536621797, codec=gelf, payloadSize=175, timestamp=2019-02-04T20:15:37.323Z, remoteAddress=/10.100.10.21:57618} on input <5be06a94f3b47363eb30fa5d>.
2019-02-04T15:15:37.324-05:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=a294c3b0-28b9-11e9-8c42-0050568cfd45, journalOffset=40536621797, codec=gelf, payloadSize=175, timestamp=2019-02-04T20:15:37.323Z, remoteAddress=/10.100.10.21:57618}
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: <172>Feb  4 15:15:37 BTF-QTS-FX2s-600MRD2 Severity: Warning, Category: Audit, MessageID: USR8500, Message: Excessive login failures from 10.100.22.90; blocked for 300 seconds.; line: 1, column: 2]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:456) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1906) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2397) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:127) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2019-02-04T15:15:37.339-05:00 ERROR [GelfCodec] Could not parse JSON, first 400 characters: <174>Feb  4 15:15:37 BTF-QTS-FX2s-600MRD2 Severity: Informational, Category: Audit, MessageID: USR8508, Message: SSH login was not successful (username=ssh123, ip=10.100.22.90, reason=IP address is in penalty box)
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: <174>Feb  4 15:15:37 BTF-QTS-FX2s-600MRD2 Severity: Informational, Category: Audit, MessageID: USR8508, Message: SSH login was not successful (username=ssh123, ip=10.100.22.90, reason=IP address is in penalty box); line: 1, column: 2]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:456) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1906) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2397) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:127) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2019-02-04T15:15:37.339-05:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=a2970da0-28b9-11e9-8c42-0050568cfd45, journalOffset=40536621798, codec=gelf, payloadSize=213, timestamp=2019-02-04T20:15:37.338Z, remoteAddress=/10.100.10.21:57618} on input <5be06a94f3b47363eb30fa5d>.
2019-02-04T15:15:37.339-05:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=a2970da0-28b9-11e9-8c42-0050568cfd45, journalOffset=40536621798, codec=gelf, payloadSize=213, timestamp=2019-02-04T20:15:37.338Z, remoteAddress=/10.100.10.21:57618}
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: <174>Feb  4 15:15:37 BTF-QTS-FX2s-600MRD2 Severity: Informational, Category: Audit, MessageID: USR8508, Message: SSH login was not successful (username=ssh123, ip=10.100.22.90, reason=IP address is in penalty box); line: 1, column: 2]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:456) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1906) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2397) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:127) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Any advice on how to resolve this?

macko003 · February 4, 2019, 8:32pm

please read you log careful…
after that do a google search for gelf, and syslog.

wazlecracker · February 4, 2019, 8:36pm

I’m somewhat new to Linux in general, so I’m not following what you’re suggesting about the log.

macko003 · February 4, 2019, 8:39pm

it’s not linux connected thing…
READ please
all error in the message

wazlecracker · February 4, 2019, 8:41pm

So it’s a JSON issue, correct?

macko003 · February 4, 2019, 8:53pm

almost.
glef is in json format. and it get a syslog message (3. line), so it can’t phrase it.
You send syslog message to a wrong input.

but it’s not connected the ‘quick values’ issue. Graylog drop this messages.

what is your elasticsearch status (system menu, overview)?
could you send pictures how do you try to create?
do you have a dashbord, and you got it when you open it?

wazlecracker · February 4, 2019, 8:57pm

worker@ElasticSearchServer:~$ systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-02-04 11:55:42 EST; 4h 0min ago
     Docs: http://www.elastic.co
  Process: 10850 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exi
 Main PID: 10857 (java)
    Tasks: 55
   Memory: 22.5G
      CPU: 6h 12min 17.671s
   CGroup: /system.slice/elasticsearch.service
           └─10857 /usr/bin/java -Xms15g -Xmx15g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancy

Feb 04 11:55:42 ElasticSearchServer systemd[1]: Starting Elasticsearch...
Feb 04 11:55:42 ElasticSearchServer systemd[1]: Started Elasticsearch.

wazlecracker · February 4, 2019, 9:01pm

This is an input that takes in logs from 4 Linux servers. I don’t get the error when the search that produces this is ran. This screen comes up when I try to add 2 stacked fields that I created.

These are the widgets from the dashboard that worked this morning:

macko003 · February 5, 2019, 8:12am

You can try to do a recalculate index ranges (System, indices, your index set, and maintenance button)

system · February 19, 2019, 8:12am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog-server web ui suddenly does not work Graylog Central (peer support)	2	638	October 22, 2020
Graylog logo and search buttons getting an error Graylog Central (peer support)	2	422	September 19, 2019
Getting error when searchin logs Graylog Central (peer support)	2	345	October 21, 2018
Error when run graylog.jar Graylog Central (peer support)	9	722	September 30, 2020
Searching problems Graylog Central (peer support)	5	1317	December 4, 2017

Graylog error when trying to create widget

Related Topics