Issues with user/roles/auth on new install of 4.1.3

Graylog Central (peer support)
1

We have a newer install of Graylog 4.1.3, in which we’ve been in the process of configuring it to match our Graylog 2.4.0 servers. At some point, something went wrong. We are now unable to bring up the Users and Teams, Roles, or Authentication choices under the System pull-down. When we try we get an error, “Loading users failed with status: IOException encountered while reading from a byte array input stream - There was an error fetching a resource: Internal Server Error. Additional information: IOException encountered while reading from a byte array input stream” and it just spins saying “Loading…”. I’m not even sure where/how to start troubleshooting this…

We are running four nodes on Graylog version 4.1.3+9d79c05, running on CentOS 7.9.2009.

Any ideas on where to look would be appreciated.

2

Graylog Server log, just after cycling server service and trying to refresh System:

2022-05-24T10:18:54.171-07:00 ERROR [GelfCodec] Could not parse JSON, first 400 characters: default send string
com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘default’: was expecting (‘true’, ‘false’ or ‘null’)
at [Source: (String)“default send string”; line: 1, column: 8]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
2022-05-24T10:18:54.171-07:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=8db7a980-db85-11ec-ab81-005056931163, messageQueueId=50479054, codec=gelf, payloadSize=19, timestamp=2022-05-24T17:18:39.640Z, remoteAddress=/10.104.72.252:37247} on input <6254cf616b834f046bcb223c>.
2022-05-24T10:18:54.172-07:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=8db7a980-db85-11ec-ab81-005056931163, messageQueueId=50479054, codec=gelf, payloadSize=19, timestamp=2022-05-24T17:18:39.640Z, remoteAddress=/10.104.72.252:37247}
com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘default’: was expecting (‘true’, ‘false’ or ‘null’)
at [Source: (String)“default send string”; line: 1, column: 8]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
2022-05-24T10:18:54.174-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=winlogs-gelf, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xed4fbef7, L:/0.0.0.0:5414]) should be 1048576 but is 212992.
2022-05-24T10:18:54.175-07:00 ERROR [GelfCodec] Could not parse JSON, first 400 characters: default send string
com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘default’: was expecting (‘true’, ‘false’ or ‘null’)
at [Source: (String)“default send string”; line: 1, column: 8]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
2022-05-24T10:18:54.176-07:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=8e116240-db85-11ec-ab81-005056931163, messageQueueId=50479058, codec=gelf, payloadSize=19, timestamp=2022-05-24T17:18:40.228Z, remoteAddress=/10.104.72.253:46333} on input <6254cf616b834f046bcb223c>.
2022-05-24T10:18:54.176-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=syslog-udp, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x9a456755, L:/0.0.0.0:5514]) should be 262144 but is 212992.
2022-05-24T10:18:54.176-07:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=8e116240-db85-11ec-ab81-005056931163, messageQueueId=50479058, codec=gelf, payloadSize=19, timestamp=2022-05-24T17:18:40.228Z, remoteAddress=/10.104.72.253:46333}
com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘default’: was expecting (‘true’, ‘false’ or ‘null’)
at [Source: (String)“default send string”; line: 1, column: 8]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
2022-05-24T10:18:54.176-07:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=Beats, type=org.graylog.plugins.beats.Beats2Input, nodeId=null} (channel [id: 0x76e49a7b, L:/0.0.0.0:5044]) should be 1048576 but is 212992.
2022-05-24T10:18:54.179-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=AFM-udp-input, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x68ae3e8d, L:/0.0.0.0:5515]) should be 262144 but is 212992.
2022-05-24T10:18:54.180-07:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=gelf-udp, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x929a55a5, L:/0.0.0.0:12202]) should be 262144 but is 212992.
2022-05-24T10:18:54.190-07:00 INFO [InputStateListener] Input [GELF TCP/6261cd495d068a3ef60f7806] is now RUNNING
2022-05-24T10:18:54.192-07:00 INFO [InputStateListener] Input [GELF UDP/6254d0d5510c102d86aea500] is now RUNNING
2022-05-24T10:18:54.194-07:00 INFO [InputStateListener] Input [Beats/62672a16fde0a96cd34aad9f] is now RUNNING
2022-05-24T10:18:54.195-07:00 INFO [InputStateListener] Input [Syslog UDP/6254d07319d6461124fbc05a] is now RUNNING
2022-05-24T10:18:54.203-07:00 INFO [InputStateListener] Input [Syslog UDP/6254cec5510c102d86aea2e6] is now RUNNING
2022-05-24T10:18:54.206-07:00 INFO [InputStateListener] Input [GELF UDP/6254cf616b834f046bcb223c] is now RUNNING
2022-05-24T10:18:54.208-07:00 INFO [InputStateListener] Input [Syslog UDP/6254cfd819d6461124fbbfbb] is now RUNNING
2022-05-24T10:18:54.727-07:00 ERROR [GelfCodec] Could not parse JSON, first 400 characters: default send string
com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘default’: was expecting (‘true’, ‘false’ or ‘null’)
at [Source: (String)“default send string”; line: 1, column: 8]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
2022-05-24T10:18:54.728-07:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=96b1f0e0-db85-11ec-9e1b-005056931163, messageQueueId=50479080, codec=gelf, payloadSize=19, timestamp=2022-05-24T17:18:54.702Z, remoteAddress=/10.104.72.252:44772} on input <6254cf616b834f046bcb223c>.
2022-05-24T10:18:54.728-07:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=96b1f0e0-db85-11ec-9e1b-005056931163, messageQueueId=50479080, codec=gelf, payloadSize=19, timestamp=2022-05-24T17:18:54.702Z, remoteAddress=/10.104.72.252:44772}
com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘default’: was expecting (‘true’, ‘false’ or ‘null’)
at [Source: (String)“default send string”; line: 1, column: 8]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
2022-05-24T10:18:55.264-07:00 ERROR [GelfCodec] Could not parse JSON, first 400 characters: default send string
com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘default’: was expecting (‘true’, ‘false’ or ‘null’)
at [Source: (String)“default send string”; line: 1, column: 8]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
2022-05-24T10:18:55.264-07:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=9706eeb0-db85-11ec-9e1b-005056931163, messageQueueId=50479081, codec=gelf, payloadSize=19, timestamp=2022-05-24T17:18:55.259Z, remoteAddress=/10.104.72.253:12889} on input <6254cf616b834f046bcb223c>.
2022-05-24T10:18:55.265-07:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=9706eeb0-db85-11ec-9e1b-005056931163, messageQueueId=50479081, codec=gelf, payloadSize=19, timestamp=2022-05-24T17:18:55.259Z, remoteAddress=/10.104.72.253:12889}
com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘default’: was expecting (‘true’, ‘false’ or ‘null’)
at [Source: (String)“default send string”; line: 1, column: 8]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
2022-05-24T10:18:59.659-07:00 ERROR [GelfCodec] Could not parse JSON, first 400 characters: default send string
com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘default’: was expecting (‘true’, ‘false’ or ‘null’)
at [Source: (String)“default send string”; line: 1, column: 8]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
2022-05-24T10:18:59.660-07:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=99a58e60-db85-11ec-9e1b-005056931163, messageQueueId=50479084, codec=gelf, payloadSize=19, timestamp=2022-05-24T17:18:59.654Z, remoteAddress=/10.104.72.252:61975} on input <6254cf616b834f046bcb223c>.
2022-05-24T10:18:59.660-07:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=99a58e60-db85-11ec-9e1b-005056931163, messageQueueId=50479084, codec=gelf, payloadSize=19, timestamp=2022-05-24T17:18:59.654Z, remoteAddress=/10.104.72.252:61975}
com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘default’: was expecting (‘true’, ‘false’ or ‘null’)
at [Source: (String)“default send string”; line: 1, column: 8]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
2022-05-24T10:19:00.211-07:00 ERROR [GelfCodec] Could not parse JSON, first 400 characters: default send string
com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘default’: was expecting (‘true’, ‘false’ or ‘null’)
at [Source: (String)“default send string”; line: 1, column: 8]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2853) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1899) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:757) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4043) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]

3

Hello && Welcome

Can you login with Graylogs default user? I believe its admin and what ever hash password that was configured.

If not, first place to look is at you Graylog/Elasticsearch/MongoDb log files depending on what OS is used you can find it here

Next, Is this Graylog Enterprise version? If not that might be the issue.

Graylog changed the way authentication works since version 2.4. Check it out here for these changes.

4

Thanks for the reply. Sorry for the slow reply, I’ve been buried…

I can login to Graylog with an account that has admin permissions, but the person who set it up didn’t give me the admin user credentials and they are no longer here. Would the default admin user have some other “extra” permissions. Otherwise, I had attached the Graylog logs above and I’m not seeing anything super-helpful. There’s also nothing in the MongoDB log file. Otherwise, our ElasticSearch is in AWS OpenSearch, so I’m trying to figure out if I can even get to related logs there.

It does not appear that we have Graylog Enterprise and I don’t believe that we’re “supposed to”. Our old Graylog cluster also doesn’t show licensing for Enterprise…

5

Hello,

If you have a user role Amin you should be good.
As for permissions might want to look here

As stated above somethings have change in User/Roles/Permissions.

6

So, it looks like I’m actually in good shape all of a sudden. We have four nodes in our Graylog cluster and two were offline, as we were working on automation to rebuild them. I just got those nodes back online yesterday and could all of a sudden get to the users/roles/and auth sections in the WebUI again. I’m not sure why that is the case, but I’m happy that it’s working again… Thanks for your assistance GSmith.

1 Like
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.