Issues with HTTPS and inputs

adrianrus · September 21, 2023, 1:38pm

Graylog 5.0.11+30bdbfa on xxxxxxxx (Eclipse Adoptium 17.0.8 on Linux 5.15.0-1038-ibm)
Opensearch

I have configured 3 inputs
Some Linux in clear and Linux TLS and Windows TLS.
All working perfectly while on HTTP.

Then i am trying to upgrade to HTTPS certificate for the graylog server.
I followed the documentation provided by GL etc.

Still get into this problem.

When attempting to START INPUT it doesn’t go to green.

2023-09-21T15:25:37.345+02:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=59f75c01-5882-11ee-b4ef-0200034690cd, messageQueueId=14531815, codec=gelf, payloadSize=81, timestamp=2023-09-21T13:25:37.344Z, remoteAddress=/10.13.200.106:52262}
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'IPfI0L': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (String)"IPfI0L.�F=v��`w�^`P�\u007FOw�fH�$�D9D��]��WU\u000F��\u0007��0�&�?�����{8\u001D\u0019/���I�G�0\u001A����ya4��\u001A�8"; line: 1, column: 7]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:745) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2961) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2939) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._matchToken(ReaderBasedJsonParser.java:2713) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1986) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:802) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4703) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3076) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:156) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:95) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
2023-09-21T15:25:37.345+02:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=59f75c08-5882-11ee-b4ef-0200034690cd, messageQueueId=14531822, codec=gelf, payloadSize=470, timestamp=2023-09-21T13:25:37.344Z, remoteAddress=/10.13.200.106:52262} on input <64fdbbd30174cb5d796c2d4e>.
2023-09-21T15:25:37.345+02:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=59f75c08-5882-11ee-b4ef-0200034690cd, messageQueueId=14531822, codec=gelf, payloadSize=470, timestamp=2023-09-21T13:25:37.344Z, remoteAddress=/10.13.200.106:52262}
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('�' (code 65533 / 0xfffd)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (String)"��~3�\u0018��\u001F�\u001D�����\u001A}ʶٛ+��5��\u001E��r��p���+�-�\u0018]��ޗ\u0010\u0012�2�+�I�W�p�����-` kGp��y�\u000Cy��5YM\u0018�8\u0016;&�\u0008���*��k(?rr$5TpK��\u0013�\u0009A*��7��-�\u0017L�s�K��`E\u000B=F\u0001\u007F+&� �\u001F�q���p�z�v�0��e�\h\u0003���1I\u001A\u00184��b��\u0003�AHƇݲ�����۔Kz$��Fo� j��/���;Lq\u0005��VWU�a,�\u0018�\u0006��-�\�\u001Bbg\u001CxA��)
|Ǜ�A��v\u0011Zi�c��{� �GC�F�yj�Q��53\u001Aa�C�.*���I(1\u00025*v���V��È��8�c\>5z�t/F�\u001Ciϓ��<a���DG�w���%m����Y-R��덼���\u0013�[�(���1�X��\u0001~�h���\N�������\u000B\&})�y:]��BE�.8F\u0017y��\u0016򮬋��* ���Z�o�O�I�s�b�^o�����
K?�^\��b$�4�u�9�\u001F��4\u000E� \u0009�n����_��\u007F"; line: 1, column: 2]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:735) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:659) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:2005) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:802) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4703) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3076) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:156) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:95) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
2023-09-21T15:25:37.345+02:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=59f75c00-5882-11ee-b4ef-0200034690cd, messageQueueId=14531814, codec=gelf, payloadSize=264, timestamp=2023-09-21T13:25:37.344Z, remoteAddress=/10.13.200.106:52262}
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('�' (code 65533 / 0xfffd)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
7<�e�\u0010��hh�V\u000F�<�<��V$�r��Y��\u0004:J-\u0007�\u0007�\u0016ā�Bp�\u001F0P\u0006'�\������ʚ���\u0014�y\u001E�tH{���w�m�%n����Y�㚗�\u001By�e�'B7�i�q͔c���\u0017\u0007�=�9�\u0006s>�\u0006R�_\u0017�\�bԕҜ����\u0017*ju���(*����e�-�W\u0006�����~+']�VR��\u0016��r��\u000B��\u0015�n,�����J�m��@�7���I��`���ni���_���o��/�x��VɎ�@\u0010�\u0015��1�����"; line: 1, column: 2]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:735) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:659) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:2005) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:802) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4703) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3076) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:156) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:95) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
2023-09-21T15:25:37.345+02:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=59f75c05-5882-11ee-b4ef-0200034690cd, messageQueueId=14531819, codec=gelf, payloadSize=60, timestamp=2023-09-21T13:25:37.344Z, remoteAddress=/10.13.200.106:52262}
com.fasterxml.jackson.core.JsonParseException: Unexpected close marker '}': expected ']' (for root starting at [Source: (String)"}\��(���8��\u0010y\u0013\u0014��7N�8\u0009�"4�p�
Uu
A菆V�H�P��΃+fD�,�~��艣�Pb"; line: 1, column: 0])
 at [Source: (String)"}\��(���8��\u0010y\u0013\u0014��7N�8\u0009�"4�p�
Uu
A菆V�H�P��΃+fD�,�~��艣�Pb"; line: 1, column: 2]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:735) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserBase._reportMismatchedEndMarker(ParserBase.java:1073) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._closeScope(ReaderBasedJsonParser.java:2982) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:710) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4703) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3076) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:156) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:95) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:?]

drewmiranda-gl · September 21, 2023, 4:20pm

This looks like a possibly mismatch of certificates configured, for example the input using a different cert than the client.

Are you having issues with only the GELF UDP input? Was it working previously and then stopped working? Did anything happen that caused it to stop working?

As far as I can tell Gelf UDP does not support TLS and does not provide options to use TLS.

What agent/collector are you using to send logs to graylog?

adrianrus · September 21, 2023, 7:32pm

I have 3 inputs.

A Linux one without TLS - plain
A Windows one with TLS
A Linux one with TLS

all theree were working flawless before it started to move the graylog server from http to https.

adrianrus · September 25, 2023, 12:03pm

still can’t figure out this one

log file tail

�)8y%`\u0003\u0012\u0007~\u001C�7�yQ2�\u000Bğ��\u0010Br<*t\u001E�:��\u0016\u000CZ��h)b��#~o\u001Bw\u0019\u001B=@O��8.�y��UƯ\u0008[�9�\u000BN/+=85��b��,��\u000F�\u000EQ\u0018�"�s!“,۬�Y��0�e\u0007~r��n\u001C�\u0016”\u0012C�\u0009�z�L�I��\u0006\u0014��\u0018�Qۖ�n�/�\u0001��2Կ�67�\u0019|c�\u001BQZ\u0011�"�m\u0014�\u0012a\u0009J6�1�Q\u0011��\u0007�8O��nN��\u0018q�t\u00153\u001A1Hܽ\u001F�\u001C��\u0014�b3j�n�\u0014\u007F�\u0001��ٿx��R�n� \u0010�\u0015�:�"; line: 1, column: 2]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:735) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:659) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:2005) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:802) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4703) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3076) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:156) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:95) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Unknown Source) [?:?]
2023-09-25T14:01:32.147+02:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=44723d20-5b9b-11ee-b216-0200034690cd, messageQueueId=40946894, codec=gelf, payloadSize=354, timestamp=2023-09-25T12:01:32.146Z, remoteAddress=/10.13.200.101:51698}
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('' (code 92)): expected a valid value (JSON String, Number, Array, Object or token ‘null’, ‘true’ or ‘false’)
�{\u001CxAH�)\u0009��\u001FŞ�\u000B��F��4��׆V�X�P�#ơ��2a�\u0003��<I�g8�\u00030�l\u0007�(x!�9\u000C8h\u0004��\u0081�b��9�^PY/�3U\u000Em�tP\u0017��va↌n0��"��O�,��:�2a�v,5��r\u0005_B�5l@�%l �\u0002v\u0001p\u0007��.�fA��\u001B)�$�OF�28V\u001AeE5�y��z\u000C��9�VP�p��8�9V��/-�±][:y��\u0015\u0019[,*Au��h:\u001CE�\u0019�_0%~��y��?�g.\u001Dx��R]k�0\u0010�+B�q��ԍ�Qw��&��"; line: 1, column: 2]gtFw\u007F�w^\u0005Z\u0017eΗ��uΖUYT�\u0012�w\u0013ظ=\u000C�
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:735) ~[graylog.jar:?]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:659) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:2005) ~[graylog.jar:?]
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:802) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4703) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3076) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:156) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:95) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Unknown Source) [?:?]

adrianrus · September 26, 2023, 3:38pm

after days of digging.

The KEY for the cer was in ".key’ instead of “.pem” as per the new 5.x HTTPS documentation.
I took my cookbook form the 4.3 where everything worked great and applied those steps i knew it worked with.

All is ok now.

Damn s*!t

joe.gross · September 29, 2023, 3:43pm

I’m glad you worked your way through it. Thank you for coming back to close the loop for future users in a similar position.

system · October 13, 2023, 3:44pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to start inputs after implementing HTTPS on Graylog web interface Graylog Central (peer support)	6	2461	January 25, 2022
Inputs not working after SSL enabled Graylog Central (peer support) basic-configuration , elastic	11	1560	May 5, 2022
Inputs not starting Graylog Central (peer support)	7	4002	March 19, 2019
Cannot create input with https enabled Graylog Central (peer support)	2	987	April 17, 2020
Unable to start any input after setting up web interface as SSL Graylog Central (peer support)	5	704	June 17, 2019

Issues with HTTPS and inputs

Related topics