I am using Graylog 2.1 and ES 2.4.3.
I created “Copy Input” extractor to get key value pair, here is my message, trying to extract level and status from all the messages
2016-09-28 19:21:52,466 level=INFO tag="run_workflow.py" msg="Run complete for appname=cils, job_date=20160912, status=Passed starttime=Wed Sep 28 19:15:25 2016, endtime=Wed Sep 28 19:21:47 2016, duration=0:6:21, inputs=[{"path": "/esss/srg/20160912_1473688239855_4f29bbb6efdb3c39", "tag": "cilPurge", "stats": {"size": "13.47MB"}}, {"path": "/compressed/cil/20160912", "tag": "cil", "stats": {"size": "580.16MB"}}], outputs=[{"path": "/processed/test/parse//cil", "tag": "cil.output.folder", "stats": {"diffSize": "645.78MB", "newFiles": [], "endSize": "18.93GB", "startSize": "18.30GB"}}]”
Its failing with below error
0]: index [graylog_8], type [message], id [c0e7ea80-f572-11e6-b21e-5254007b267d], message [java.lang.IllegalArgumentException: Document contains at least one immense term in field="level" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[70, 101, 98, 32, 49, 55, 32, 49, 57, 58, 51, 57, 58, 51, 53, 32, 115, 106, 101, 108, 107, 51, 51, 32, 115, 121, 115, 116, 101, 109]...', original message: bytes can be at most 32766 in length; got 34944]
2017-02-17T19:39:56.795-05:00 ERROR [Extractor] Could not apply converter [tokenizer] of extractor [77e451d0-f3b9-11e6-b21e-5254007b267d].
java.lang.IllegalArgumentException: Multiple entries with same key: id=4038, and id=4038,
at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:136) ~[graylog.jar:?]
at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:98) ~[graylog.jar:?]
at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:84) ~[graylog.jar:?]
at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:295) ~[graylog.jar:?]
at org.graylog2.inputs.converters.TokenizerConverter.convert(TokenizerConverter.java:55) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.Extractor.runConverters(Extractor.java:242) [graylog.jar:?]
at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:228) [graylog.jar:?]
at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:73) [graylog.jar:?]
at org.graylog2.messageprocessors.MessageFilterChainProcessor.process(MessageFilterChainProcessor.java:100) [graylog.jar:?]
at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:56) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:82) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:61) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:35) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
2017-02-17T19:39:57.677-05:00 ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[0]: index [graylog_8], type [message], id [c560a160-f572-11e6-b21e-5254007b267d], message [java.lang.IllegalArgumentException: Document contains at least one immense term in field="level" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[70, 101, 98, 32, 49, 55, 32, 49, 57, 58, 51, 57, 58, 52, 53, 32, 115, 106, 101, 108, 107, 51, 51, 32, 115, 121, 115, 116, 101, 109]...', original message: bytes can be at most 32766 in length; got 34944]
^Z