Copy input extractor failure


(Rayees Namathponna) #1

I am using Graylog 2.1 and ES 2.4.3.

I created “Copy Input” extractor to get key value pair, here is my message, trying to extract level and status from all the messages

2016-09-28 19:21:52,466 level=INFO tag="run_workflow.py" msg="Run complete for appname=cils, job_date=20160912, status=Passed starttime=Wed Sep 28 19:15:25 2016, endtime=Wed Sep 28 19:21:47 2016, duration=0:6:21, inputs=[{"path": "/esss/srg/20160912_1473688239855_4f29bbb6efdb3c39", "tag": "cilPurge", "stats": {"size": "13.47MB"}}, {"path": "/compressed/cil/20160912", "tag": "cil", "stats": {"size": "580.16MB"}}], outputs=[{"path": "/processed/test/parse//cil", "tag": "cil.output.folder", "stats": {"diffSize": "645.78MB", "newFiles": [], "endSize": "18.93GB", "startSize": "18.30GB"}}]”

Its failing with below error

0]: index [graylog_8], type [message], id [c0e7ea80-f572-11e6-b21e-5254007b267d], message [java.lang.IllegalArgumentException: Document contains at least one immense term in field="level" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[70, 101, 98, 32, 49, 55, 32, 49, 57, 58, 51, 57, 58, 51, 53, 32, 115, 106, 101, 108, 107, 51, 51, 32, 115, 121, 115, 116, 101, 109]...', original message: bytes can be at most 32766 in length; got 34944]
2017-02-17T19:39:56.795-05:00 ERROR [Extractor] Could not apply converter [tokenizer] of extractor [77e451d0-f3b9-11e6-b21e-5254007b267d].
java.lang.IllegalArgumentException: Multiple entries with same key: id=4038, and id=4038,
	at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:136) ~[graylog.jar:?]
	at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:98) ~[graylog.jar:?]
	at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:84) ~[graylog.jar:?]
	at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:295) ~[graylog.jar:?]
	at org.graylog2.inputs.converters.TokenizerConverter.convert(TokenizerConverter.java:55) ~[graylog.jar:?]
	at org.graylog2.plugin.inputs.Extractor.runConverters(Extractor.java:242) [graylog.jar:?]
	at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:228) [graylog.jar:?]
	at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:73) [graylog.jar:?]
	at org.graylog2.messageprocessors.MessageFilterChainProcessor.process(MessageFilterChainProcessor.java:100) [graylog.jar:?]
	at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:56) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:82) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:61) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:35) [graylog.jar:?]
	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
2017-02-17T19:39:57.677-05:00 ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[0]: index [graylog_8], type [message], id [c560a160-f572-11e6-b21e-5254007b267d], message [java.lang.IllegalArgumentException: Document contains at least one immense term in field="level" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[70, 101, 98, 32, 49, 55, 32, 49, 57, 58, 51, 57, 58, 52, 53, 32, 115, 106, 101, 108, 107, 51, 51, 32, 115, 121, 115, 116, 101, 109]...', original message: bytes can be at most 32766 in length; got 34944]

^Z


(Jan Doberstein) #2

first - I just had edited your posting that it can be read a little better.

Could you please post your Extractor you are using?

Additional you have the same field more than once (when looking at the error message)

Multiple entries with same key: id=4038, and id=4038,

When using Key-Value you need to watch that the Key is uniq in each message you send.

regards
Jan


(Jochen) #3

This has been fixed in Graylog 2.2.x:
https://github.com/Graylog2/graylog2-server/issues/1663