Data adapter failure with multiple value key

Graylog Central (peer support)
1

when i create a CSV data adapter, system report errors as:

image

and there error reports in server.log as following

2018-03-14T20:37:01.063+08:00 ERROR [LookupDataAdapter] Couldn't refresh data adapter <city-name/5aa90e9d28a47a0b22cac59e/@27528fde>
java.lang.IllegalArgumentException: Multiple entries with same key: 0023=ǭ�� and 0023=����
	at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:190) ~[graylog.jar:?]
	at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:109) ~[graylog.jar:?]
	at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:95) ~[graylog.jar:?]
	at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:373) ~[graylog.jar:?]
	at org.graylog2.lookup.adapters.CSVFileDataAdapter.parseCSVFile(CSVFileDataAdapter.java:168) ~[graylog.jar:?]
	at org.graylog2.lookup.adapters.CSVFileDataAdapter.doRefresh(CSVFileDataAdapter.java:109) ~[graylog.jar:?]
	at org.graylog2.plugin.lookup.LookupDataAdapter.refresh(LookupDataAdapter.java:89) ~[graylog.jar:?]
	at org.graylog2.lookup.LookupDataAdapterRefreshService.lambda$schedule$0(LookupDataAdapterRefreshService.java:142) ~[graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_161]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

I am using lastest graylogs release:

# rpm -qa|grep graylog-server
graylog-server-2.4.3-1.noarch

any idea on this?

2

Given that there aren’t duplicate entries in the key column of your CSV file, this looks like a bug.

Could you please provide the CSV file causing this bug and create an issue at https://github.com/Graylog2/graylog2-server/issues?

Also, can you say which character set has been used to create the CSV file? The CSV data adapter assumes UTF-8 by default.

3

Does the mutiple value lookup table come with a key with multiple attributes so multiple values, or just repeat the same key with other values? my case being:

city.code  city.name
code1      name1
code1      name2

those chars are in UTF-8.

4

This would work if city.name was your key column. The values in the key column need to be unique across the complete file.

5

thanks. so my table wrong with duplicate keys.

6

i realized that you should have mentioned also the display issue of those chinese words.

i upload the file in UTF-8 without BOM. but the result for the lookup doesn’t display correctly. let me open a issue for this.

7

It was proven that the wrong display issue due to the charset encoding issue. when i upload the CSV file with UTF-8 , keep push the lookup button, sometimes display correctly, most of time display wrongly. when reload the CSV file with UTF-8 without BOM, the display always correctly.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.