CSV Adapter cannot parse file, but Lookup still works

derPhlipsi · July 11, 2017, 4:25pm

Hey,

I just noticed this in my Graylog logfiles:

2017-07-11T17:57:09.379+02:00 INFO  [LookupTableService] Data Adapter cisco-mnemonics/5964ec0f8773ef433de0754c [@5cbf248b] RUNNING
2017-07-11T17:57:09.379+02:00 INFO  [LookupDataAdapterRefreshService] Adding job for <cisco-mnemonics/5964ec0f8773ef433de0754c/@5cbf248b> [interval=30000ms]
2017-07-11T17:57:09.379+02:00 INFO  [LookupTableService] Starting lookup table cisco-mnemonics/5964ed388773ef433de07691 [@105138fd] using cache do-not-cache/5964ed1a8773ef433de0766e [@469b316d], data adapter cisco-mnemonics/5964ec0f8773ef433de0754c [@5cbf248b]
2017-07-11T17:57:09.380+02:00 INFO  [LookupTableService] Replaced previous lookup table cisco-mnemonics [@2875e8ac]
2017-07-11T17:57:09.381+02:00 INFO  [LookupTableService] Data Adapter cisco-mnemonics/5964ec0f8773ef433de0754c [@3c573f3] STOPPING, was RUNNING
2017-07-11T17:57:09.381+02:00 INFO  [LookupDataAdapterRefreshService] Removing job for <cisco-mnemonics/5964ec0f8773ef433de0754c/@3c573f3>
2017-07-11T17:57:09.382+02:00 INFO  [LookupTableService] Data Adapter cisco-mnemonics/5964ec0f8773ef433de0754c [@3c573f3] TERMINATED, was STOPPING
2017-07-11T17:57:39.380+02:00 ERROR [CSVFileDataAdapter] Couldn't parse CSV file /usr/share/graylog-server/lookups/csv/ciscoMnemomics.csv (settings separator=<;> quotechar=<"> key_column=<key> value_column=<value>)
java.lang.ArrayIndexOutOfBoundsException: 1
        at org.graylog2.lookup.adapters.CSVFileDataAdapter.parseCSVFile(CSVFileDataAdapter.java:153) ~[graylog.jar:?]
        at org.graylog2.lookup.adapters.CSVFileDataAdapter.doRefresh(CSVFileDataAdapter.java:108) ~[graylog.jar:?]
        at org.graylog2.plugin.lookup.LookupDataAdapter.refresh(LookupDataAdapter.java:89) ~[graylog.jar:?]
        at org.graylog2.lookup.LookupDataAdapterRefreshService.lambda$schedule$0(LookupDataAdapterRefreshService.java:145) ~[graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_121]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_121]
2017-07-11T17:58:09.380+02:00 ERROR [LookupDataAdapter] Couldn't refresh data adapter <cisco-mnemonics/5964ec0f8773ef433de0754c/@5cbf248b>
java.lang.NullPointerException: null
        at org.graylog2.lookup.adapters.CSVFileDataAdapter.doRefresh(CSVFileDataAdapter.java:101) ~[graylog.jar:?]
        at org.graylog2.plugin.lookup.LookupDataAdapter.refresh(LookupDataAdapter.java:89) ~[graylog.jar:?]
        at org.graylog2.lookup.LookupDataAdapterRefreshService.lambda$schedule$0(LookupDataAdapterRefreshService.java:145) ~[graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_121]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_121]
2017-07-11T17:58:39.380+02:00 ERROR [LookupDataAdapter] Couldn't refresh data adapter <cisco-mnemonics/5964ec0f8773ef433de0754c/@5cbf248b>
java.lang.NullPointerException: null
        at org.graylog2.lookup.adapters.CSVFileDataAdapter.doRefresh(CSVFileDataAdapter.java:101) ~[graylog.jar:?]
        at org.graylog2.plugin.lookup.LookupDataAdapter.refresh(LookupDataAdapter.java:89) ~[graylog.jar:?]
        at org.graylog2.lookup.LookupDataAdapterRefreshService.lambda$schedule$0(LookupDataAdapterRefreshService.java:145) ~[graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_121]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_121]

Looking at the Graylog Source Code, the first traced line (l. 153) is a closing } of an if-clause and the second trace (l. 101) starts a try { block. I guess the code changed between 2.3.0beta1 and now, right?

I don’t see any configuration error or malformed csv (tried with and without trailing ;), but the best thing is, the lookup still works!
"key";"value" "UPDOWN";"true" "FAN_LOW_RPM";"true" "MACFLAP_NOTIF";"true" "IPACCESSLOGP";"true" "ADDRESS_ASSIGN";"true" "CONFIG_I";"true"

Does anybody have the same issue, or is it just me? I’ll open a github issue if this isn’t a fault of my stupidness

Greetings - Phil

jochen · July 11, 2017, 4:36pm

Please create an issue for this bug at Issues · Graylog2/graylog2-server · GitHub if it still occurs on Graylog 2.3.0-rc.1.

That’s what tags are for

github.com

Graylog2/graylog2-server/blob/2.3.0-beta.1/graylog2-server/src/main/java/org/graylog2/lookup/adapters/CSVFileDataAdapter.java#L101


      
          }
          
          @Override
          public Duration refreshInterval() {
              return Duration.standardSeconds(Ints.saturatedCast(config.checkInterval()));
          }
          
          @Override
          protected void doRefresh(LookupCachePurge cachePurge) throws Exception {
              try {
                  final FileInfo.Change fileChanged = fileInfo.checkForChange();
                  if (!fileChanged.isChanged() && !getError().isPresent()) {
                      // Nothing to do, file did not change
                      return;
                  }
          
                  LOG.debug("CSV file {} has changed, updating data", config.path());
                  lookupRef.set(parseCSVFile());
                  cachePurge.purgeAll();
                  fileInfo = fileChanged.fileInfo();
                  clearError();

derPhlipsi · July 11, 2017, 4:40pm

Will do after the update tomorrow

Welp, I thought I selected it Of course my stupidness messes with me again

Greetings - Phil

derPhlipsi · July 12, 2017, 11:33am

Hey @jochen,

the release candidate solved the error btw.
This topic can be closed

Greetings - Phil

Topic		Replies	Views
CSV lookup table delete fails Graylog Central (peer support)	3	729	October 9, 2017
[RESOLVED] - Couldn't refresh data adapter- Lookup table issue Graylog Central (peer support)	7	2669	June 19, 2018
CSV lookup tables cannot be refreshed Graylog Central (peer support)	11	383	February 9, 2022
Graylog 2.4.3 Crash from Lookup Table Graylog Central (peer support)	8	2228	February 20, 2018
CSV lookup not working Graylog Central (peer support)	6	570	February 23, 2023

CSV Adapter cannot parse file, but Lookup still works

Related topics