Server.log Error for CSV Conversion

loggingone · October 3, 2021, 7:29am

Noticed that a periodic error is presented in the logs. Here is a sample:

ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.

Given an array of extractors and multiple ones qualify (by field name count) for matching the number of columns for “configured field names” - having a little trouble determining ?which? input extractor that this is failing on. Suspect its likely a result of a field match regex that needs to be adjusted.

Is there a means to enable greater log detail that would reveal the specific extractor? (Tried debug and trace - the 10-20 lines before and after didn’t appear to suggest an extractor name).

Any suggestions on how to determine which extractor and/or corresponding source $message would be greatly appreciated.

From a diagnostics perspective, having the calling routine’s name (the specific rule/extractor/…) [at a minimum] and $message (or [message id]/[index]) as part of the same log line would seem pragmatic. Otherwise, the error is simply too generic to efficiently find the problem.

Thanks!

gsmith · October 5, 2021, 1:19am

Hello,

The only debugging I know of that would be any help is the following.

if additional debugging is needed add -Djavax.net.debug=all
System/Logging (I noticed you have done this already)

Need to ask a couple question

How are you ingesting logs?
Can you explain what version/s you have in your environment ( Graylog, ES, MongoDb) ?
Is it possible to see the rest of the messages leading up to your Error?
Do you have a CSV data adapter?

loggingone · October 5, 2021, 5:41am

Ingestion of logs is via syslog format.

Install is via FreeBSD pkg:
Graylog 4.1.5
MongoDB 4.4.8
ElasticSearch 6.8.16

Here’s some output from the log (while in debug mode).

2021-10-03 02:25:11,097 INFO o.g.i.r.s.AbstractIndexCountBasedRetentionStrategy [scheduled-0] Running retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for indices <OBSCURED>
2021-10-03 02:25:11,163 INFO o.g.i.r.s.DeletionRetentionStrategy [scheduled-0] Finished index retention strategy [delete] for index <OBSCURED> in 64ms.
2021-10-03 02:28:20,746 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:28:20,748 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:28:20,749 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:28:20,749 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:28:20,750 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:28:20,753 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,829 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,829 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,829 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,830 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,830 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,831 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,831 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,831 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,832 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,832 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,833 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,833 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,834 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,834 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,834 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,835 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,835 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,835 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,836 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,836 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,837 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,837 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,838 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,838 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,839 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,839 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,840 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,840 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,840 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,841 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,841 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,842 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,842 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,843 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,843 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,844 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,844 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,844 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,845 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,845 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,846 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,846 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,847 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,847 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,848 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,849 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,849 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,850 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,851 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,851 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,852 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,852 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,852 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,853 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:30:10,168 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:30:10,168 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:30:10,168 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:30:10,170 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:30:10,170 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:31:09,465 INFO o.g.r.r.s.i.ExtractorsResource [http-worker-8] Updated extractor <OBSCURED> of type [regex] in input <OBSCURED>.

Looking through the logs it appears to be file management, CsvConverter errors and an occasional java.lang.NullPointerException which is apparently stemming from a pipeline rule checker.

at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser$RuleTypeChecker.exitMessageRef(PipelineRuleParser.java:842) ~[graylog.jar:?]

In terms of data adapters (lookup tables): Whois, GeoIP: City and OTX are in use. The service-port-numbers is present, but unused in anything.

Based on (non-debug log output), it suggests that the CsvConverter error stems from an input extractor that uses the CSV converter, of which there are an interesting number of them configured. For example, in one case there are ~7 that have the same number of columns but the field names differ based on the value in one (or more) of the fields (regex utilized for match). Suspect that there may be a periodic message that matches on the regex, but because the number of columns (via commas) differs - that its throwing the error. Given that $message itself isn’t included in the error output - trying to match up what message(s) is central to the issue. Really wish there was a flag/GUI config point to append to_string($message.message) to the end of error message, itself. That would help far more than even knowing the input extractor name. Then the entry could be readily located and backtracked.

Philosophically the data flow appears as:

source → Syslog/Port# (per source type) → Static variable → input extractor → initial move to Stream (stream rule keying on static variable) → pipelines/rules for all subsequent processing and message routing (if applicable) to end-state stream.

loggingone · October 14, 2021, 2:54pm

Any thoughts on how to hone in on the either the specific extractor and/or the line item that is failing?

gsmith · October 14, 2021, 10:09pm

Hello,

Only advice I can think of, if this was in my environment is to reduce the amount of sources sending logs? maybe you can find the source.
Basically start small and work your way up. If this was just recently, then I would look at what configuration were made prior to your issue. I don’t have a good visual on your environment so it hard to say exactly what or why this is happening. I can only go off of,

ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.

And

INFO o.g.r.r.s.i.ExtractorsResource [http-worker-8] Updated extractor of type [regex] in input .

Knowing what you have tried and the steps you made would be helpful.

Is this just one input or do you have multiple ones?
Have you tried using a different type of input? If so what was the out come?
How many extractors do you have per input?

loggingone · October 18, 2021, 5:25pm

In this specific case (specific input), numerous firewalls all converge on a specific UDP/<PORT#>. Error was likely present previously and simply didn’t “see” it as things appeared to have been working. It wasn’t until running into something that was obscure and ended up going to the log file - realizing that these errors were present. In total for this specific input - there are 16 extractors. All use a regex condition to target specific entries. The extractors are either Regex, Regex w/CSV or Grok. 7 of the 16 extractors process entries that have the same number of fields (comma separated), but values vary with a regex condition to match on each specific type.

gsmith · October 19, 2021, 1:11am

How many extractors do you have with

Reason I’m asking is that I haven’t seen that yet Regex w/CSV extractor or its under a different name.

Have you tested Raw/Plaintext UDP input?

loggingone · October 19, 2021, 5:47am

12 extractors utilize “regular expression” with the “Add Converter” with a selection of CSV to fields.

system · November 2, 2021, 5:47am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ERROR [CsvConverter] Different number of columns in CSV data (21) and configured field names (23). Discarding input Graylog Central (peer support)	4	258	November 26, 2023
How to troubleshoot "Different number of columns in CSV data" error Graylog Central (peer support) debuggingpl	3	633	August 6, 2022
Apply Extractor Rules based on data input Graylog Central (peer support)	3	1538	May 8, 2017
Trouble with CSV Converter Graylog Central (peer support)	1	579	June 3, 2020
Timestamp extractor error Graylog Central (peer support) pipeline-rules	2	979	December 31, 2021

Server.log Error for CSV Conversion

Related topics