JOIN GRAYLOG FOR OUR

ANNUAL CONFERENCE

Thursday,
October 21, 2021
10am-5pm CT
REGISTER NOW

Server.log Error for CSV Conversion

Noticed that a periodic error is presented in the logs. Here is a sample:

ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.

Given an array of extractors and multiple ones qualify (by field name count) for matching the number of columns for “configured field names” - having a little trouble determining ?which? input extractor that this is failing on. Suspect its likely a result of a field match regex that needs to be adjusted.

Is there a means to enable greater log detail that would reveal the specific extractor? (Tried debug and trace - the 10-20 lines before and after didn’t appear to suggest an extractor name).

Any suggestions on how to determine which extractor and/or corresponding source $message would be greatly appreciated.

From a diagnostics perspective, having the calling routine’s name (the specific rule/extractor/…) [at a minimum] and $message (or [message id]/[index]) as part of the same log line would seem pragmatic. Otherwise, the error is simply too generic to efficiently find the problem.

Thanks!

Hello,

The only debugging I know of that would be any help is the following.

  • if additional debugging is needed add -Djavax.net.debug=all
  • System/Logging (I noticed you have done this already)

Need to ask a couple question

  • How are you ingesting logs?
  • Can you explain what version/s you have in your environment ( Graylog, ES, MongoDb) ?
  • Is it possible to see the rest of the messages leading up to your Error?
  • Do you have a CSV data adapter?

Ingestion of logs is via syslog format.

Install is via FreeBSD pkg:
Graylog 4.1.5
MongoDB 4.4.8
ElasticSearch 6.8.16

Here’s some output from the log (while in debug mode).

2021-10-03 02:25:11,097 INFO o.g.i.r.s.AbstractIndexCountBasedRetentionStrategy [scheduled-0] Running retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for indices <OBSCURED>
2021-10-03 02:25:11,163 INFO o.g.i.r.s.DeletionRetentionStrategy [scheduled-0] Finished index retention strategy [delete] for index <OBSCURED> in 64ms.
2021-10-03 02:28:20,746 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:28:20,748 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:28:20,749 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:28:20,749 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:28:20,750 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:28:20,753 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,829 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,829 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,829 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,830 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,830 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,831 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,831 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,831 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,832 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,832 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,833 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,833 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,834 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,834 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,834 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,835 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,835 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,835 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,836 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,836 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,837 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,837 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,838 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,838 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,839 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,839 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,840 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,840 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,840 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,841 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,841 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,842 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,842 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,843 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,843 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,844 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,844 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,844 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,845 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,845 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,846 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,846 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,847 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,847 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,848 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,849 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,849 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,850 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,851 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,851 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,852 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,852 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,852 ERROR o.g.i.c.CsvConverter [processbufferprocessor-1] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:29:49,853 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:30:10,168 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:30:10,168 ERROR o.g.i.c.CsvConverter [processbufferprocessor-3] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:30:10,168 ERROR o.g.i.c.CsvConverter [processbufferprocessor-4] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:30:10,170 ERROR o.g.i.c.CsvConverter [processbufferprocessor-0] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:30:10,170 ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.
2021-10-03 02:31:09,465 INFO o.g.r.r.s.i.ExtractorsResource [http-worker-8] Updated extractor <OBSCURED> of type [regex] in input <OBSCURED>.

Looking through the logs it appears to be file management, CsvConverter errors and an occasional java.lang.NullPointerException which is apparently stemming from a pipeline rule checker.

at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser$RuleTypeChecker.exitMessageRef(PipelineRuleParser.java:842) ~[graylog.jar:?]

In terms of data adapters (lookup tables): Whois, GeoIP: City and OTX are in use. The service-port-numbers is present, but unused in anything.

Based on (non-debug log output), it suggests that the CsvConverter error stems from an input extractor that uses the CSV converter, of which there are an interesting number of them configured. For example, in one case there are ~7 that have the same number of columns but the field names differ based on the value in one (or more) of the fields (regex utilized for match). Suspect that there may be a periodic message that matches on the regex, but because the number of columns (via commas) differs - that its throwing the error. Given that $message itself isn’t included in the error output - trying to match up what message(s) is central to the issue. Really wish there was a flag/GUI config point to append to_string($message.message) to the end of error message, itself. That would help far more than even knowing the input extractor name. Then the entry could be readily located and backtracked.

Philosophically the data flow appears as:

source → Syslog/Port# (per source type) → Static variable → input extractor → initial move to Stream (stream rule keying on static variable) → pipelines/rules for all subsequent processing and message routing (if applicable) to end-state stream.

Any thoughts on how to hone in on the either the specific extractor and/or the line item that is failing?

Hello,

Only advice I can think of, if this was in my environment is to reduce the amount of sources sending logs? maybe you can find the source.
Basically start small and work your way up. If this was just recently, then I would look at what configuration were made prior to your issue. I don’t have a good visual on your environment so it hard to say exactly what or why this is happening. I can only go off of,

ERROR o.g.i.c.CsvConverter [processbufferprocessor-2] Different number of columns in CSV data (24) and configured field names (23). Discarding input.

And

INFO o.g.r.r.s.i.ExtractorsResource [http-worker-8] Updated extractor of type [regex] in input .

Knowing what you have tried and the steps you made would be helpful.

  • Is this just one input or do you have multiple ones?
  • Have you tried using a different type of input? If so what was the out come?
  • How many extractors do you have per input?