IPFIX input Multiple entries with same key

Graylog Central (peer support)
,
1

I’m exporting IPFIX (netflow v10) flows from pfSense to Graylog using the IPFIX UDP input.
I created the .json with the IPFIX field definitions found here iana and here RFC5102 and it is working.

However, when I set pfSense to send bidirectional flows, I get this error in my graylog server.log

Obs: Included the ID 239 field, biflowDirection, data_type": “unsigned8” as referenced in RFC5103 in my .json file.

Error: Multiple entries with same key: octetDeltaCount=2936 and octetDeltaCount=5016

java.lang.IllegalArgumentException: Multiple entries with same key: octetDeltaCount=2936 and octetDeltaCount=5016

My environment:
pfSense 23.05.1 amd64 (appliance SG-4100)
softflowd version: 1.2.6_1 | softflowd-1.0.0_1

Raspberry PI 4b 4GB @ Ubuntu 20.04.6 LTS
Graylog 5.1.5+993cd0f on rpi4.home.arpa (Eclipse Adoptium 17.0.8 on Linux 5.4.0-1093-raspi)
Mongod: 6.0.8
Elasticsearch: 7.10.2

Logs that only happen when bidirectional flow is enabled in pfSense:

2023-09-07T16:35:01.800-03:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=a338ebe0-4db5-11ee-bd4e-dca632a54719, messageQueueId=17413083, codec=ipfix, payloadSize=485, timestamp=2023-09-07T19:35:01.790Z, seqenceNr=2, remoteAddress=/192.168.255.249:23692}
java.lang.IllegalArgumentException: Multiple entries with same key: octetDeltaCount=2581 and octetDeltaCount=6224
	at com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:376) ~[graylog.jar:?]
	at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:370) ~[graylog.jar:?]
	at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:153) ~[graylog.jar:?]
	at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:115) ~[graylog.jar:?]
	at com.google.common.collect.ImmutableMap$Builder.buildOrThrow(ImmutableMap.java:574) ~[graylog.jar:?]
	at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:538) ~[graylog.jar:?]
	at org.graylog.integrations.ipfix.IpfixParser.parseDataSet(IpfixParser.java:580) ~[?:?]
	at org.graylog.integrations.ipfix.codecs.IpfixCodec.lambda$decodeMessages$3(IpfixCodec.java:206) ~[?:?]
	at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
	at org.graylog.integrations.ipfix.codecs.IpfixCodec.decodeMessages(IpfixCodec.java:212) ~[?:?]
	at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:154) ~[graylog.jar:?]
	at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:98) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
	at java.lang.Thread.run(Unknown Source) [?:?]

Tried to force flows to expire, performed packet captures to search for the field octetDeltaCount, included field ID 239 in my .json… But I’m unable to diagnose it further…

I wonder if Graylog IPFIX UDP input is compatible with bidirectional flows or not.
And if is, if the community somehow can help me.

Thanks in advance, any help is much appreciated.

Edit:

I replaced OctetDeltaCount for inOctetDeltaCount and PacketDeltaCount for inPacketDeltaCount.
The error mentioned above disappeared, but now I’m getting a very similar error, but for ipClassOfService

Multiple entries with same key: ipClassOfService=0 and ipClassOfService=0

As we can see in the below packet capture, pfSense is sending both values (TOS and reverse TOS).
It seems that Graylog is not inserting the reverse of ipClassOfService (5) in the field postIpClassOfService (55) PEN.

image

2

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.