Daily Challenge: IPFIX Input issue

Hello,

The version of Graylog I am using is 4.01 and I’d like to collect logs from IPFIX.

I’ve followed the below article:
https://docs.graylog.org/en/latest/pages/integrations/inputs/ipfix_input.html

Right after that Graylog started processing logs but they didn’t show up in dashboard. I found there were some errors in logs while trying to parse data:
org.graylog.integrations.ipfix.IpfixException: Missing information element definitions for private enterprise number 29305
org.graylog.integrations.ipfix.IpfixException: Missing information element definitions for private enterprise number 45346

Then I included json files for translating incoming logs for both velocloud (45346 ) and ipfix (29305):

45346: VMware Knowledge Base
29305: IP Flow Information Export (IPFIX) Entities

At this moment I am encountering the following error in graylog.log:

2021-05-31T07:25:01.846Z ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=4ff8a630-c1e1-11eb-a4f5-005056919081, journalOffset=44413432, codec=ipfix, payloadSize=1817, timestamp=2021-05-31T07:25:01.843Z, remoteAddress=/172.23.9.132:54112} on input <60af6f3b3f1dd3671d48e2fc>.
2021-05-31T07:25:01.846Z ERROR [DecodingProcessor] Error processing message RawMessage{id=4ff8a630-c1e1-11eb-a4f5-005056919081, journalOffset=44413432, codec=ipfix, payloadSize=1817, timestamp=2021-05-31T07:25:01.843Z, remoteAddress=/172.23.9.132:54112}
java.lang.NullPointerException: null
at org.graylog.integrations.ipfix.IpfixParser.parseDataSet(IpfixParser.java:338) ~[?:?]
at org.graylog.integrations.ipfix.codecs.IpfixCodec.lambda$decodeMessages$3(IpfixCodec.java:206) ~[?:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_282]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) ~[?:1.8.0_282]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_282]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_282]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_282]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_282]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_282]
at org.graylog.integrations.ipfix.codecs.IpfixCodec.decodeMessages(IpfixCodec.java:212) ~[?:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:147) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]

Any ideas ?

Hello, Aliaksandr,

It looks like your question has remained unanswered for a few days. Let’s move it to “Daily Challenges” where it’s more likely to get noticed in the community.

In the meanwhile, you may want to check to ensure that you’re configured to send data to the correct port

@aazherelyeu
Hello,

I have found that Graylog Version 4.0.7 has a built-in INPUT for IPFIX UDP as shown below. Have you tried using this?

hope that helps

1 Like

Hello,
Yes, sure. I’ve installed graylog-integrations-plugins according to IPFIX Input — Graylog 4.0.0 documentation

I’ve also added my own json files as described in my first message.

Hello,

Sorry for that late responce. I dont have much knoweldge of this type of IPFIX input. What I would like to do is establish where the problem is at (i.e. Dashboard, configuration issue, firewall, permission issue, etc…). I need to ask some question to understand what is going on.

Do you see log/s from the input when you click “Show received messages”?

If you do not see messages on that input, did you try setting the input to default, meaning without
a definition configured? If so, when adding your ipfix_definition_path configuration have you use just one definition see if that helps?

Hello,

Messages are coming in IPFIX input:

But they are not shown up when I click on “Show received messages” button:

When the Input is changed to without any definitions paths I get the following error in graylog logs:
Missing information element definitions for private enterprise number 45346
Missing information element definitions for private enterprise number 29305
which requires file definitions for both velocloud and ipfix.

When both definition files are in place I get
2021-06-09T07:31:57.281Z ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=c54e98d0-c8f4-11eb-a11d-005056919081, journalOffset=54629489, codec=ipfix, payloadSize=1742, timestamp=2021-06-09T07:31:57.277Z, remoteAddress=/172.23.9.132:46012} on input <60af6f3b3f1dd3671d48e2fc>.
2021-06-09T07:31:57.281Z ERROR [DecodingProcessor] Error processing message RawMessage{id=c54e98d0-c8f4-11eb-a11d-005056919081, journalOffset=54629489, codec=ipfix, payloadSize=1742, timestamp=2021-06-09T07:31:57.277Z, remoteAddress=/172.23.9.132:46012}
java.lang.NullPointerException: null

I tried to use accept messages on RAW/Plaintext UDP" Input. Messages were there but they looked as follows:

Hello,

Thank you for the added information.

I believe this is normal because GL is not using the right INPUT codec (i.e. RAW/PLaintext).
Is this the only INPUT you have thats having problems?

Have you checked the Date/Time on Graylog and the device thats sending logs?
System/Overview --> Time configuration

IPFIX is very similar to Netflow have you tried uing that type of INPUT instead?
I have a Netflow INPUT in my lab below is an example:


But as you can see I do not have definitions paths configured.

1 Like

Hello,

This is the only one input where I experience issues. All others work fine. Time is correct on both sides.

I’ve also tried Netflow Input as well but it’s not compatible with IPFIX 10 version:
2021-06-10T07:04:30.445Z ERROR [NetFlowCodec] Error parsing NetFlow packet <1a20f680-c9ba-11eb-a11d-005056919081> received from <172.23.9.132:51639>
org.graylog.plugins.netflow.flows.InvalidFlowVersionException: Invalid NetFlow version 10

Hello,
Maybe I overlooked the past post, but have you tried just one definition?
I know you said you put two definition for velocloud and ipfix. Are these two different devices or one?
If these are two different devices try testing just one, like IPFIX. Trying to eliminate all possibilities of why this is not working for you.
At this point I’m only offering suggestion. I was looking around for a device that sends IPFIX logs here to test those configurations, but I could not find one.
Sorry I can’t be more help

1 Like

Hello,

This is the one device but somehow keeps sending packets including both velocloud and ipfix. I’ve tried to use the only one definition for IPFIX, no luck. Thank you for your help anyway.