Cannot parse msg - xm_kvp

Graylog Central (peer support)
1

Good afternoon
I have a problem when using the module … The bottom line is that some messages get into the stream, some don’t.
And I can’t figure out the reason why, I’m already giving up - I need your help!


<Extension FIXLogParserSecondTest>
    Module          xm_kvp
    KVPDelimiter    
    KVDelimiter     =
</Extension>

<Input TestReader>
	Module im_file
	File 'C:\Users\ishpileuski\Desktop\testReade.log'
	PollInterval 1
	SavePos	True
	ReadFromLast True
	Recursive False
	RenameCheck False
	<Exec>
        $FileName = file_name(); # Send file name with each message
	    $Component = 'TestReaderCOMP';
	    $GW_Name_FIX_Message = 'TestReader';
	    #FIXLogParser->parse_kvp($raw_event);
	                
        if $raw_event =~ /(.*?)\s(.*)/
        {
            $DateTime = $1;
            $Message = $2;
            log_info("Message delayed more than 2 minute" + $Message);
            FIXLogParserSecondTest->parse_kvp($Message);
            log_info("Message delayed more than 3 minute" + $Message);

            #log_info("OWN Parsed result: $parsed_result");
        }
	    $full_message = $Message;
	</Exec>

The above is my configuration.
Here is the log from the server:
2024-10-10 14:09:12 INFO Message delayed more than 2 minute8=FIX.4.2☺9=645☺35=8☺49=CQG_Gateway☺56=JUST2TRADE-2☺34=2663☺57=Just2Trade Online☺143=RU☺129=JA50148☺52=20241008-12:04:05.66
2☺20173=20241008-12:04:05.662025☺150=8☺20=0☺20026=FIX.41.2☺1=J2TFIX2
The same message without the last tag goes through and is displayed in the stream
2024-10-10 14:09:12 INFO Message delayed more than 3 minute8=FIX.4.2☺9=645☺35=8☺49=CQG_Gateway☺56=JUST2TRADE-2☺34=2663☺57=Just2Trade Online☺143=RU☺129=JA50148☺52=20241008-12:04:05.66
2☺20173=20241008-12:04:05.662025☺150=8☺20=0☺20026=FIX.41.2☺1=J2TFIX2

2

Hey @Ivan1,

Interesting use case, I’ve not seen Graylog used to ingest fix messages before.

Is the second example the log that does not parse ingest into Graylog? Are there any logs that point to indexing failures within the /var/log/graylog-server/server.log?

3

I can’t analyze graylog-server logs properly at the moment. He’s in my docker, which causes some difficulties.
tried to debug nxlog, but I didn’t see anything clear either. I do this in order to highlight each field in the fix message.picture->

image
image1190×551 27.3 KB

4

@Wine_Merchant I’ll try to look at the error logs now, but I’m unlikely to find anything there.
I’m completely stumped, I don’t understand why he parses some logs correctly, but the second ones don’t. Is there any approach to achieve the same result as in my screenshot?
Or how can I catch the error?

5

To narrow down the issue, I would try having these logs arrive on a raw input. See if the logs previously missing logs are now present, that way we know it would be a parsing issue on the input.

6

That’s exactly what I’m testing right now. With this use, the log appears
if $raw_event =~ /(.?)\s(.)/
{
$DateTime = $1;
$Message = $2;
}

    $full_message = $raw_event;

If I add “FIXLogParserSecondTest->parse_kvp($Message);”
That new message doesn’t even appear

7

So it’s failing to parse to key value pairs, do the logs generated by the running instance of NXlog reveal anything?

Why not just parse to key value pairs within a pipeline/rule in Graylog?

8
  1. there are no errors, if I turn on DEBUG, there is nothing either
  2. You mean like this? parse_kvp($Message);
9

My first step would be to output the data local as json output. Then you can see if all fields are created as expected.
If this is the case then I would continue searching in Graylog.
With something like this:

<Extension json>
	Module  xm_json
</Extension>
<Output test>
	Module om_file
	File 'C:\Users\ishpileuski\Desktop\testOutput.json'
	<Exec>
		to_json();
	</Exec>
</Output>
<Route Test>
	Path TestReaderRoute => test
10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.