multiline log index


(JK) #1

I am trying to index multiline input into graylog
and following below steps,

In my collector , configure beats input I enabled Enable Multiline option and in Start pattern of a multiline message I mentioned my inputs start with Mon so I give [’^Mon’]

In this case its taking only first line and ignoring remaining lines .

Multiline pattern is negated:

If I enable Multiline pattern is negated ,its taking all lines in to single message .

please help me to resolve the multiline index.

Input:

Mon Dec 11 09:26:37 2017 777777 : : The selected Change Requester is not valid. Use the Return function on the Last Name, First Name, or Phone Number fields to retrieve the Requester’s information. (ERROR 77777)

Mon Dec 11 09:26:37 2017 888888 : : The Assignment Information is not valid.
Please use the menus provided on the ’ Company’, ’ Organization’ and ’ Group’ or the type ahead return function on the ’ Group’ field to select this information. (ERROR 88888)

Thanks,
K.Jayakumar


#2

If I understand correctly, you want to collect each multiline message starts with Mon, you can try this Beats input configuration:


(JK) #3

Hi Shelin,

Thanks for your reply. I followed the same steps which you mentioned. but ,

> it took whole input as single message .

Please help me to resolve.

Thanks,
K.Jayakumar


#4

Hi,
Can you show the input message? Sorry, I’m not sure where the problem is. Maybe I need to look at the specific format.


(JK) #5

Hi Shelin,
Below lines are my inputs
First one(input1) is in single line next one(input2) is multiline format.

input1:
Mon Dec 11 09:26:37 2017 777777 : : The selected Change Requester is not valid. Use the Return function on the Last Name, First Name, or Phone Number fields to retrieve the Requester’s information. (ERROR 77777)

input2:
Mon Dec 11 09:26:37 2017 888888 : : The Assignment Information is not valid.
Please use the menus provided on the ’ Company’, ’ Organization’ and ’ Group’ or the type ahead return function on the ’ Group’ field to select this information. (ERROR 88888)

Thanks,
K.Jayakumar


#6

Hi,
I used to use the same method to collect multi-line logs that start with the date “XXXX-XX-XX”, the only difference is regular expression. My expression is ^[0-9]{4}-[0-9]{2}-[0-9]{2}, it can work.
Sorry, I do not know why this method does not work for you.
You can reference those documents:

I hope that will help you.


(JK) #7

Hey Shelin,
Thank you. :slight_smile:

Here is my complete settings to index multiline input. Please check.

My input:
Wed Dec 13 06:10:22 2017 390603 : : The Assignment Information is not valid.
Please use the menus provided on the ‘Assignee Company’, ‘Assignee Organization’ and ‘Assignee Group’ or the type ahead return function on the ‘Assignee Group’ field to select this information. (ARERR 10000)

My grok:
(?%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})\s*(%{NUMBER:message_id})\s(?<error_msg>(?m:.*))
Screenshot from online grok constructor:

output:


The output clearly shows that the logs are parsed properly.

Settings graylog:
1)Collector input settings: (done as you suggested earlier)
collector_input

2)output in graylog messages.

From the graylog output we can see that the second line of the log is not included under error_msg.
But it is taking only the data from the first line.
From our input log:
Wed Dec 13 06:10:22 2017 390603 : : The Assignment Information is not valid.
Please use the menus provided on the ‘Assignee Company’, ‘Assignee Organization’ and ‘Assignee Group’ or the type ahead return function on the ‘Assignee Group’ field to select this information. (ARERR 10000)

Ideally it should show, error_msg :
: : The Assignment Information is not valid.
Please use the menus provided on the ‘Assignee Company’, ‘Assignee Organization’ and ‘Assignee Group’ or the type ahead return function on the ‘Assignee Group’ field to select this information. (ARERR 10000)

But its taking only the first line:
: : The Assignment Information is not valid.

Please help me to resolve this .

Thanks in advance.

Regards,
Jayakumar


(Jan Doberstein) #8

Graylogs GROK Pattern are not able to work with Multiline messages!

You had never before written that you do a GROK Extractor on that message …

You should work with a processing pipeline a create one string from the message before you process that message with GROK - or use some other extractors.


(system) #9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.