I am trying to index multiline input into graylog
and following below steps,
In my collector , configure beats input I enabled Enable Multiline option and in Start pattern of a multiline message I mentioned my inputs start with Mon so I give [‘^Mon’]
In this case its taking only first line and ignoring remaining lines .
Multiline pattern is negated:
If I enable Multiline pattern is negated ,its taking all lines in to single message .
please help me to resolve the multiline index.
Input:
Mon Dec 11 09:26:37 2017 777777 : : The selected Change Requester is not valid. Use the Return function on the Last Name, First Name, or Phone Number fields to retrieve the Requester’s information. (ERROR 77777)
Mon Dec 11 09:26:37 2017 888888 : : The Assignment Information is not valid.
Please use the menus provided on the ’ Company’, ’ Organization’ and ’ Group’ or the type ahead return function on the ’ Group’ field to select this information. (ERROR 88888)
Hi Shelin,
Below lines are my inputs
First one(input1) is in single line next one(input2) is multiline format.
input1:
Mon Dec 11 09:26:37 2017 777777 : : The selected Change Requester is not valid. Use the Return function on the Last Name, First Name, or Phone Number fields to retrieve the Requester’s information. (ERROR 77777)
input2:
Mon Dec 11 09:26:37 2017 888888 : : The Assignment Information is not valid.
Please use the menus provided on the ’ Company’, ’ Organization’ and ’ Group’ or the type ahead return function on the ’ Group’ field to select this information. (ERROR 88888)
Hi,
I used to use the same method to collect multi-line logs that start with the date “XXXX-XX-XX”, the only difference is regular expression. My expression is ^[0-9]{4}-[0-9]{2}-[0-9]{2}, it can work.
Sorry, I do not know why this method does not work for you.
You can reference those documents:
Here is my complete settings to index multiline input. Please check.
My input:
Wed Dec 13 06:10:22 2017 390603 : : The Assignment Information is not valid.
Please use the menus provided on the ‘Assignee Company’, ‘Assignee Organization’ and ‘Assignee Group’ or the type ahead return function on the ‘Assignee Group’ field to select this information. (ARERR 10000)
My grok:
(?%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})\s*(%{NUMBER:message_id})\s(?<error_msg>(?m:.*))
Screenshot from online grok constructor:
From the graylog output we can see that the second line of the log is not included under error_msg.
But it is taking only the data from the first line. From our input log:
Wed Dec 13 06:10:22 2017 390603 : : The Assignment Information is not valid.
Please use the menus provided on the ‘Assignee Company’, ‘Assignee Organization’ and ‘Assignee Group’ or the type ahead return function on the ‘Assignee Group’ field to select this information. (ARERR 10000)
Ideally it should show, error_msg :
: : The Assignment Information is not valid.
Please use the menus provided on the ‘Assignee Company’, ‘Assignee Organization’ and ‘Assignee Group’ or the type ahead return function on the ‘Assignee Group’ field to select this information. (ARERR 10000)
But its taking only the first line:
: : The Assignment Information is not valid.
Graylogs GROK Pattern are not able to work with Multiline messages!
You had never before written that you do a GROK Extractor on that message …
You should work with a processing pipeline a create one string from the message before you process that message with GROK - or use some other extractors.
