thanks for the reply - as i mentioned the Date format can be extracted from different GROK patterns -it works - have changed that also.
I have also having issue using content pack old to new
thanks for the reply - as i mentioned the Date format can be extracted from different GROK patterns -it works - have changed that also.
I have also having issue using content pack old to new
Yes, I was thinking about that. One thing that I know is, newer content packs have a version check and the old github thing does not have that line.
2016 is long ago, I have no idea about what changed in terms of required fields to load a content pack. Maybe you can create a new empty pack and check what you see.
Insert the old pack into the new pack, change the id and upload it.
I never tried to upload such an old pack. It might work, but maybe even the whole structure changed.
thank yoy, may be i am not expert of creating content pack - I will read and try.
i able to make some grok pattern progress. as below image :
but if the logs coming difference format, how does he different pattern to match and organise the logs.
i see json format for the windows logs stream lined, how I can make same as it is ?
I was looking into your GROK pattern and I saw some mistakes.
I do not know the log format, but the GROK for this log looks like this:
… %{WORD:process} %{NONNEGINT:process_number}…
or
…Process %{NONNEGINT:process_number}, Nbr %{IP:nbr_ip}…
depending on what you want to parse out.
Greedydata is easy to use, but it is the most expensive choice in terms of CPU.
But WORD did not work at all.
as per I know compared to Json, Grok is more overhead on CPU I guess, if the server is processing millions of logs (I will dig into this later, when it comes to performance, for now, I am looking to streamline the data)
If you look into json because you are getting the logs in json, Graylog offers a json parser. In that case, you will get the fields nicely. I never worked with that, so I cannot give you a working example.
parse_json(value, [depth]) : JsonNode Parses a string as a JSON tree
If you want to continue down the GROK path, you can use a GROK debugger online like here to see how your GROK does with a log message. In the case where you are using WORD for the log_type, %OSPF-5-ADJCHG:
is not a WORD because of the % and the dashes… If you add in the parts of your GROK to the debugger you can see which ones work and which you need to change…
Thank you let me try that - with this example. i take that CPU less impact as you suggest not to use greedydata
thank i do come across this debugger, its very confusing for me to try that tool
1 I type message
2 pattern here
nothing happens.
Don’t put in the entire pattern. Add in each section for each capture, when you add a capture for a word that doesn’t work the entire thing breaks and you get NUL. Fix the one you just added.
Greedydata is some kind of dangerous in therms of CPU usage. You easily could build a Regexploit. I throw this link already in a few times here and still like it a lot
i tried each line, but somehow not got what I expected, maybe I am trying wrong I guess.
i try to manage using grok pattern in Graylog testing and thank you @tmacgbay and @StefanAustin for helping (now I have some idea how the data need to strip and do in a better way in grok pattern) - somehow still learning.
The problem I see here - instead grok like to use the pipeline as suggested to save the CPU here.
when I add below line in rule its not taking not sure why
<%{POSINT:syslog_priority}>%{POSINT:syslog_version}: \*%{CISCOTIMESTAMP:timestamp}: %{NOTSPACE:event_source}: Line protocol on Interface %{NOTSPACE:network_interface}, changed state to %{WORD:status1}
especially due to
\*%{CISCOTIMESTAMP:timestamp}: (if I remove \* that rule take it and accept)
also this is the best way to create a rule (follows some document people doing this way - this works, but any draw back here ?)
rule “Cisco pipeline testing”
when
has_field(“message”)
then
let message_field = to_string($message.message);
let cs1 = grok(pattern: "", value: message_field,only_named_captures: true);
let cs2 = grok(pattern: "", value: message_field,only_named_captures: true);
let cs3 = grok(pattern: "", value: message_field,only_named_captures: true);
set_fields(cs1);
set_fields(cs2);
set_fields(cs3);
end
Rather than have every message GROK’ed three times (not efficient) - set up a separate rule that uses the when/then to figure out the right message that it will GROK. You likely can’t create a GROK that covers all messages, instead you may have a rule where the when
finds a message of a particular type and the GROK you use it in handles it properly.
If some part of the message is consistent you could have a rule separate out that (With GROK , regex, etc…) for all messages and then have a separate stage that figures out which message it is based on what the previous stage broke out. For instance if you figure out the EventID and put it into a field in the initial stage, then in a subsequent stage you use GROK to pull out the description with rules that each have a different when/then that handles the EventID and the GROK for that description… not a great example but you get the picture I hope…
with you testing you would test against the example like this:
<%{POSINT:syslog_priority}>
%{POSINT:syslog_priority}>%{POSINT:syslog_version}:
<%{POSINT:syslog_priority}>%{POSINT:syslog_version}: \*%{CISCOTIMESTAMP:timestamp}:
Keep in mind that each regex is shorthand for what really is a regex pattern…
Be sure to post the solution that works for future searchers!!!
thank you for the suggestion, unfortunatly i am not that expert as I mentioned, but getting there learning few tips from the forum here.
take example as below messages I am parsing from cisco device :
<45>52: Jan 1 00:58:43.864: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to down
<45>54: Jan 1 01:00:19.558: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to up
<43>53: Jan 1 01:00:18.558: %LINK-3-UPDOWN: Interface Ethernet0/1, changed state to up
<45>51: Jan 1 00:58:42.864: %LINK-5-CHANGED: Interface Ethernet0/1, changed state to administratively down
Only first 3 pattern only matches as below :
<%{POSINT:syslog_priority}>%{POSINT:syslog_version}: \*%{CISCOTIMESTAMP:timestamp}:
. For instance, if you figure out the EventID and put it into a field in the initial stage, then in a subsequent stage you use GROK to pull out the description with rules that each have a different when/then that handles the EventID and the GROK for that description… not a great example but you get the picture I hope…
Some how the above suggestion made not able to get in to my brain appreciated if you can give some examples based on the messages I have posted.
Other note :
especially due to
\*%{CISCOTIMESTAMP:timestamp}: (if I remove \* that rule take it and accept)
I could able to fix this by adding grok pattern and use in the rule.
If that is your plan, you really need to be super efficient. I see this as working out in stages…
Pipeline rules Stage One - break out the following into fields:
(allowing the message to pass to stage two if at least one rule matches)
syslog_priority
syslog_version
timestamp_c
(don’t get it confused with the Graylog Timestamp)log_type
Then in Stage Two you can create a series of rules that only act on the message if it’s relevant so here are examples:
rule "Cisco - LineProto"
when
to_string($message.log_type) == "LINEPROTO-5-UPDOWN"
then
<script that GROK's the data after LINEPROTO-5-UPDOWN into the correct fields>
end
rule "Cisco - Link-Updown"
when
to_string($message.log_type) == "LINK-3-UPDOWN"
then
<script that GROK's the data after LINK-3-UPDOWN into the correct fields>
end
rule "Cisco - Link-Changed"
when
to_string($message.log_type) == "LINK-5-CHANGED"
then
<script that GROK's the data after LINK-5-CHANGED into the correct fields>
end
Thank you and much appreciated your value time and guiding me right direction.
One last one before I start building rule and test.
all above mentioned can be in one rule
or I need to create each one as separate rules.
Like I have below
Stage1 going to be
example :
log stream line rule
Stage2 going to be
example :
log pasrser line rule
or you suggesting - stage 2 ( each one go different Rule ?)
Cisco - LineProto
Cisco - Link-Updown
Cisco - Link-Changed
if more rule - do we see any delay ?
You want separate rules so that you are not trying to parse messages that won’t match. it is better to have several quick deciding rules than to have each message fail several parsing tries.
Thank you, i will test that separate rule.
Also of note, rules within a single stage generally run in “parallel” as much as possible. So to take action on the results of a rule, the next rule needs to be in a following stage… like what we are doing here.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.