Apologies if this question was addressed before - I do search only 2 threads were found, but that not met my requirement and solve the issue.
Let me explain my problem, I am trying to get Cisco switch logs to graylogs ad try to format that logs using GROK, it did not worked.
Example Log :
<45>62: *Dec 25 17:21:14.062: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.9.100 on GigabitEthernet0/1 from FULL to DOWN, Neighbor Down: Interface down or detached
GROK :
<%{POSINT:syslog_priority}>%{POSINT:syslog_version}: %{DATESTAMP_RFC2822:timestamp}: %{WORD:log_type}: %{WORD:process}, %{WORD:nbr} %{IP:ip_address} on %{WORD:interface} from %{WORD:state_from} to %{WORD:state_to}, %{WORD:description}: %{GREEDYDATA:message}
But I get error all time, tried different pattern combination nothing worked ( all grok patterns available and checked)
Then I come across below content pack, download and try to install not have error ( I come to know this build on old Graylog,as per the document this is no Longer works) - I am not a great programmer to write some from scratch. as a network admin try to use what available in market and make any small changes to meet the requirement.
again apologies any ignorance already addressed, if any one can point me right direction that will be very helpful.
Yes, I was thinking about that. One thing that I know is, newer content packs have a version check and the old github thing does not have that line.
2016 is long ago, I have no idea about what changed in terms of required fields to load a content pack. Maybe you can create a new empty pack and check what you see.
Insert the old pack into the new pack, change the id and upload it.
I never tried to upload such an old pack. It might work, but maybe even the whole structure changed.
as per I know compared to Json, Grok is more overhead on CPU I guess, if the server is processing millions of logs (I will dig into this later, when it comes to performance, for now, I am looking to streamline the data)
This is a working grok pattern for the provided log.
If you look into json because you are getting the logs in json, Graylog offers a json parser. In that case, you will get the fields nicely. I never worked with that, so I cannot give you a working example.
parse_json(value, [depth]) : JsonNode Parses a string as a JSON tree
If you want to continue down the GROK path, you can use a GROK debugger online like here to see how your GROK does with a log message. In the case where you are using WORD for the log_type, %OSPF-5-ADJCHG: is not a WORD because of the % and the dashes… If you add in the parts of your GROK to the debugger you can see which ones work and which you need to change…
Don’t put in the entire pattern. Add in each section for each capture, when you add a capture for a word that doesn’t work the entire thing breaks and you get NUL. Fix the one you just added.
Greedydata is some kind of dangerous in therms of CPU usage. You easily could build a Regexploit. I throw this link already in a few times here and still like it a lot
i tried each line, but somehow not got what I expected, maybe I am trying wrong I guess.
i try to manage using grok pattern in Graylog testing and thank you @tmacgbay and @StefanAustin for helping (now I have some idea how the data need to strip and do in a better way in grok pattern) - somehow still learning.
The problem I see here - instead grok like to use the pipeline as suggested to save the CPU here.
when I add below line in rule its not taking not sure why
<%{POSINT:syslog_priority}>%{POSINT:syslog_version}: \*%{CISCOTIMESTAMP:timestamp}: %{NOTSPACE:event_source}: Line protocol on Interface %{NOTSPACE:network_interface}, changed state to %{WORD:status1}
especially due to
\*%{CISCOTIMESTAMP:timestamp}: (if I remove \* that rule take it and accept)
also this is the best way to create a rule (follows some document people doing this way - this works, but any draw back here ?)
rule “Cisco pipeline testing”
when
has_field(“message”)
then
let message_field = to_string($message.message);
let cs1 = grok(pattern: "", value: message_field,only_named_captures: true);
let cs2 = grok(pattern: "", value: message_field,only_named_captures: true);
let cs3 = grok(pattern: "", value: message_field,only_named_captures: true);
set_fields(cs1);
set_fields(cs2);
set_fields(cs3);
Rather than have every message GROK’ed three times (not efficient) - set up a separate rule that uses the when/then to figure out the right message that it will GROK. You likely can’t create a GROK that covers all messages, instead you may have a rule where the when finds a message of a particular type and the GROK you use it in handles it properly.
If some part of the message is consistent you could have a rule separate out that (With GROK , regex, etc…) for all messages and then have a separate stage that figures out which message it is based on what the previous stage broke out. For instance if you figure out the EventID and put it into a field in the initial stage, then in a subsequent stage you use GROK to pull out the description with rules that each have a different when/then that handles the EventID and the GROK for that description… not a great example but you get the picture I hope…
with you testing you would test against the example like this:
<%{POSINT:syslog_priority}>
– if it works fine then add to it…
<%{POSINT:syslog_priority}>%{POSINT:syslog_version}:
– if that works then add to it…
thank you for the suggestion, unfortunatly i am not that expert as I mentioned, but getting there learning few tips from the forum here.
take example as below messages I am parsing from cisco device :
<45>52: Jan 1 00:58:43.864: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to down
<45>54: Jan 1 01:00:19.558: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to up
<43>53: Jan 1 01:00:18.558: %LINK-3-UPDOWN: Interface Ethernet0/1, changed state to up
<45>51: Jan 1 00:58:42.864: %LINK-5-CHANGED: Interface Ethernet0/1, changed state to administratively down
Only first 3 pattern only matches as below :
<%{POSINT:syslog_priority}>%{POSINT:syslog_version}: \*%{CISCOTIMESTAMP:timestamp}:
after that content changed, so I need to make a pattern to match next, so I need to have different match pattern here for each message, (this is some how works in test environment ) - I am suspecting if I go production environment I expected 100000 or + logs coming in ( even though I have dedicated hardware, I am sure there is LAG when parse takes place.
. For instance, if you figure out the EventID and put it into a field in the initial stage, then in a subsequent stage you use GROK to pull out the description with rules that each have a different when/then that handles the EventID and the GROK for that description… not a great example but you get the picture I hope…
Some how the above suggestion made not able to get in to my brain appreciated if you can give some examples based on the messages I have posted.
Other note :
especially due to
\*%{CISCOTIMESTAMP:timestamp}: (if I remove \* that rule take it and accept)
I could able to fix this by adding grok pattern and use in the rule.
If that is your plan, you really need to be super efficient. I see this as working out in stages…
Pipeline rules Stage One - break out the following into fields:
(allowing the message to pass to stage two if at least one rule matches)
syslog_priority
syslog_version
timestamp_c (don’t get it confused with the Graylog Timestamp)
log_type
Then in Stage Two you can create a series of rules that only act on the message if it’s relevant so here are examples:
rule "Cisco - LineProto"
when
to_string($message.log_type) == "LINEPROTO-5-UPDOWN"
then
<script that GROK's the data after LINEPROTO-5-UPDOWN into the correct fields>
end
rule "Cisco - Link-Updown"
when
to_string($message.log_type) == "LINK-3-UPDOWN"
then
<script that GROK's the data after LINK-3-UPDOWN into the correct fields>
end
rule "Cisco - Link-Changed"
when
to_string($message.log_type) == "LINK-5-CHANGED"
then
<script that GROK's the data after LINK-5-CHANGED into the correct fields>
end
Thank you and much appreciated your value time and guiding me right direction.
One last one before I start building rule and test.
all above mentioned can be in one rule
or I need to create each one as separate rules.
Like I have below
Stage1 going to be
example :
log stream line rule
Stage2 going to be
example :
log pasrser line rule
or you suggesting - stage 2 ( each one go different Rule ?)
You want separate rules so that you are not trying to parse messages that won’t match. it is better to have several quick deciding rules than to have each message fail several parsing tries.
