Old content pack import failed in graylog 4.3

[ramindia (R!)]

thanks for the reply - as i mentioned the Date format can be extracted from different GROK patterns -it works - have changed that also.

I have also having issue using content pack old to new

[StefanAustin]

Yes, I was thinking about that. One thing that I know is, newer content packs have a version check and the old github thing does not have that line.

2016 is long ago, I have no idea about what changed in terms of required fields to load a content pack. Maybe you can create a new empty pack and check what you see.
Insert the old pack into the new pack, change the id and upload it.
I never tried to upload such an old pack. It might work, but maybe even the whole structure changed.

[ramindia (R!)]

thank yoy, may be i am not expert of creating content pack - I will read and try.
i able to make some grok pattern progress. as below image :

but if the logs coming difference format, how does he different pattern to match and organise the logs.

i see json format for the windows logs stream lined, how I can make same as it is ?

[StefanAustin]

I was looking into your GROK pattern and I saw some mistakes.
I do not know the log format, but the GROK for this log looks like this:

… %{WORD:process} %{NONNEGINT:process_number}…

or

…Process %{NONNEGINT:process_number}, Nbr %{IP:nbr_ip}…

depending on what you want to parse out.
Greedydata is easy to use, but it is the most expensive choice in terms of CPU.

[ramindia (R!)]

But WORD did not work at all.

as per I know compared to Json, Grok is more overhead on CPU I guess, if the server is processing millions of logs (I will dig into this later, when it comes to performance, for now, I am looking to streamline the data)

[StefanAustin]

This is a working grok pattern for the provided log.

If you look into json because you are getting the logs in json, Graylog offers a json parser. In that case, you will get the fields nicely. I never worked with that, so I cannot give you a working example.

parse_json(value, [depth]) : JsonNode Parses a string as a JSON tree

[tmacgbay]

If you want to continue down the GROK path, you can use a GROK debugger online like here to see how your GROK does with a log message. In the case where you are using WORD for the log_type, %OSPF-5-ADJCHG: is not a WORD because of the % and the dashes… If you add in the parts of your GROK to the debugger you can see which ones work and which you need to change…

[ramindia (R!)]

Thank you let me try that - with this example. i take that CPU less impact as you suggest not to use greedydata

[ramindia (R!)]

thank i do come across this debugger, its very confusing for me to try that tool
1 I type message
2 pattern here

nothing happens.

[tmacgbay]

Don’t put in the entire pattern. Add in each section for each capture, when you add a capture for a word that doesn’t work the entire thing breaks and you get NUL. Fix the one you just added.

[ihe]

Greedydata is some kind of dangerous in therms of CPU usage. You easily could build a Regexploit. I throw this link already in a few times here and still like it a lot

[ramindia (R!)]

i tried each line, but somehow not got what I expected, maybe I am trying wrong I guess.

i try to manage using grok pattern in Graylog testing and thank you @tmacgbay and @StefanAustin for helping (now I have some idea how the data need to strip and do in a better way in grok pattern) - somehow still learning.

The problem I see here - instead grok like to use the pipeline as suggested to save the CPU here.

when I add below line in rule its not taking not sure why

<%{POSINT:syslog_priority}>%{POSINT:syslog_version}: \*%{CISCOTIMESTAMP:timestamp}: %{NOTSPACE:event_source}: Line protocol on Interface %{NOTSPACE:network_interface}, changed state to %{WORD:status1}

especially due to

\*%{CISCOTIMESTAMP:timestamp}: (if I remove \* that rule take it and accept)

also this is the best way to create a rule (follows some document people doing this way - this works, but any draw back here ?)

rule “Cisco pipeline testing”
when
has_field(“message”)
then
let message_field = to_string($message.message);

let cs1 = grok(pattern: "", value: message_field,only_named_captures: true);
let cs2 = grok(pattern: "", value: message_field,only_named_captures: true);
let cs3 = grok(pattern: "", value: message_field,only_named_captures: true);

set_fields(cs1);
set_fields(cs2);
set_fields(cs3);

end

[tmacgbay]

Rather than have every message GROK’ed three times (not efficient) - set up a separate rule that uses the when/then to figure out the right message that it will GROK. You likely can’t create a GROK that covers all messages, instead you may have a rule where the when finds a message of a particular type and the GROK you use it in handles it properly.

If some part of the message is consistent you could have a rule separate out that (With GROK , regex, etc…) for all messages and then have a separate stage that figures out which message it is based on what the previous stage broke out. For instance if you figure out the EventID and put it into a field in the initial stage, then in a subsequent stage you use GROK to pull out the description with rules that each have a different when/then that handles the EventID and the GROK for that description… not a great example but you get the picture I hope…

with you testing you would test against the example like this:

• <%{POSINT:syslog_priority}>
  – if it works fine then add to it…
• <%{POSINT:syslog_priority}>%{POSINT:syslog_version}:
  – if that works then add to it…
• <%{POSINT:syslog_priority}>%{POSINT:syslog_version}: \*%{CISCOTIMESTAMP:timestamp}:

Keep in mind that each regex is shorthand for what really is a regex pattern…

Be sure to post the solution that works for future searchers!!!

[ramindia (R!)]

thank you for the suggestion, unfortunatly i am not that expert as I mentioned, but getting there learning few tips from the forum here.

take example as below messages I am parsing from cisco device :

<45>52: Jan 1 00:58:43.864: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to down
<45>54: Jan 1 01:00:19.558: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to up
<43>53: Jan 1 01:00:18.558: %LINK-3-UPDOWN: Interface Ethernet0/1, changed state to up
<45>51: Jan 1 00:58:42.864: %LINK-5-CHANGED: Interface Ethernet0/1, changed state to administratively down

Only first 3 pattern only matches as below :

• <%{POSINT:syslog_priority}>%{POSINT:syslog_version}: \*%{CISCOTIMESTAMP:timestamp}:
  after that content changed, so I need to make a pattern to match next, so I need to have different match pattern here for each message, (this is some how works in test environment ) - I am suspecting if I go production environment I expected 100000 or + logs coming in ( even though I have dedicated hardware, I am sure there is LAG when parse takes place.

. For instance, if you figure out the EventID and put it into a field in the initial stage, then in a subsequent stage you use GROK to pull out the description with rules that each have a different when/then that handles the EventID and the GROK for that description… not a great example but you get the picture I hope…

Some how the above suggestion made not able to get in to my brain appreciated if you can give some examples based on the messages I have posted.

Other note :

especially due to

\*%{CISCOTIMESTAMP:timestamp}: (if I remove \* that rule take it and accept)

I could able to fix this by adding grok pattern and use in the rule.

[tmacgbay]

@ramindia

100000 or + logs coming in

If that is your plan, you really need to be super efficient. I see this as working out in stages…

Pipeline rules Stage One - break out the following into fields:
(allowing the message to pass to stage two if at least one rule matches)

• syslog_priority
• syslog_version
• timestamp_c (don’t get it confused with the Graylog Timestamp)
• log_type

Then in Stage Two you can create a series of rules that only act on the message if it’s relevant so here are examples:

rule "Cisco - LineProto"
when
   to_string($message.log_type) == "LINEPROTO-5-UPDOWN"
then
   <script that GROK's the data after LINEPROTO-5-UPDOWN into the correct fields>
end

rule "Cisco - Link-Updown"
when
   to_string($message.log_type) == "LINK-3-UPDOWN"
then
   <script that GROK's the data after LINK-3-UPDOWN into the correct fields>
end

rule "Cisco - Link-Changed"
when
   to_string($message.log_type) == "LINK-5-CHANGED"
then
   <script that GROK's the data after LINK-5-CHANGED into the correct fields>
end

[ramindia (R!)]

Thank you and much appreciated your value time and guiding me right direction.
One last one before I start building rule and test.
all above mentioned can be in one rule
or I need to create each one as separate rules.

Like I have below

Stage1 going to be
example :
log stream line rule

Stage2 going to be
example :
log pasrser line rule

or you suggesting - stage 2 ( each one go different Rule ?)

Cisco - LineProto
Cisco - Link-Updown
Cisco - Link-Changed

if more rule - do we see any delay ?

[tmacgbay]

You want separate rules so that you are not trying to parse messages that won’t match. it is better to have several quick deciding rules than to have each message fail several parsing tries.

[ramindia (R!)]

Thank you, i will test that separate rule.

[tmacgbay]

Also of note, rules within a single stage generally run in “parallel” as much as possible. So to take action on the results of a rule, the next rule needs to be in a following stage… like what we are doing here.