Upgrade of Graylog from 2.5.x to 3.x results in

So I just upgraded from 2.5.x to 3.0, and I believe I have some broken grok extractors, related to this issue: https://github.com/Graylog2/graylog2-server/issues/5704

I finally sorted out all the other warnings/errors on the graylog server.log file. I believe that I only have issues related to pfsense log extraction. I deleted the two extractors (which may have now made things worse), both gave errors in the webUI and when accessing the UI produced the able error in the log. Unfortunately, the errors are still there even after a server restart. It looks like all my other inputs are fine, but my pfSense isn’t. It’s still capturing data, but it’s not transforming it in any way.

I’m not sure how to fix any of this, or if I can. I thought I had it figured out and that I’d be able to just remove those two extractors and then the data would then correctly be parsed again. That is apparently not the case though.

Is there anything I can do myself? Can someone point me in the right direction? I don’t know how it can still be throwing the errors after I removed those two extractors.

If it helps, I used this guide to add the information to grafana: hxxps://github.com/opc40772/pfsense-graylog (had to nerf the link due to being a “new user”.

Also, here is a log from a “clean” startup: https://hastebin.com/raw/eluqaresat

1 Like

how did you do the extraction of pfsense? What did you do where and what is the error you have?

I followed this guide here: https://github.com/opc40772/pfsense-graylog

Everything was working perfectly well on 2.5.x, and now its stopped working due to the mentioned issue above.

I’m open to something new, but I’m very fresh in how Graylog works, and how I would parse those values myself.

Hey Jacob,

we will release a fix for the underscore problem with 3.0.1. It will hopefully be ready next week but it might take another week.

So here is what you can do, to fix your problem:

  1. You could fix up the grok patterns which are the root of the problem, but replaceing the underscore with a dash. But this will lead into new fields being shown in you index.
  2. You can take a look at your mongodb, and look if the rouge extractors are still there… You will find the extractors as a sub field of the inputs collection. But I highly recommend to backup your mongodb before messing around with it!

@konrad = Thanks!

It’s not critical right now, so if there is a definite plan to support the underscores, I can let it sit where it is, and then update so that things start to work again.

Is it a confirmed fix for 3.0.1? That way I can just be on the lookout for the new package in the repo and will know it’s time to upgrade and test again.

Thanks for all the great work you guys do!

@konrad - Not to be a pest, but is there any info on progress?

as conrad wrote 5 days in the paste “the next week or the week after”

Yes… Which implies a completion date (which I specifically did not ask for). What I asked for, was any progress toward a solution. It’s not uncommon to find out as you’re crafting a solution to a problem to suddenly find yourself in a situation where either, the fix you’re planning isn’t going to work, or the fix is worse than the symptom. I’m more asking for a “confidence factor” as to how confident they are that the implemented fix will be transparent, or if there is something the user community is going to have to do.

Right now, given the problem description, all I’m planning on needing to do is upgrade to the release that fixes the issue, and everything should suddenly start to work again. If that’s not the case, it would be nice to know that sooner rather than later.

Asking questions is never a problem. And I can totally understand the uncertainty.

A solution for the problem was found and a PR is waiting for review. The plan is to to include the PR in the upcoming patch release. Since the decision of when the patch will be released is not in my hands, I can’t tell you a certain date by now.

@konrad - thanks for the update! I know better than to ask for actual release dates LOL. I’ve been around enough open source projects to know it only aggravates developers because things like that are usually out of their control.

Crossing my fingers that the release happens soon though. :grin:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.