I’ve been using graylog as a raw dump for some logs that expire more frequently than I’d like on some other systems, but have recently wanted to come back to it and implement some of the more advanced features. I’m having two issues:
When I make changes to the configuration (e.g. change/add an input, change/add an extractor) the changes don’t seem to be reflected in a timely fashion. The delay is at least 20-30 minutes, and may be several hours. For instance, I added an extractor yesterday mid-day to copy some input from the message field to another field. Even at the end of the day, that wasn’t happening. However, when I returned this morning, it was happening. Is this normal? Is there something to do to speed it up? Where should I start troubleshooting?
I have messages coming in from a WiFi controller that have a bunch of key/value pairs included in the message, in the format:
(controllername) Core: @@236,clientInfoUpdate,“apMac”="(value)",“clientMac”="(value)",“ssid”="(value)", (and so forth - about 20 key/value pairs in total)
I’ve tried using a regex extractor to retrieve just the “key”=“value” part of the message (which seems to work) and added the “Add Key=Value pairs as fields” converter in it, but the converter doesn’t seem to be doing anything. Is this how it should be working? Are there other ways to do this? Any tips on where to start troubleshooting?
- Subpoint: What I’d really like is to extract about 8-10 of those key-pair values, and ignore the rest. Is there a better way to do that?
My Graylog instance is a everything running on a single server running Ubuntu 20.04 as a Hyper-V guest. It was installed following the Ubuntu installation directions in the Graylog documentation. It’s Graylog 4.2.5.
Package Version: 4.2.5+59802bf (I think this is what this means?)
Service logs, configurations, and environment variables: (I’m sorry - I’m not entirely sure what information is being requested here)
3. What steps have you already taken to try and solve the problem?
I have done several searches about both problems, both general internet searches as well as on the reddit and in this forum, as well as watched several graylog-101-style youtube videos. I’ve also read the graylog documentation page on extractors and converters although I’m sure I’ve missed things.
When trying to extract the key-value pairs I came across the key=value converter which seemed promising, but doesn’t seem to work the way the internet tells me it should - this could easily be a failure in my understanding. I understand a pipeline may also be a solution to my problem, but haven’t yet gone down that road.
I’ve been completely unable to find anyone suffering from the ‘changes to configurations take a long time to apply’ problem I’m having, so I’m not even sure where to begin. It doesn’t seem(??) like a resource problem based on what I can see, but am happy to do any tests y’all think would be useful.
4. How can the community help?
I’d appreciate any troubleshooting tips, strategies, or starting points. Any links to documents that you think would be useful for me to read would also be valued. And if you’ve solved or experienced either of these issues and know exactly what I need to do, so much the better
Thank you in advance for your help and expertise.