I have Graylog up and running with Opensearch in Docker. As a side note, the documentation recommends using Opensearch but all of the install information is for Elasticsearch.
I’ve configured several things to send logs to Graylog but I’m running into two issues.
If a message doesn’t match what the input expects, it’s silently dropped. There’s no indication that I can see that Graylog even received the message, which makes troubleshooting difficult.
I found some extractors and they mostly work, but I get errors in the logs occasionally about csv field mismatches. I can’t seem to find anyplace that tells me what these messages are that have this field mismatch.
It looks like I need Operations to be able to log error messages, but that seems to be for the Indices, not the Inputs. Other than that, I haven’t been able to find any information about locating incoming messages that didn’t work for the Input.
I have run into this issue before, When log/message drops its pretty much gone. But If you set up your input and group the type of devices /w type of formated logs together, your good to go. Take note, If you sending Database logs with some linux syslogs, there might be a issue but a RawplaintextInput should work with most everything. Some times its not one size fits all.
This is kind of up to you, If your grabbing all the different types of log from, lets say Linux /var/log you might run into issues. For example my firewall logs go to RawPlaintext TCP input, Linux logs go to Sysylog TCP/TLS, Windows logs either go to GELF TCP/TLS input OR a Beat’s input that is if im using Winlogbeat as a log shipper.
Hope that helps
I understand that I don’t want to mix logs on inputs. But when trying to set things up and organize them, it can be helpful see what’s getting dropped. For example, if I have a syslog input and for some reason one of my devices is sending logs that aren’t in the correct syslog format. Right now I have no way to know that I’m missing any logs.
My issue with Input errors isn’t caused by mixing log types. I’m just processing syslog currently. But occasionally some of the log messages are having different field counts, hence the csv field mismatch error. Right now I don’t have a way to troubleshoot those as Graylog seems to just blackhole them with only a generic error message in the logs to tell me it happened.
Where would I do this and what would I be looking for? I tried setting everything in System/Logging to Debug but I’m not finding anything in the logs relating to dropped messages or what their content was.
I didn’t realize Graylog was written in Java. Also, ROFLCopter?
It does seem to have a few parsing error outputs but I never saw any of those appear in my logs.
In regards to the Csv extractor errors, I ended up redoing things with Pipelines, which at least provides a somewhat useable error message. However, unless I’m mistaken, there isn’t a way to run a simulation on anything other than the message field. Which makes conditionals, etc difficult to debug.
if you want to debug your rules/pipelines I tend to attach them to a stream view with a decorator in a second tab. This way you will not mess up your logs while trying, and if it is working you can attach the pipeline/rules at the right place.
