OK, I have Graylog installed on a standalone CentOS 7 machine and currently it is ingesting from a number of servers, but just the /var/log/messages file. I really want to be able to gather up all of our various Apache log files here into Graylog as well as IIS (especially to find patterns of IPs). I’m still not real sure how to do this OR how to provide data to my supervisor in a way that is easy to understand using Graylog. Any guidance and/or suggestions are much appreciated. Just don’t want to have spent my time on getting this set up and then not be able to use it. Thank you.
(Oh, and related to this - we have another service [proxy service] that records a lot of data in the logs; how can I manipulate that within Graylog? I would really, REALLY love to do that!!!)
We have been able to do just what you describe. Only in my environment it is mostly just IIS, NGINX, and HAProxy. If you would like help sometime I can maybe show you the ropes. Getting the logs parsed and shipped is most of the battle. After that it is pretty smooth sailing for setting up dashboards and pulling together searches to find problem IP’s or user agents etc. Let me know if you run into something that you need help with.
This is great to hear that someone else is doing the very thing we are wanting to do! OK, so first, did you install the Collector on the servers you were wanting data from (just having a hard time getting all the pieces in place)? And what do I need on the Graylog server (Input? Output? Etc.)?
At first we didn’t use collector on the first few servers and instead just ran NXLOG by itself. After seeing the power and benefits of using the collector we now use it EVERYWHERE. It just makes managing hundreds of instances so much easier. So yes. Install collector and NXLOG on your the servers you want to ship logs from. You can ship them raw but that would mean you would have to create extractors for all the fields you need in graylog itself. This can be a resource intensive operation for graylog, especially if not keeping regex and GROK pattern usage under control. As such we try to keep the need for extractors to a minimum. Do this by having NXLOG do the work for us as a log shipper. In a perfect world all logs would be in an easily parsed format like JSON but unfortunately this isn’t always possible. (IIS,NGINX,etc) NXLOG helps with this by not only shipping your local logs to graylog but it can also reformat the outgoing messages and output them in whatever format you require. I’d recommend JSON or GELF as these are natively supported and easily parsed by Graylog as the fields are predefined before being shipped. On the graylog side you would create an input that you point the NXLOG to output to. Say like in our case NXLOG reformats our IIS logs to JSON and outputs GELF to our Graylog GELF input at whatever IP and Port you set. I recently shared an NXLOG config in another post to help out another member with the IIS log shipping via NXLOG. Here is a link to that post from last week. NXLOG IIS CONFIG That config is a very good starting point for IIS but the same concepts apply to using it to ship any other type of log. Once logs start flowing in you should find it super simple to start searching and creating dashboards/reports in graylog for whatever you need to pull out of them. Let us know how it goes. If you have any more questions just ask.
I have had one issue with nxlog and IIS. In my environment, not every IIS server logs the same fields. This is configurable in IIS. So, I have to make sure that the “Fields” line in the nxlog section matches what my IIS server is actually writing to the logs. Otherwise, it does work well.
We ran into the same thing Beth and adjusted just the same as you have. Standardizing logs where possible really helps in this regard. Even within the same IIS server different sites can have different logging requirements and configurations. You can actually account for this. Have your cake and eat it too so to speak. parsing multiple site logs from a single IIS server using NXLOG. You just have to set up multiple inputs with each their own “field” formatting and all should be good. Viola, multiple log format shipping using a single NXLOG agent.
Matt – thank you for this guidance, but I am confused by something. The Collector (by default) rewrites the nxlog.conf file that it uses whenever it connects to the Graylog server, so how did you get around that?
If I’m following you correctly. It sounds like you have created a working nxlog.conf on the server you are wanting to ship logs from but also have collector installed. Did you use collector to make the configuration or did you do it by hand. Part of the collectors functionality is to indeed overwrite the old configuration with a new one that has been modified via the graylog interface. If you create or input your config via graylog and let the collector push the config that nxlog uses on those servers all should work seamlessly for you. This way you can automate single or mass deployments of a standard config to the appropriately tagged servers without having to script out or manually modify all the nxlog instance configs yourself. Kind of a one to many config push so to speak. If i’m not understanding what you are doing let me know but it sounds like you may be loading/building the nxlog configs manually. Okay for a few machines. cumbersome for many servers.
Yes, that is what I was doing – modifying the nxlog.conf file on my server that I want to send the data to my Graylog server. I guess I am confused as to how to make my Graylog server generate the nxlog conf file with the data I am looking for. Do I use the web interface for that or do I need to do it from the command line? (we are using a CentOS 7 box for our Graylog server, which sits in y office currently).
Yes. You do this through the graylog interface. Under “System > Collectors” From there you can create NXLOG or Beats configs via the web interface. This way the config you create is pushed by graylog to the collector sidecar on your server that replaces the config for NXLOG and reloads for you. You create “Tags” that correspond to server type. For instance. Say you have 5 IIS servers. All identical. You have collector-sidecar and nxlog on them ingesting and shipping logs. By “tagging” them as say “IIS” you can effectively create one config via graylog and it will not only ship it out to those IIS servers for you but collector will reset nxlog so the new config takes effect for almost immediately. Graylog even monitors the servers and reports on whether the remote Sidecar is “running” and what OS type its on as well as the ID. You can even show messages incoming from just that webnode with a single click this way. Here is an example of what it looks like. It is really more simple that you think when starting out. One of those things that seems more complicated than it really is. You can even issue multiple tags to the same server if it is a multirole server running multiple applications or services you need to log.
Thank you - I am getting close now as I am beginning to understand how all of the various pieces work together. My thinking is that once I get one going, I should be able to do the others pretty quickly. Right now, I am seeing the following errors in my collector log:
ERROR om_tcp detected a connection error; End of file found
and
ERROR om_tcp detected a connection error; Connection reset by peer
I’m not sure how long it takes for the logs to go from my server to my Graylog server but currently I have no “messages” there (Graylog server).
