One of the reasons we want a Logging server in our environment was to track issues from multiple Servers. Having to log into each one and looking through logs became a tedious task. We need a quicker response to issues. Our devices consist of switches, firewalls, cluster servers, virtual machines, access points, etc…. Also, we had different Operating systems such as Linux and Windows. Once our Graylog server was functioning correctly my task was to find out who logged into what machine. At this time, we were only using Syslog UDP but had multiple UDP inputs for specific devices. One input would collect logs from windows devices then the next one would collect logs from Linux devices and so on. One of the main reasons for this was I need to create a new field for user’s logon such as EventID form windows logs and extract the EventID from the block of messages.
Using the Input Syslog UDP did not help for global search we were limited by fields that were needed. This made our global searches inaccurate. Since our Global searches were limited, for example I could not search EventID this field was in the block of Messages. When executing search EventID: 4624, this showed all messages with 4624 within the messages not necessarily the EventID. We did create extractor but had to many of them which was causing issues.
We started realizing that Fields were the secret to finding out who logged in and/or logged out. Then a community member suggested using GELF inputs. I was like
“well if I’m going to do that, I’m configure TCP/TLS for the inputs also”.
This created all my fields needed to solve my issue for finding the field needed and other fields that I needed later. Now were able to use a sting of fields in our search criteria which showed us who logged into which node.
source: some_node AND EventID: 4624
We had to fine tune this search. Just to give you an idea below are two types of Event IDs for logging on.
EventID 4624 (viewed in Windows Event Viewer) documents every successful attempt at logging on to a local computer.
EventID 4648: A logon was attempted using explicit credentials.
After finding out EventID 4648 could be a service like Veeam or Hyper-V with credentials for RunAs (false/positive). Again, we had to fine tune our global search and using the autocomplete for fields in the search bar, which helped a lot. We were able to execute something like this.
source: some node AND EventID: 4624 OR EventID: 4648
Most of our mistakes were in upper-case and lower-case letters. Using this proper took a few minutes.
As for windows devices, executing a search for Event ID’s, the results were a lot and not precise.
We had to adjust our searches. Our flow looked something like this.
INPUT → STREAM → SOURCE_FIELD → EVENTID_FIELD
gl2_source_input:5a62903bffe8b1e04bd2fe89 AND streams:5b9201d583d72e03a33047d6 AND source: gsmitt.domain.com AND EventID:4624
Now we were able to create a dashboard with widgets for user logons. Creating widgets in our dashboards for an overall view with a quick glance was key. The widgets on our dashboard/s showed us what user/s were logging into windows devices. This purpose was used mainly for production servers like DNS, AD, Gateway servers or any critical server in our production environment. The data on the widget was a lot better, this gave us the ability to see if anyone was logging into any server.
These adjustment flowed down to our Event Definitions. So, now when alerts were triggered it was what we wanted and not from a service starting up that was running under a local user.
Fine tuning our alerts from user/s logons we started using pipelines which came in handy. For example, “User logged in Outside Business Hours” Since we started using the correct Input format and stream filtering exactly what we wanted. Now we get alerted if anyone logs on to any device or server within the environment after hours, one the weekends or devices that no user should log into without permissions. This all happens when the correct INPUT is made with the proper fields created and the global search are executed correctly.