Step 9 and 10: First Search and Figuring our Syntax

Contests, Podcasts, Blogs, Give-Aways & Polls Contests & Give-Aways
1

BONUS WEEK! We’re combining last week’s Graylog User Journey question with this week’s. Please respond to BOTH prompts to play. Remember, we’ll be awarding one lucky participant with a $100 Amazon Gift Certificate after Week 12. If you haven’t played ,there’s still time to catch our top contenders, Just post a detailed response for each week’s step. It’ll give you one chance for each week’s post (12 in all).

Week 9 question: Think back at the first search you performed (or even a typical search you perform) in Graylog. What were you looking for? Why? What were your results. What tips or advice can you offer other members on searching?

Week 10 question: Having told us what you were searching for, post the syntax you used. If you’re unsure of the syntax, ask here for help (that’s what we’re all about!). Were you results what you expected? How did you tweak the syntax to make your search more exact?

1 Like
4

When we first deployed Graylog it was to meet a very specific need. We were evaluating a couple of potential solutions. We had quite a bit of experience using another familiar log management solution. Despite that, the first searches we constructed (which we believed for a bit were working correctly) did not meet our needs.

What we had wanted to find:

  • A near-realtime feed of actions taken by a new firewall solution we were using to migrate microsegmentation rules from a solution we no longer needed.

What we ended up with (we realized after some time):

  • Not that at all.

We had multiple issues to resolve and they were confounding one another because there were too many unknown variables.

  1. We weren’t already deeply familiar with the new firewall solution we were implementing so we didn’t know what to expect the logs to look like.
  2. We also weren’t deeply familiar with Graylog so we didn’t know what to expect when using it.
  3. We didn’t really know how to construct a search in Graylog to achieve our goal.

So, we spent some time reading the manual, which is tip #1. There’s a lot of good documentation available for Graylog (and the Lucene syntax) and we have seen repeatedly it’s pretty important to read it. Tip #2 is to spend some time becoming familiar with your logs before you start trying to construct things around them.

There were two issues we were running into with the search syntax specifically. We had initially constructed our query to look like this:

message: "action=<ACTION>" and message: "dest=<destination>"

The problem with this is that since ‘and’ isn’t all caps, it becomes part of the query string. So, we replaced ‘and’ with ‘AND’ and things worked better.

We also were having an issue with exclusions. We found that we would get inconsistent results if our criteria were

message: "string" AND NOT message: "string" AND NOT message: "string"

So, we started using grouping for exclusions:

message: "string" AND NOT (message: "string" OR message: "string")

We find this works consistently.

2 Likes
5

Awesome, Andrew. Thanks for posting and playing. You’ve gained another chance to win the Amazon Gift Certificate. Good luck!

6

One of the reasons we want a Logging server in our environment was to track issues from multiple Servers. Having to log into each one and looking through logs became a tedious task. We need a quicker response to issues. Our devices consist of switches, firewalls, cluster servers, virtual machines, access points, etc…. Also, we had different Operating systems such as Linux and Windows. Once our Graylog server was functioning correctly my task was to find out who logged into what machine. At this time, we were only using Syslog UDP but had multiple UDP inputs for specific devices. One input would collect logs from windows devices then the next one would collect logs from Linux devices and so on. One of the main reasons for this was I need to create a new field for user’s logon such as EventID form windows logs and extract the EventID from the block of messages.

Using the Input Syslog UDP did not help for global search we were limited by fields that were needed. This made our global searches inaccurate. Since our Global searches were limited, for example I could not search EventID this field was in the block of Messages. When executing search EventID: 4624, this showed all messages with 4624 within the messages not necessarily the EventID. We did create extractor but had to many of them which was causing issues.

We started realizing that Fields were the secret to finding out who logged in and/or logged out. Then a community member suggested using GELF inputs. I was like

“well if I’m going to do that, I’m configure TCP/TLS for the inputs also”.

This created all my fields needed to solve my issue for finding the field needed and other fields that I needed later. Now were able to use a sting of fields in our search criteria which showed us who logged into which node.

source: some_node AND EventID: 4624

We had to fine tune this search. Just to give you an idea below are two types of Event IDs for logging on.

EventID 4624 (viewed in Windows Event Viewer) documents every successful attempt at logging on to a local computer.

EventID 4648: A logon was attempted using explicit credentials.

After finding out EventID 4648 could be a service like Veeam or Hyper-V with credentials for RunAs (false/positive). Again, we had to fine tune our global search and using the autocomplete for fields in the search bar, which helped a lot. We were able to execute something like this.

source: some node AND EventID: 4624 OR EventID: 4648

Most of our mistakes were in upper-case and lower-case letters. Using this proper took a few minutes.
As for windows devices, executing a search for Event ID’s, the results were a lot and not precise.

We had to adjust our searches. Our flow looked something like this.

INPUT → STREAM → SOURCE_FIELD → EVENTID_FIELD

gl2_source_input:5a62903bffe8b1e04bd2fe89 AND streams:5b9201d583d72e03a33047d6 AND source: gsmitt.domain.com AND EventID:4624

Now we were able to create a dashboard with widgets for user logons. Creating widgets in our dashboards for an overall view with a quick glance was key. The widgets on our dashboard/s showed us what user/s were logging into windows devices. This purpose was used mainly for production servers like DNS, AD, Gateway servers or any critical server in our production environment. The data on the widget was a lot better, this gave us the ability to see if anyone was logging into any server.
These adjustment flowed down to our Event Definitions. So, now when alerts were triggered it was what we wanted and not from a service starting up that was running under a local user.

Fine tuning our alerts from user/s logons we started using pipelines which came in handy. For example, “User logged in Outside Business Hours” Since we started using the correct Input format and stream filtering exactly what we wanted. Now we get alerted if anyone logs on to any device or server within the environment after hours, one the weekends or devices that no user should log into without permissions. This all happens when the correct INPUT is made with the proper fields created and the global search are executed correctly.

1 Like
8

As usual my first patter was to find errors and warning in my collected log messages. I started with simple pattern: (error OR warn OR fail) which works fine, but not for every operating systems and network device. So I pre-created saved search for most used error, warnings and so patterns for every system, so I could easily quick load them to search field.

My tips:

  • My first mistake was not to use uppercase AND, OR, NOT keywords in search. That way my search pattern look fine, but result wasn’t as expected. So beware of correct keywords in your search pattern.
  • TO in range operator is also case sensitive. So don’t forgot to use correct syntax: response:[400 TO 599]
  • If you want to speedup your searches, use field specific search like field:error and not only free search error, as that will search in more fields, and will be slower.
  • Don’t forgot that ES support special regex syntax, so sometimes it’s handy to use it:
    Regular expression syntax | Elasticsearch Guide [7.14] | Elastic
    • /abc|def/ - abc or def
    • /[0-9]{2}/ - Two numbers
    • /[^abc]/ - Everything except a, b, c
  • Field search must include full match. It’s a common mistake by beginners. If you find for file:example it won’t match example.txt. You need to use exact match file:example.txt. Or use ? for one character or wildcard * for more characters to partial mach: file:example*, file:example.*