Minimizing the amount of log data

[DanishIT_Guy (Patrick Kristensen)]

Heeyy
I’m new in here.
Soo we just got our Graylog server up and running last month.
And the logdata is just flying in.

So my issue is that i’m getting abit to much logdata.
We have only gotten the 10Gb license.
And for just our 3 domain controlleres, i’m getting 10 - 11 GB logs each day.
So i’m looking for at way to maybe get that around 9GB or below that.

So i’m hoping that there are some smart people in here.

Right now i have 3 inputs.
1 for each DC
input 1 29GB
input 2 15GB
input 3 20GB
All 3 is set up using GELF TCP.

And thanks to the Illuminate, i have a single stream where all the data basically is going to.
Windows Security Event Log Messages.

This is all i have set up as of right now.

Already got declined a upgrade to the 20GB License.
Budget is to tight right now (working in the Public sector).

So i’m really hoping someone can tell me there are a neet little trick.

Thanks in advance

[tmacgbay]

happy to help but since you are a paying customer, Graylog Support prefers you to go to them.

If you were only using opensource, I would ask more information about how you are using GELF to send in logs from the windows machines (not beats or nxlog?) and suggest either dropping message at whatever collector you are using at the DC’s or secondarily you could use Graylog rules in the pipeline to drop message (less efficient)

Still, it is better to work with Graylog support so that you continue to get information that Illuminate requires - all that is beyond the bounds of OpenSource.

Tell them I said hello!

[DanishIT_Guy (Patrick Kristensen)]

Thx for the Reply tmacgbay

Ohhh so the support can help out with something like this?
I thought they only could help if my system had an error of some kind…

but just to answer you questions, if i was using the OpenSource.
I’m using NxLog to send in my data

<Extension _charconv>
    Module      xm_charconv
    AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>

<Extension _exec>
    Module      xm_exec
</Extension>

<Extension _fileop>
    Module      xm_fileop

    # Check the size of our log file hourly, rotate if larger than 5MB
    <Schedule>
        Every   1 hour
        Exec    if (file_exists('%LOGFILE%') and \
                   (file_size('%LOGFILE%') >= 5M)) \
                    file_cycle('%LOGFILE%', 8);
    </Schedule>

    # Rotate our log file every week on Sunday at midnight
    <Schedule>
        When    @weekly
        Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
    </Schedule>
</Extension>

# Snare compatible example configuration
# Collecting event log
<Input in>
    Module      im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='1'>
                <Select Path='Application'>*</Select>
                <Select Path='Security'>*</Select>
                <Select Path='System'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>
</Input>
# 
# Converting events to Snare format and sending them out over TCP syslog
 <Output out>
     Module      om_tcp
     Host        Server IP
     Port        Port_Used
     OutputType  GELF_TCP
 </Output>
# 
# Connect input 'in' to output 'out'
 <Route 1>
     Path        in => out
 </Route>

So what you are saying, is that i should customize my config file here in another way?

[tmacgbay]

We (the OpenSource community) were talking about support at the recent Graylog Go conference and it was suggested that paying customers wouldn’t need to use the OpenSource Forums

I refer to @gsmith for all NXLog tuning since he knows that better than I, on the other hand, I have a bunch of windows machines (using winlogbeat) that I have tuned out some of the rifraff that comes in … an example (in beats format) below.

Best to work with Graylog Support - while I may not be interested in some reoccurring events, it may break parts of your illuminate to prematurely drop them.

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["${user.beathost}"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
 - DomainController
logging.metrics.enabled: false
winlogbeat:
  event_logs:
   - name: Application
   - name: System
   - name: Security
     processors:
       - drop_event.when.or:
           - equals.winlog.event_id: "4634"
           - equals.winlog.event_id: "4658"
       - drop_event.when:
           and:
             - equals.winlog.event_id: "4672"
             - regexp.winlog.event_data.SubjectUserName: '.*\$$'
       - drop_event.when:
           and:
             - equals.winlog.event_id: "4648"
             - equals.winlog.event_data.TargetUserName: "Batmans-left-boot"
             - equals.winlog.event_data.TargetServerName: 'CoolSrv'


   - name: DFS Replication
   - name: Directory Service
   - name: DNS Server
   - name: File Replication Service
   - name: Microsoft\Windows\NTLM\Operational
   - name: Microsoft\Windows\DHCP-Server\Microsoft-Windows-DHCP Server Events/Admin
   - name: Microsoft\Windows\DHCP-Server\Microsoft-Windows-DHCP Server Events/Operational

[gsmith]

Hello @DanishIT_Guy

Just chiming in, as @tmacgbay suggested since you have a paid version it is advisable to talk to Graylog Support.

But with Nxlog to reduce the amount of log as @tmacgbay showed above in his example is just send the Event ID’s needed not all of them.

Example for Nxlog

<Input in>
    Module      im_msvistalog
<QueryXML>
   <QueryList>                     
     <Query Id="0">  
        <Select Path="Security">*[System[(EventID=EventID=42 or EventID=4242)]]</Select>
     </Query>
   </QueryList>
</QueryXML>
</Input in>

[DanishIT_Guy (Patrick Kristensen)]

Alright.

Thank you @gsmith and @tmacgbay