I’m new in here.
Soo we just got our Graylog server up and running last month.
And the logdata is just flying in.
So my issue is that i’m getting abit to much logdata.
We have only gotten the 10Gb license.
And for just our 3 domain controlleres, i’m getting 10 - 11 GB logs each day.
So i’m looking for at way to maybe get that around 9GB or below that.
So i’m hoping that there are some smart people in here.
Right now i have 3 inputs.
1 for each DC
input 1 29GB
input 2 15GB
input 3 20GB
All 3 is set up using GELF TCP.
And thanks to the Illuminate, i have a single stream where all the data basically is going to.
Windows Security Event Log Messages.
This is all i have set up as of right now.
Already got declined a upgrade to the 20GB License.
Budget is to tight right now (working in the Public sector).
So i’m really hoping someone can tell me there are a neet little trick.
Thanks in advance
happy to help but since you are a paying customer, Graylog Support prefers you to go to them.
If you were only using opensource, I would ask more information about how you are using GELF to send in logs from the windows machines (not beats or nxlog?) and suggest either dropping message at whatever collector you are using at the DC’s or secondarily you could use Graylog rules in the pipeline to drop message (less efficient)
Still, it is better to work with Graylog support so that you continue to get information that Illuminate requires - all that is beyond the bounds of OpenSource.
Tell them I said hello!
Thx for the Reply tmacgbay
Ohhh so the support can help out with something like this?
I thought they only could help if my system had an error of some kind…
but just to answer you questions, if i was using the OpenSource.
I’m using NxLog to send in my data
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
# Check the size of our log file hourly, rotate if larger than 5MB
Every 1 hour
Exec if (file_exists('%LOGFILE%') and \
(file_size('%LOGFILE%') >= 5M)) \
# Rotate our log file every week on Sunday at midnight
Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
# Snare compatible example configuration
# Collecting event log
# Converting events to Snare format and sending them out over TCP syslog
Host Server IP
# Connect input 'in' to output 'out'
Path in => out
So what you are saying, is that i should customize my config file here in another way?
We (the OpenSource community) were talking about support at the recent Graylog Go conference and it was suggested that paying customers wouldn’t need to use the OpenSource Forums
I refer to @gsmith for all NXLog tuning since he knows that better than I, on the other hand, I have a bunch of windows machines (using winlogbeat) that I have tuned out some of the rifraff that comes in … an example (in beats format) below.
Best to work with Graylog Support - while I may not be interested in some reoccurring events, it may break parts of your illuminate to prematurely drop them.
# Needed for Graylog
data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
logs: C:\Program Files\Graylog\sidecar\logs
- name: Application
- name: System
- name: Security
- equals.winlog.event_id: "4634"
- equals.winlog.event_id: "4658"
- equals.winlog.event_id: "4672"
- regexp.winlog.event_data.SubjectUserName: '.*\$$'
- equals.winlog.event_id: "4648"
- equals.winlog.event_data.TargetUserName: "Batmans-left-boot"
- equals.winlog.event_data.TargetServerName: 'CoolSrv'
- name: DFS Replication
- name: Directory Service
- name: DNS Server
- name: File Replication Service
- name: Microsoft\Windows\NTLM\Operational
- name: Microsoft\Windows\DHCP-Server\Microsoft-Windows-DHCP Server Events/Admin
- name: Microsoft\Windows\DHCP-Server\Microsoft-Windows-DHCP Server Events/Operational
Just chiming in, as @tmacgbay suggested since you have a paid version it is advisable to talk to Graylog Support.
But with Nxlog to reduce the amount of log as @tmacgbay showed above in his example is just send the Event ID’s needed not all of them.
Example for Nxlog
<Select Path="Security">*[System[(EventID=EventID=42 or EventID=4242)]]</Select>
Thank you @gsmith and @tmacgbay