Windows Log ingestion into graylog server


(Nav) #1

I am absolutely newbee to graylog. I have downloaded the latest 2.2.x OVA appliance of graylog, installed it.
I have also installed side car to server 2012 r2 x64 bit machine

I have 2 questions.

  1. How do i push windows extended logs (microsoft-printservice-operational) logs (specific events, 307 and 805, xml data) into gray log server.

  2. How do i push csv files on windows share into graylog server.

any blogpost, pointers, help will be great.

Thanks
Nav


(Jan Doberstein) #2

Hej Nav,

maybe this blog posting will give you an idea how to get the different event logs.

The filebeat Input is what you are looking for to get some files, line by line into Graylog.

You might find the documentation useful.

/jd


(Nav) #3

I tried to follow the post, it’s difficult to follow as too much [basic building block] info is on the external links.

I currently focusing on getting CSV file data [generated everday] into the gray. Then i will look into the windows log which are more tricky, at least at my level.

I have managed to install sidecar and the service graylog collector is running now.
I don’t see default filebeat.yml file under generated folder and error is logged

I have searched for a sample filebeat.yml file and configured it to get started. After i restart the service, i get the following.
Exiting: error initializing publisher: No outputs are defined. Please define one under the output section.

The folder contains csv files in 2 subfolders, like csv/FolderA/ and csv/FolderB/

filebeat.yml
prospectors:
- C:\csv**
encoding: utf-8
exclude_lines: ["^#"]
exclude_files: [".zip"]
ignore_older: 240h
registry_file: “C:/ProgramData/filebeat/registry”

Regards,
Navdeep


(Bill Murrin) #4

Configure the output.logstash section of your filebeat.yml to send data to your graylog server IP address. Setup a beats input on your Graylog server. Ensure that you use the same port on both. Beats typically uses TCP port 5044 by default.


(Nav) #5

I have updated the filebeat.yml with the following
filebeat:
prospectors:

  • input_type: log
    • paths:
      • C:/csv/DocumentPrinted//
        encoding: utf-8
        ignore_older: 240h

output:
logstash:
hosts: [“172.30.48.145:5044”]

and now i get this in the logs … last few lines
Exiting: Error in initing prospector: Invalid input type: log - C:\csv\DocumentPrinted*
Exiting: Error in initing prospector: Invalid input type: log - C:\csv\DocumentPrinted**
Exiting: Error in initing prospector: Invalid input type: log - C:\csv\DocumentPrinted**
Exiting: Error in initing prospector: Invalid input type: log - C:\csv\DocumentPrinted**
Exiting: Error in initing prospector: Invalid input type: log - C:\csv\DocumentPrinted**
Exiting: error loading config file: yaml: line 4: did not find expected
Exiting: error loading config file: yaml: line 4: did not find expected
Exiting: error loading config file: yaml: line 4: did not find expected
Exiting: error loading config file: yaml: line 4: did not find expected
Exiting: error loading config file: yaml: line 3: did not find expected key
Exiting: error loading config file: yaml: line 3: did not find expected key


(Nav) #6

looks like there was some whitespace inthe filebeat.yml file, now the config looks like

filebeat:
prospectors:

  • paths:
    • c:\csv\RenderJobDiag*.csv
    • c:\csv\DocumentPrinted*.csv
      input_type: log
      encoding: utf-8
      ignore_older: 240h

output:
logstash:
hosts: [“172.30.48.145:5044”]
compression_level: 3

and i don’t get any error in the logs

so what is the next step from here on. I still don’t see anything on the graylog server. I have configure filebeat collector running on port 5044 on graylog server.


(Nav) #7

On graylog server, under collectors, i see graylog-sidecar-collector with status failing
It says
Status: No configuration found for configured tags!

on client, collector_sidecar.log states
time=“2017-04-25T15:49:41+08:00” level=info msg="[RequestConfiguration] No configuration found for configured tags!"

looks i am missing some configuration here.


(Bill Murrin) #8

Is your beats input in Graylog started?

If you look in your filebeat mybeat logfile on the server shipping the logs, do you see any events being sent (should see that x registry updated) xxxx events sent or something along those lines.

Can you see the connection between the two systems using netstat on the graylog server using something like netstat.

netstat -antp | grep 5044

(Nav) #9

I checked from the webinterface, it seems to be started
https://imgur.com/n4D2M3V


(Bill Murrin) #10

It looks like it has received 183.5kb so far. Can you see the data show up under All Messages? Also try clicking on the Show Received Messages.


(Nav) #11

It’s empty, do i need to configure collector configurations, i suspect problem is somewhere here.
https://imgur.com/Z8kzIQ3

I tried to follow this article, http://docs.graylog.org/en/2.2/pages/collector_sidecar.html#sidecar-step-by-step
however i am not sure if i have got output part right.


(Bill Murrin) #12

I haven’t used SideCar Collector so I can’t say, but you should be able to get filebeat sending to Graylog without Collector even being involved. Try to minimize the number of variables involved to just get it working, Once it is working, add collector so you can manage filebeat on your server remotely.

By default, filebeat will send its logs in Windows to C:\ProgramData<beat-name>\Logs. Check your mybeat file to see if it shows anything being sent.


(Nav) #13

i think i am near to the fix but not so near. Do you know how can we specify multiple paths in input under configuration

I tried this but getting error.
[‘C:\csv\DocumentPrinted*.csv’], [‘C:\csv\RenderJobDiag*.csv’]


(Bill Murrin) #14

Check this out. https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html


(Nav) #15

i checked that, but now the filebeat.yml file is being updated via graylog server through webinterface.
i suspect the problem lies somwhere between inputs and outputs.


(Nav) #16

does this filebeat.yml content looks fine to you?
filebeat:
prospectors:

  • document_type: log
    encoding: plain
    fields:
    gl2_source_collector: 25e87973-306f-4169-9dea-2144b5448206
    ignore_older: 4800h
    input_type: log
    paths:
    • C:\csv\DocumentPrinted*.csv
      scan_frequency: 10s
      tail_files: true
      output:
      logstash:
      hosts:
    • 172.30.48.145:5044
      path:
      data: C:\Program Files\graylog\collector-sidecar\cache\filebeat\data
      logs: C:\Program Files\graylog\collector-sidecar\logs
      tags:
  • windows
  • csv

it’s auto updated.


(Nav) #17

do i need to create system>outputs type gelf ?
I have create inputs and outputs under system>collector> manage configuration.

output is set to filebeat with graylog server ip and 5044 port
input is set to filebeat with output create above


(Bill Murrin) #18

No, an output would be if you wanted to send data from Graylog to somewhere else. You are working on getting an input working. Can you try to post a chunk of your filebeat log file. Please encapsulate in ``` code ``` so it gets rendered properly. Your filebeat logs might be under C:\Program Files\graylog\collector-sidecar\logs


(Nav) #19

File beat log is empty, i was getting too many errors when troubleshooting, so i cleared them after success. I haven’t seen any logs populating in the logs file after that. I have restarted the service few times.
https://imgur.com/OfMMxUe


(Bill Murrin) #20

Is that the only log you have for filebeat?