Please post the complete configuration of Filebeat.
According to https://www.elastic.co/guide/en/beats/filebeat/5.6/configuration-filebeat-options.html#prospector-paths, using these globs should work.
filebeat:
prospectors:
- encoding: plain
fields:
collector_node_id: graylog-collector-sidecar
gl2_source_collector: 215c8afc-8842-4394-9781-d09355560338
type: log
ignore_older: 0
paths:
- C:\inetpub\logs\LogFiles\*\*
scan_frequency: 10s
tail_files: true
type: log
output:
logstash:
hosts:
- localhost:5045
path:
data: C:\Program Files\graylog\collector-sidecar\cache\filebeat\data
logs: C:\Program Files\graylog\collector-sidecar\logs
tags:
- Windows
- iisWhich version of Filebeat are you using?
What’s the output of the following command in the Command Prompt? (https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/tree.mspx)
tree C:\inetpub\logs\LogFiles /f
Filebeat was in the Collector_sidecar_installer_0.1.4-1 package. I don’t know which version it is.
The result of tree is this and there about 25 lines in the log file.
Structure du dossier pour le volume Programmes
Le numéro de série du volume est BED0-4466
C:\INETPUB\LOGS\LOGFILES
└───W3SVC1
u_ex171012.log
Have lines been added to the log file while Filebeat was running?
What’s in the log output of Filebeat itself?
Here it is.
2017-10-13T11:12:47-04:00 INFO Home path: [C:\Program Files\graylog\collector-sidecar] Config path: [C:\Program Files\graylog\collector-sidecar] Data path: [C:\Program Files\graylog\collector-sidecar\cache\filebeat\data] Logs path: [C:\Program Files\graylog\collector-sidecar\logs]
2017-10-13T11:12:47-04:00 INFO Setup Beat: filebeat; Version: 5.5.1
2017-10-13T11:12:47-04:00 INFO Max Retries set to: 3
2017-10-13T11:12:47-04:00 INFO Activated logstash as output plugin.
2017-10-13T11:12:47-04:00 INFO Publisher name: QD303167
2017-10-13T11:12:47-04:00 INFO Flush Interval set to: 1s
2017-10-13T11:12:47-04:00 INFO Max Bulk Size set to: 2048
2017-10-13T11:12:47-04:00 ERR Not loading modules. Module directory not found: C:\Program Files\graylog\collector-sidecar\module
2017-10-13T11:12:47-04:00 INFO filebeat start running.
2017-10-13T11:12:47-04:00 INFO Registry file set to: C:\Program Files\graylog\collector-sidecar\cache\filebeat\data\registry
2017-10-13T11:12:47-04:00 ERR Error: The service process could not connect to the service controller.
2017-10-13T11:12:47-04:00 INFO Loading registrar data from C:\Program Files\graylog\collector-sidecar\cache\filebeat\data\registry
2017-10-13T11:12:47-04:00 INFO States Loaded from registrar: 1
2017-10-13T11:12:47-04:00 INFO Loading Prospectors: 2
2017-10-13T11:12:47-04:00 INFO Starting Registrar
2017-10-13T11:12:47-04:00 INFO Start sending events to output
2017-10-13T11:12:47-04:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-10-13T11:12:47-04:00 INFO Prospector with previous states loaded: 1
2017-10-13T11:12:47-04:00 INFO Starting prospector of type: log; id: 5460310343517729515
2017-10-13T11:12:47-04:00 INFO Prospector with previous states loaded: 0
2017-10-13T11:12:47-04:00 INFO Starting prospector of type: log; id: 13856082215811419279
2017-10-13T11:12:47-04:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 2
2017-10-13T11:13:17-04:00 INFO Non-zero metrics in the last 30s: publish.events=1 registrar.states.current=1 registrar.states.update=1 registrar.writes=1
2017-10-13T11:13:47-04:00 INFO No non-zero metrics in the last 30s
UPDATE: In the meantime, I ran the “filebeat -c filebeat.yml -configtest” command and my config is ok
In fact I tried with generic application logs (not IIS) as well with Filebeat and it doesn’t work either. How should I write the Path to log if the logs file are for example in c:\logs?
In the example I see /var/logs/*.log but I don’t even understand where this would point on my Windows machine.
Winlogbeat works fine but the configuration is different.
UPDATE: Looking back at the config, I see output to logstash. Is this ok with Graylog? I have no logstash in this setup. However, it is when I modify the Collector configuration that those settings are set. I did not modify it manually.
output:
logstash:
hosts:
- localhost:5045
I tried playing around with the path using globs or based on examples in the filebeat.full example on elastic site.
I even tried updating the Executable file of filebeat to the latest.
I know it should work but the conclusions of our tests is that Filebeat doesn’t work in our POC Sadly. I reviewed all configurations and looked at the logs but can’t find why, This seems not specific to IIS Logs (as IIS Logs are juste log files anyway)
At least there are no problem with Winlogbeat.
Are you sure that the system user running Filebeat has permission to read the directory you’ve configured?
You can also activate debug mode for Filebeat to see what it’s doing in detail.
https://www.elastic.co/guide/en/beats/filebeat/current/enable-filebeat-debugging.html
Good point, but it’s the SYSTEM account that runs filbeat. It has access on all subfolders and files in c:\inetpub\ including logs.
As for running in debug, I saw that but as I’m not explicitly starting Filbeat as it is the Collector_Sidecar that fires Filebeat, I’m not shure where I would add the flags -e -d “publish” in this case.
I also tried setting the logging.level: debug in the filebeat.yml but each time the Collector Sidecar is restarted the filebeat.yml config is reset and this line disappears.
You can stop the Sidecar service with graylog-collector-sidecar.exe -service stop or in the system configuration under services. Then you can start filebeat manually with debug options and the last generated configuration. Please post the output so that we can take a look.
Thanks for the idea, I tried that and found that since I want to test with old logs, that may be the problem. But, then I changed the older_ignore to 5000000 and even I I see “new state added” in the logs", I see nothing in my Graylog input. Here are two parts of the output which I found different :
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv-stdout.2017-06-12.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv-stdout.2017-07-31.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsearch-2017-07-31.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsearch_index_indexing_slowlog.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv-stderr.2017-06-05.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv-stderr.2017-06-14.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv-stderr.2017-09-18.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv.2017-06-12.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsearch-2017-06-05.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsearch-2017-06-06.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsearch-2017-06-14.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsearch-2017-09-18.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsearch_access.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv-stderr.2017-06-12.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv-stdout.2017-06-07.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv.2017-09-18.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv.2017-09-29.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv.2017-10-12.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsearch-2017-06-08.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsearch_index_search_slowlog.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv-stderr.2017-06-07.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv-stderr.2017-06-13.log
2017/10/18 17:22:18.532045 state.go:64: DBG New state added for C:\ELKlogs\elasticsrv-stderr.2017-09-28.log
Here is another part
2017/10/18 18:07:30.915916 prospector_log.go:313: DBG File didn't change: C:\ELKlogs\elasticsrv-stderr.2017-10-11.log
2017/10/18 18:07:30.915916 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv.2017-06-07.log
2017/10/18 18:07:30.915916 prospector_log.go:320: DBG Ignore file because ignore_older reached: C:\ELKlogs\elasticsrv.2017-06-07.log
2017/10/18 18:07:30.915916 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv.2017-10-12.log
2017/10/18 18:07:30.915916 prospector_log.go:259: DBG Update existing file for harvesting: C:\ELKlogs\elasticsrv.2017-10-12.log, offset: 1377
2017/10/18 18:07:30.915916 prospector_log.go:313: DBG File didn't change: C:\ELKlogs\elasticsrv.2017-10-12.log
2017/10/18 18:07:30.915916 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv-stderr.2017-06-06.log
2017/10/18 18:07:30.915916 prospector_log.go:320: DBG Ignore file because ignore_older reached: C:\ELKlogs\elasticsrv-stderr.2017-06-06.log
2017/10/18 18:07:30.916417 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv-stderr.2017-06-08.log
2017/10/18 18:07:30.916417 prospector_log.go:320: DBG Ignore file because ignore_older reached: C:\ELKlogs\elasticsrv-stderr.2017-06-08.log
2017/10/18 18:07:30.916417 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsearch_index_indexing_slowlog.log
2017/10/18 18:07:30.916417 prospector_log.go:320: DBG Ignore file because ignore_older reached:
C:\ELKlogs\elasticsearch_index_indexing_slowlog.log
2017/10/18 18:07:30.916417 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv-stderr.2017-06-13.log
2017/10/18 18:07:30.916417 prospector_log.go:320: DBG Ignore file because ignore_older reached: C:\ELKlogs\elasticsrv-stderr.2017-06-13.log
2017/10/18 18:07:30.916417 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsearch_deprecation.log
2017/10/18 18:07:30.916916 prospector_log.go:259: DBG Update existing file for harvesting: C:\ELKlogs\elasticsearch_deprecation.log, offset: 21149
2017/10/18 18:07:30.916916 prospector_log.go:313: DBG File didn't change: C:\ELKlogs\elasticsearch_deprecation.log
2017/10/18 18:07:30.916916 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv.2017-09-29.log
2017/10/18 18:07:30.916916 prospector_log.go:259: DBG Update existing file for harvesting: C:\ELKlogs\elasticsrv.2017-09-29.log, offset: 617
2017/10/18 18:07:30.916916 prospector_log.go:313: DBG File didn't change: C:\ELKlogs\elasticsrv.2017-09-29.log
2017/10/18 18:07:30.916916 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv-stdout.2017-10-11.log
2017/10/18 18:07:30.916916 prospector_log.go:259: DBG Update existing file for harvesting: C:\ELKlogs\elasticsrv-stdout.2017-10-11.log, offset:
118376
2017/10/18 18:07:30.916916 prospector_log.go:313: DBG File didn't change: C:\ELKlogs\elasticsrv-stdout.2017-10-11.log
2017/10/18 18:07:30.916916 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv-stdout.2017-06-05.log
2017/10/18 18:07:30.916916 prospector_log.go:320: DBG Ignore file because ignore_older reached: C:\ELKlogs\elasticsrv-stdout.2017-06-05.log
2017/10/18 18:07:30.916916 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv-stdout.2017-09-18.log
2017/10/18 18:07:30.916916 prospector_log.go:259: DBG Update existing file for harvesting: C:\ELKlogs\elasticsrv-stdout.2017-09-18.log, offset:
4321
2017/10/18 18:07:30.916916 prospector_log.go:313: DBG File didn't change: C:\ELKlogs\elasticsrv-stdout.2017-09-18.log
2017/10/18 18:07:30.916916 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv.2017-06-06.log
2017/10/18 18:07:30.916916 prospector_log.go:320: DBG Ignore file because ignore_older reached: C:\ELKlogs\elasticsrv.2017-06-06.log
2017/10/18 18:07:30.916916 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv.2017-06-13.log
2017/10/18 18:07:30.917418 prospector_log.go:320: DBG Ignore file because ignore_older reached: C:\ELKlogs\elasticsrv.2017-06-13.log
2017/10/18 18:07:30.917418 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv-stderr.2017-09-29.log
2017/10/18 18:07:30.917418 prospector_log.go:259: DBG Update existing file for harvesting: C:\ELKlogs\elasticsrv-stderr.2017-09-29.log, offset: 65
2017/10/18 18:07:30.917418 prospector_log.go:313: DBG File didn't change: C:\ELKlogs\elasticsrv-stderr.2017-09-29.log
2017/10/18 18:07:30.917418 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv-stdout.2017-07-31.log
2017/10/18 18:07:30.917418 prospector_log.go:320: DBG Ignore file because ignore_older reached: C:\ELKlogs\elasticsrv-stdout.2017-07-31.log
2017/10/18 18:07:30.917418 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv-stdout.2017-09-28.log
2017/10/18 18:07:30.917418 prospector_log.go:259: DBG Update existing file for harvesting: C:\ELKlogs\elasticsrv-stdout.2017-09-28.log, offset:
1708262
2017/10/18 18:07:30.917418 prospector_log.go:313: DBG File didn't change: C:\ELKlogs\elasticsrv-stdout.2017-09-28.log
2017/10/18 18:07:30.917418 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsrv.2017-09-28.log
2017/10/18 18:07:30.917418 prospector_log.go:259: DBG Update existing file for harvesting: C:\ELKlogs\elasticsrv.2017-09-28.log, offset: 5747
2017/10/18 18:07:30.917418 prospector_log.go:313: DBG File didn't change: C:\ELKlogs\elasticsrv.2017-09-28.log
2017/10/18 18:07:30.917919 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsearch-2017-06-05.log
2017/10/18 18:07:30.917919 prospector_log.go:320: DBG Ignore file because ignore_older reached: C:\ELKlogs\elasticsearch-2017-06-05.log
2017/10/18 18:07:30.917919 prospector_log.go:226: DBG Check file for harvesting: C:\ELKlogs\elasticsearch.log
2017/10/18 18:07:30.917919 prospector_log.go:259: DBG Update existing file for harvesting: C:\ELKlogs\elasticsearch.log, offset: 1700501
2017/10/18 18:07:30.917919 prospector_log.go:313: DBG File didn't change: C:\ELKlogs\elasticsearch.log
2017/10/18 18:07:30.917919 prospector_log.go:91: DBG Prospector states cleaned up. Before: 56, After: 56
2017/10/18 18:07:32.577484 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/10/18 18:07:37.579416 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/10/18 18:07:37.587265 prospector.go:183: DBG Run prospector
2017/10/18 18:07:37.587265 prospector_log.go:70: DBG Start next scan
2017/10/18 18:07:37.588246 prospector_log.go:226: DBG Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC1\u_ex171012.log
2017/10/18 18:07:37.589227 prospector_log.go:259: DBG Update existing file for harvesting: C:\inetpub\logs\LogFiles\W3SVC1\u_ex171012.log,
offset: 3568
2017/10/18 18:07:37.589227 prospector_log.go:313: DBG File didn't change: C:\inetpub\logs\LogFiles\W3SVC1\u_ex171012.log
2017/10/18 18:07:37.589227 prospector_log.go:226: DBG Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC1\u_ex171016.log
2017/10/18 18:07:37.589227 prospector_log.go:259: DBG Update existing file for harvesting: C:\inetpub\logs\LogFiles\W3SVC1\u_ex171016.log,
offset: 1296
2017/10/18 18:07:37.589227 prospector_log.go:313: DBG File didn't change: C:\inetpub\logs\LogFiles\W3SVC1\u_ex171016.log
2017/10/18 18:07:37.589227 prospector_log.go:91: DBG Prospector states cleaned up. Before: 2, After: 2
Nothing in the logs that explains your problem. Could check the log for messages like: INFO Connecting error publishing events (retrying): dial tcp xxxx:5045: getsockopt: connection refused
Maybe it’s a simple networking problem, that your virtual machine is not reachable from the outside?
I had no issues getting IIS logs to graylog using NXLOG. If that is an option I could work with you to get it setup and send you a working config. Bonus NXLOG works fine with collector and can reformat as well as ship to a gelf input for you. Not needing extractors on the graylog side is super nice. Let me know.
I would be interested Matt. I am not importing IIS logs right now, but I would be interested in seeing what you have used for a conf file.
would you like the config here or in PM?
Just post it here so that other users having the same question don’t have to bother you each individually. 
Good call Jochen. Here you go Sparky. Keep in mind anything within this config is “tunable” depending on what fields you choose to use. This is a mostly default format with “x-forwarded-for” added in our case as we need that passed for our own uses. You can easily remove that if not needed in your setup. I have also set “SavePos” to false so if either an input was turned off, graylog went down, or network availability became a problem we wouldn’t get a sudden influx of logs when things were back online. Turn “SavPos” on at your own risk as a buildup of logs on the IIS servers suddenly being shipped to graylog can sometimes quickly overwhelm graylog/elasticsearch depending on your setup and what it can handle. In our case 10k per second sustained isn’t a problem but when all of the sudden graylog is being shipped 100k+ per second it can turn into an issue very quickly. The following config should get you off the ground with NXLOG. Keep in mind you’ll have to set the “Input” file path to where your logs are stored and point the “Output” to correspond to your graylog input IP and Port. Those should be easy enough. Remember to make your Graylog input GELF if you want this config to work. Other output types are supported if you want to use them but this IMO seems to be the ideal setup. Let me know if you have any questions or need further help with the setup. Here is the config.
## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension _syslog>
Module xm_gelf
</Extension>
<Extension w3c>
Module xm_csv
Fields $date $time $s-sitename $cs-method $cs-uri-stem $cs-uri-query $cs-username $c-ip $cs-user-agent $cs-referer $cs-host $sc-status $sc-substatus $sc-win32-status $sc-bytes $cs-bytes $time-taken $x-forwarded-for
FieldTypes string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer, string
Delimiter ' '
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input IISIN>
Module im_file
File 'W:\logfiles\\u_ex*.log'
SavePos False
Recursive True
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$SourceName = "IIS"; \
$EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ"); \
$Message = to_json(); \
}
</Input>
<Output IISOUT>
Module om_tcp
Host <GRAYLOG INPUT IP HERE>
Port <GRAYLOG INPUT PORT HERE>
OutputType GELF_TCP
</Output>
<Route 1>
Path IISIN => IISOUT
</Route>I’ve been on another project for a couple of days but will be checking about potential network problems today.
Strange thing is Winlogbeat works fin but not Filebeat between the same two machines.
I will also check if NXLog could be an alternative.
Thanks for all you ideas guys
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.