[Graylog sidecar] Winlogbeat working but no logs can be retrieved in Graylog GUI

Hi, some time ago I’ve set up graylog sidecar 1.3.0 on a Windows host to test out winlogbeat. It has been working for quite a while. Today I wanted to do something more with these logs and I logged in to see that Graylog hasn’t received any messsages for 9 days from this host.
Sidecar status was “running”, so I logged onto the host and checked the logs, The sidecar logs showed that everything was working properly and I also saw that there were winlogbeat logs files created for every day (including the days when it supposedly stopped working). I also reviewed the logs and nothing unusual there, just normal event logs, no errors.
I have then restarted the sidecar for good measure and checked the logs again: nothing. According to the logs, everything was working just fine.

So I went for the ultimate solution of reinstalling graylog sidecar completely and setting it up all over again. And so I did, graylog-sidecar (now version 1.4.0) is up and running and still no errors and still no logs.

ONE INTERESTING THING that I noticed was that when I started the new sidecar, in the top right corner I saw that there was a huge amount of logs incoming per second. It jumped from 300/s to 2100/s for a couple of seconds. SURELY these were winlogbeat logs, but why they don’t appear in the GUI? I have no idea and hence I came here asking for your help.

Posting some info that I guess is relevant:
Graylog version: Graylog 4.3.7+05bccc

sidecar.yml configuration:

server_url: "https://x.x.x.x:9000/api/"
server_api_token: "my_token_wont_share_it"
node_id: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
node_name: ""
update_interval: 10
tls_skip_verify: true
send_status: true

Winlogbeat configuration:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["x.x.x.x:5044"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
winlogbeat:
  event_logs:
   - name: Application
   - name: System
   - name: Security
   - name: DFS Replication
   - name: Directory Service
   - name: DNS Server
   - name: File Replication Service

Input configuration:

bind_address: 0.0.0.0
no_beats_prefix: false
number_worker_threads: 6
override_source: <empty>
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password:********

So that’s all, I also wanted to emphasise that these settings were used initially when things worked, nothing here has changed. I’ve also checked jira to make sure that I wasn’t fumbling around with stuff the day it broke so I am completely out of ideas.

Search for the logs received by that input. I sometimes saw very old logs beeing ingested by winlogbeat, and the timestamp correctly in graylog - the time when the log was written in Windows and not the arrival-time in Graylog.
If you find logs: have a look in which stream they are. Maybe your routing is broken?

1 Like

I am viewing the logs through graylog-sidecar, so I see them by clicking “Show messages” byt the associated host:


And if there were any old messages, I’d see them after filtering by “All time”, so unfortunately that’s not the issue.

Try going to the beats input and click the show messages button from there. Set it to all messages.

hello,

Just chiming in, by chance did you try tcpdump on Graylog /w host and port?

1 Like

@gsmith @chris.black Just moments ago I saw the messages coming in as soon as I logged into the Graylog GUI. I just… don’t know… Glad it is working again, not glad if it would happen again since this situation resolved itself in a way I don’t understand. Closing this issue of course but this is the weirdest thing that I’ve experienced with Graylog in a while.

2 Likes