Windows Event Logs shipped through Winlogbeat contain unparsed data within message field

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Hi - I would to report an issue very identical to one already documented here
[winlogbeat-is-shipping-windows-events-with-placeholders-instead-of-actual-data]

However, since the thread is already closed, I was unable to add to it.
Summarily, here’s what I’m seeing.

  • the message field in ingested log shows unparsed data - in terms of variables or placeholders.
  • while the actual winlogbeat fields that correspond to same data contain the expected values

And the 2 fields in below screenshot do show the correct expected value - "true’
image

Same event ID 3 hours before from the same machine, showing the correct expected values in ‘message’ field

And this is the same
2. Describe your environment:

  • OS Information:
  • Graylog 4.3.9, running on arm64
  • Multiple Windows VM’s - a mix of Windows 10, 11 and Windows server 2016/2019 machines
  • All VM’s are shipping Event logs (both native and Sysmon) to Graylog through either Sidecar 1.1 or 1.2
  • The issue is being exhibited on the only Windows 11 machine in the network
  • Package Version:
    Sidecar 1.1.0 / 1.2.0 >> Issue exhibited in both
    Winlogbeat 7.11.1

  • Service logs, configurations, and environment variables:
    Please advise if any additional information or logs are needed

3. What steps have you already taken to try and solve the problem?
Now i did find something which might or might not be related.

  • the last major change just before this broke was install of upgrade ‘Windows 11, version 22H2’
  • The point where the events break is indicated in red line (between 9:37 - 9:54 UTC)

image

  • I have ruled out Sidecar version as a factor, since both 1.1.0 and 1.2.0 (as well as 1.3.0.beta) result in same issue.
  • What is remaining to be seen is - If an uninstall of this major upgrade makes any difference ?
    I’ll try this in the next 2-3 days and update

4. How can the community help?
Is there something that I can check ? Any debug logging or anything else ?

Further Update:

  • This apparently does look like an issue between 22H2 update for Windows 10/11, and winlogbeat.
  • Possibly the xml format (My guess)
    Based on this post in elastic forums
    winlogbeat-8-5-and-windows-11-22h2

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

Hi, I’m the creator of the referenced post above, just to confirm that I am still having the issue with Windows 10 22H1, I have not traced the root of the issue very thoroughly but hope someone will have an idea, thank you.

Hi - What editon/build of windows are seeing this issue on ? Any idea what led to it ?

I just double checked against my windows machine and I am not seeing this problem. One thing I did not note on the other thread but very likely could be the culprit - I am running winlogbeat 7.15.1 in this windows machine. (I know we are not supposed to go beyond 7.10 on Elastic)

Try swapping out the winlogbeat.exe on a test machine to at least 7.15.1 to see if it solves the problem.

I would note that in some instances, winlogbeat versions newer than 7.10 may change default field names so keep an eye on that!

1 Like

Hi, thanks for taking a look and for your suggestions.
I tried swapping winlogbeat.exe binary to 7.17.0 and 8.5.2 >> same issue with both.

Was searching around more, an came across more than 1 threads on different forums about multiple people reporting the exact same issue after upgrading to Windows 11 22H2 build.
Ref:
https://discuss.elastic.co/t/winlogbeat-8-5-and-windows-11-22h2/318676/4
https://github.com/Graylog2/collector-sidecar/issues/449

You mentioned you’re not running into this issue, are you also on the same build of Win 11 - 22H2?
I have machines that run both Windows 10 22H2 and Windows 11 22H2 , and the issue manifests ONLY in windows 11 build.

HI -
I am seeing the same issue with Workstations that are running Windows 11 22H2 OS build 22621.819. I currently have two workstations that are having this issue both on the same OS build version. Windows 11 21H2 Build 22000.1219 seem to be fine.
Screenshot 2022-11-30 135119

That’s what I get when I do a quick read - I do not currently have Win11 in my environment so I can’t test yet. Looking to change that.

There is a post (yours maybe) on the Elastic side where this issue resides - It is also placed in Graylog’s gitub - hopefully it will gain some traction!

1 Like

Thanks for your time on this, @tmacgbay .

Yes, at this point - this is pretty much over to folks at Elastic to come up with a version fix for Winlogbeat.
I reckon this will become quite widespread with the increased adoption of this build of Win11, as well as there are a lot many other SIEM solutions besides Graylog that use Winlogbeat to forward Event logs.

@rsanghoi

I’ve been watching this thread, I was going to upgrade a couple of servers to Win 11 but now im going to wait. Thanks for the update :+1:

1 Like

Hi everyone, I wish I had a better answer than this, but unfortunately, this is a known bug that exists within winlogbeat when collecting logs from Windows 11. A member of my team put in an issue with the Beats team for this specific issue, but we have not heard back around prioritization or timing of a fix. The only workaround we have found at this point is a painful one, switching to nxlog for windows event collection. I know this is an untenable solution, and we are working as hard as we can to get the Beats teams to tackle this issue ASAP. Also, if you are utilizing the Graylog Illuminate content for your Windows log processing, the swap of the agent would not require any further work on your part as Illuminate is built to work with both winlogbeat and nxlog.

3 Likes

@joe.gross Thanks for the update

Thanks @joe.gross , appreciate the update, and thanks for raising this with the beats team.
I’ll try switching to nxlog in the interim.
I have an Input Pipeline processor as well to rename the winlogbeatxxxxx fields, I guess I’ll have to build another pipeline processor for this. But that’s alright :slight_smile: - will perhaps learn something new.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.