Filebeat: Using multiline to split Microsoft Defender for Endpoint events into separate messages in Graylog

[SalC]

I’m grabbing Microsoft Defender for Endpoint events using a powershell script, which outputs all the events for a given duration to a single file. I’m then reading the file into Graylog using filebeat on Windows. The entries are on separate lines with [CR][LF] at the end, so I think I need to use a multiline pattern in the filebeat config, otherwise each line is a separate message.

This works fine, if there’s only 1 event in the log - the problem I’m running into, is that if there’s multiple events in the log, they all go into 1 message. Any ideas on how to tackle this? Maybe using a pipeline would be a better idea? Each event starts with “id” and ends with },

I’m using the following in the filebeat config:

multiline.pattern: '\"id\"'
multiline.negate: true
multiline.match: after

I tried a multiline_flush_pattern, but that just results in nothing making it into graylog
multiline.flush_pattern: '\}\,'

Thanks for any help/advice!

Here’s an edited sample of a log file:

[
    {
        "id":  "12345",
        "providerAlertId":  "1234567",
        "incidentId":  "29",
        "status":  "resolved",
        "severity":  "medium",
        "classification":  null,
        "determination":  null,
        "serviceSource":  "microsoftDefenderForEndpoint",
        "detectionSource":  "antivirus",
        "detectorId":  "abcdefg12345",
        "tenantId":  "abcdefg12345",
        "title":  "Bloodhound post-exploitation tool",
        "description":  "Bloodhound, a post-exploitation open-source reconnaissance tool, has been detected on this device. Bloodhound has been used in a wide range of documented attacks, including attacks involving state-sponsored groups and groups associated with ransomware campaigns. An attacker might be attempting to collect information about users, user sessions, groups, accounts, domain controller properties and permissions. Detections of Bloodhound tools and activity should be thoroughly investigated.",
        "recommendedActions":  "A. Validate the alert.\n1. Contact the user who ran the tool to verify whether the activity was legitimate and inspect the endpoints for suspicious behavior.\n2. Locate unfamiliar processes in the process tree. Check files for prevalence, their locations, and digital signatures.\n3. Submit relevant files for deep analysis and review file behaviors.\n4. Identify unusual system activities with system owners.\n\nB. Scope the incident. Find related devices, network addresses, and files in the incident graph.\n\nC. Contain and mitigate the breach. Stop suspicious processes, isolate affected devices, decommission compromised accounts, or reset passwords, block IP addresses and URLs, and install security updates.\n\nD. Contact your incident response team, or contact Microsoft support for investigation and remediation services.",
        "category":  "SuspiciousActivity",
        "assignedTo":  "Automation",
        "alertWebUrl":  "https://security.microsoft.com/alerts/blahblahblahblah",
        "incidentWebUrl":  "https://security.microsoft.com/incidents/blahblahblah",
        "actorDisplayName":  null,
        "threatDisplayName":  "HackTool:MSIL/SharpHound!MSR",
        "threatFamilyName":  "SharpHound",
        "mitreTechniques":  [
                                "T1087"
                            ],
        "createdDateTime":  "2022-12-21T18:40:21.7967173Z",
        "lastUpdateDateTime":  "2022-12-21T18:52:56.91Z",
        "resolvedDateTime":  "2022-12-21T18:52:56.7897235Z",
        "firstActivityDateTime":  "2022-12-21T18:38:49.4766284Z",
        "lastActivityDateTime":  "2022-12-21T18:48:11.3281966Z",

    },
    
	{
        "id":  "1234567",
        "providerAlertId":  "1234567",
        "incidentId":  "29",
        "status":  "resolved",
        "severity":  "medium",
        "classification":  null,
        "determination":  null,
        "serviceSource":  "microsoftDefenderForEndpoint",
        "detectionSource":  "microsoftDefenderForEndpoint",
        "detectorId":  "1a2b3c4d",
        "tenantId":  "1a2b3c4d5e",
        "title":  "\u0027SharpHound\u0027 malware was detected during lateral movement",
        "description":  "\u0027DefenderObservation.ThreatFamily\u0027 malware was detected on this device. An attacker might be attempting to move laterally to this device from another device on the network.",
        "recommendedActions":  "A. Validate the alert.\n1. Inspect the file, process, commandline or amsi buffer that was detected.\n2. Check for other suspicious activities in the machine timeline.​\n3. Locate unfamiliar processes in the process tree. Check files for prevalence, their locations, and digital signatures.​\n4. Submit relevant files for deep analysis and review file behaviors. ​\n5. Identify unusual system activity with system owners.​\n​\nB. Scope the incident. Find related machines, network addresses, and files in the incident graph.​\n​\nC. Contain and mitigate the breach. Stop suspicious processes, isolate affected machines, decommission compromised accounts or reset passwords, block IP addresses and URLs, and install security updates.​\n​\nD. Contact your incident response team, or contact Microsoft support for investigation and remediation services.",
        "category":  "LateralMovement",
        "assignedTo":  "Automation",
        "alertWebUrl":  "https://security.microsoft.com/alerts/12345_12345",
        "incidentWebUrl":  "https://security.microsoft.com/incidents/29?tid=1ab3ddddd",
        "actorDisplayName":  null,
        "threatDisplayName":  null,
        "threatFamilyName":  null,
        "mitreTechniques":  [
                                "T1021.001",
                                "T1078"
                            ],
        "createdDateTime":  "2022-12-21T18:40:49.4268815Z",
        "lastUpdateDateTime":  "2022-12-21T18:52:56.91Z",
        "resolvedDateTime":  "2022-12-21T18:52:56.7897235Z",
        "firstActivityDateTime":  "2022-12-21T18:06:37.0236159Z",
        "lastActivityDateTime":  "2022-12-21T18:47:23.7266277Z",
        
    },

[gsmith]

Hello @SalC Welcome back.

So I had this configured awhile back , I believe your talking about Microsoft Antimalware?

Example:

I didnt use a powershell script, tbh im cureiuous on the script your using. Is it posible to show your PS?

[SalC]

Hey there, thanks for the reply. It’s Microsoft Defender for Endpoint (EDR). Here’s the Powershell script that’s pulling in the logs:

# Returns Alerts created in the indicated time.

$token = ./Get-Token.ps1       #run the script Get-Token.ps1  - make sure you are running this script from the same folder of Get-Token.ps1

# Get Alert from the last 5 minutes.
$dateTime = (Get-Date).ToUniversalTime().AddMinutes(-5).ToString("o")

# The URL contains the type of query and the time filter we create above
# Read more about [other query options and filters](get-alerts.md).
# $url = "https://api.securitycenter.microsoft.com/api/alerts?`$filter=alertCreationTime ge $dateTime"
# $url = "https://graph.microsoft.com/beta/security/alerts_v2?$filter=alertCreationTime ge $($dateTime)"
$url = "https://graph.microsoft.com/beta/security/alerts_v2?`$filter=createdDateTime ge $dateTime"

# Set the WebRequest headers
$headers = @{
    'Content-Type' = 'application/json'
    Accept = 'application/json'
    Authorization = "Bearer $token"
}

# Send the webrequest and get the results.
$response = Invoke-WebRequest -Method Get -Uri $url -Headers $headers -ErrorAction Stop

# Extract the alerts from the results.
$alerts =  ($response | ConvertFrom-Json).value | ConvertTo-Json

# Get string with the execution time. We concatenate that string to the output file to avoid overwrite the file
$dateTimeForFileName = Get-Date -Format o | foreach {$_ -replace ":", "."}

# Save the result as json and as csv
$outputJsonPath = "Logs/Latest Alerts $dateTimeForFileName.log"
$outputCsvPath = "Logs/Latest Alerts $dateTimeForFileName.csv"

Out-File -FilePath $outputJsonPath -InputObject $alerts
# ($alerts | ConvertFrom-Json) | Export-CSV $outputCsvPath -NoTypeInformation
($alerts | ConvertFrom-Json)

[SalC]

PS, here’s a link to Microsoft which is what the script is based from:

Hello World for Microsoft Defender for Endpoint API

[gsmith]

Hey,

Oh i see now , thanks for the link.

@SalC

Any ideas on how to tackle this? M

So it doesnt now when to stop grouping lines togethers. I see using “id” for your pattern. multiline.pattern: '\"id\"', Lookingover the logs I noticed "[]" & "{}" have you tried any other configuration patterns? Perhaps something similar like multiline.pattern: '^\[', just an idea.

[SalC]

@gsmith

‘^[’

Yep, and I just tried multiline.pattern: ‘^[’

Filebeat still doesn’t know when to stop reading and start a new entry … doh!

I’m going to try posting over on the Filebeat forum, and I’ll update this post if I have any luck.