Pipeline Issue - set_fields w/ JSON not working

[abraxas]

I am using filebeat to read a file which is getting logs in JSON format. It is putting one JSON object per line.

Here is a copy of one of the items from a line of the text file being watched by FileBeat:

{"CreationTime":"2019-11-01T08:32:17","Id":"xxxxxxx-1534-4539-a7e9-xxxxxf","Operation":"FileSyncDownloadedPartial","OrganizationId":"exxxxx-xxx-xxxxxxxxb","RecordType":6,"UserKey":"i:0h.f|membership|10xxxxxxxxxxxxxxd@live.com","UserType":0,"Version":1,"Workload":"OneDrive","ClientIP":"xxx.xxx.x66","ObjectId":"https:\/\/nxxxxxxxxxxxxxy.sharepoint.com\/personal\/sxxxxxxe_com\/Documents\/4xxxxxxxxxy\/HOWxxxxxxxxxx\/Euxenxxxxtxxxx6.pdf","UserId":"xxxxxxxxxxxxx@xxxxx.com","CorrelationId":"4xxxxxxxxxxxx632359fb88c8","EventSource":"SharePoint","ItemType":"File","ListId":"8cxxxx-x-xxxxxx-x24","ListItemUniqueId":"638xxxxxxxx-x-xxxxxxx-xxxx55","Site":"34xxxxxxxx-xxxxx-xxxxxx-ade21","UserAgent":"Microsoft Office Upload Center 2014 (16.0.4849) Windows NT 6.1","WebId":"6863xxxxxxxxxx-xxxxx-xxxxxxxx8245","SourceFileExtension":"pdf","SiteUrl":"https:\/\/xxxxxxxxxxxxxxxx.sharepoint.com\/personal\/xxxxxxxxxx_com\/","SourceFileName":"Excxxxxxxxxxxxxxxx.pdf","SourceRelativeUrl":"Dxxxxxxxxxx\/4Rexxxxxxxxxxx\/HOxxxxxxxxxxxxx"}

My Filebeat Config for this endpoint is using the following:

output.logstash:
   hosts: ["graylog.xxxxxxxxxxx.com:9876"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\filebeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
filebeat.inputs:
- paths:
   - C:\Users\xxxxxxxx\Desktop\Log_Directory\AuditRecords.txt
  input_type: log
  type: log
#Ignore these, was testing earlier.
#  json.keys_under_root: true
#filebeat.processors:
#  - decode_json_fields:
#    fields: ['message']
#    target: json
#  json.keys_under_root: true
#  json.add_error_key: true
#  json.ignore_decoding_error: true
#  json.message_key: message

This is properly (seemingly) creating messages where “$message.message” matches the JSON of the line, here’s an example of what ends up in Graylog:

However, I’ve yet to find a good way to actually extract the JSON from $message.message and have it create the appropriate fields. Not all messages have the same syntax but they are all json…so I can’t do a manual set of extractors but I’m not finding a way to solve this.

An example of a pipeline I’ve tried is the following (which adds the jtest:true item in the screenshot above):

rule "extract O365 data from JSON"
when
  contains(to_string($message.filebeat_source),"C:\\Users\\xxxxxx\\Desktop\\Log_Directory\\AuditRecords.txt",true)
then
  let json_data = parse_json(to_string($message.message));
  let jmap = to_map(json_data);
  let jtest = is_json(json_data);
  set_field("jtest",jtest);
  set_fields(jmap);
end

Any help or guidance would be EXTREMELY appreciated.

[jan (Jan Doberstein)]

a really simple json parser is something like:

rule "extract-json"
when
    starts_with(to_string($message.message), "{") && ends_with(to_string($message.message), "}")
then
    let json = parse_json(to_string($message.message));
    let map = to_map(json);
    set_fields(map);
end

Not sure why this is not working in your case - from my knowledge it should.

Which Graylog version are you using?

[abraxas]

Okay, so this was a really, really weird one. Apparently the file I was writing had null-bytes inserted between each character… It came down to how Powershell was writing this data to file:

ForEach ($result in $results.auditdata) {$result >> $OutputFile}

versus

ForEach ($result in $results.auditdata) {$result | Out-File -Append -Encoding Ascii $OutputFile}

For whatever reason the >> in the command was inserting a %00 between each character which I was able to figure out by sending it over a socket like below:

I’m confused why the is_json() function returned true? but it explains why the data looked correct but in fact wasn’t.

[jan (Jan Doberstein)]

do you mind opening a bug report over at github for this?

That is something that should work correct.

[abraxas]

I’m busy with some work things currently and won’t have much time to sit down and put it all together until a week or two from now. I’ll throw a reminder to check in then but feel free to submit one in the meantime if you’d like it before then; otherwise I’ll try to get it in when I have some downtime.