Replace or Trim from message field in pipeline rule

[rfinney]

Hi,

I’m sending logs from Fortinet FortiAnalyzer to Graylog in CEF formatting. I’m having issues with forwarded logs from a certain type of device. The first issue is that the field values have " .ad" added to the end of them. The second issue is the timestamp is coming in as UTC but being seen as EDT/EST so logs are ahead 4 hours.

I setup a raw TCP input to send the CEF TCP messages to, they look like this:

Sep 12 17:20:15 ems CEF:0|Fortinet|FortiClient-EMS|1.2|0| traffic|5|start=Sep 12 2019 17:20:15 ad.logver=N/A cat=traffic deviceSeverity=notice externalID=N/A dhost=N/A ad.pcdomain=subdomain.company.com ad.uid=N/A deviceExternalId=FCT8104243435915 ad.fgtserial=N/A ad.emsserial=FCTEMS0000000824 ad.regip=N/A shost=chrome.exe ad.srcproduct=Chrome src=10.10.100.101 spt=N/A ad.direction=outbound dst=N/A ad.remotename=yahoo.com dpt=443 duser=remployee@subdomain.company.com proto=6 in=N/A out=N/A ad.utmaction=userbrowsed ad.utmevent=webfilter ad.threat=Search ad.vd=root ad.fctver=1.0.1.0020 ad.os=cros ad.usingpolicy=RR Chromebook Student app=https request=/ ad.userinitiated=1 ad.browsetime=0.12

When sending to the CEF input it’s giving me:

start
Sep 12 2019 17:20:15 ad.
timestamp
2019-09-12 17:20:15.000 -04:00

Can someone show me an example of how to remove " ad." ?

Having read @jan blog https://jalogisch.de/2018/working-with-cisco-asa-nexus-on-graylog/
I think I can get the timezone straightened out with this once I have the " ad." removed.

rule “type fortinet-faz-utc-ems-chromebook”
// we want to convert utc to edt timestamp
when
has_field(“os”) && to_string($message.os) == “cros ad.” &&
grok(pattern: “%{MONTH} +%{MONTHDAY}(?: %{YEAR})? %{TIME}”, value:to_string($message.start)).matches == true
then
let time = parse_date(value:to_string($message.start), pattern:“yyyy MMM dd HH:mm:ss.SSS”, timezone:“UTC”);
set_field(“timestamp”,time);

end

[rfinney]

I’m thinking substring or trim_value_chars might be able to remove the " ad."

[rfinney]

Thanks to @jan great post at https://jalogisch.de/2018/working-with-cisco-asa-nexus-on-graylog/
and https://anotheritblog.net/tag/graylog/ I’ve got this working.

What I’m currently working with…

• Remove certain number of characters from the end of a string
• Set timestamp without correct timezone to UTC so it displays correctly in local est/edt

Pipeline stage 0

rule “type fortinet-faz-utc-ems-chromebook-os”
// we want to remove " .ad" from the os field
when
has_field(“os”) && to_string($message.os) == “cros ad.”
then
let var_os1 = to_string($message.os);
remove_field(“os”);
set_field(“os”, substring(var_os1, 0, -4));
end

rule “type fortinet-faz-utc-ems-chromebook-start”
// we want to remove " .ad" from the start field
when
has_field(“start”) &&
grok(pattern: “%{MONTH} +%{MONTHDAY}(?: %{YEAR})? %{TIME} ad.”, value:to_string($message.start)).matches == true
then
let var_start1 = to_string($message.start);
remove_field(“start”);
set_field(“start”, substring(var_start1, 0, -4));
end

Pipeline Stage 1

rule “type fortinet-faz-utc-ems-chromebook-timestamp-2n”
// we want to convert timestamp without timezone into utc two number day
when
has_field(“start”) &&
grok(pattern: “%{MONTH} %{MONTHDAY} %{YEAR} %{TIME}”, value:to_string($message.start)).matches == true
then
let time = parse_date(value:to_string($message.start), pattern:“MMM dd yyyy HH:mm:ss”, timezone:“UTC”);
set_field(“timestamp”,time);
end

rule “type fortinet-faz-utc-ems-chromebook-timestamp-1n”
// we want to convert timestamp without timezone into utc one number day
when
has_field(“start”) &&
grok(pattern: “%{MONTH} %{MONTHDAY} %{YEAR} %{TIME}”, value:to_string($message.start)).matches == true
then
let time = parse_date(value:to_string($message.start), pattern:“MMM d yyyy HH:mm:ss”, timezone:“UTC”);
set_field(“timestamp”,time);
end