JSON Extractor and Field Converter

[juaromu (Juan J Romero)]

Hi guys:

Using JSON log format in Apache and sending access logs to Graylog via FileBeats. In httpd.conf file:

LogFormat “{ “time”:”%{%Y-%m-%d}tT%{%T}t.%{msec_frac}tZ", “process”:"%D", “filename”:"%f", “src_ip”:"%a", “webserver”:"%V", “webserver_port”:"%p", “request”:"%U", “query”:"%q", “method”:"%m", “status”:"%>s", “userAgent”:"%{User-agent}i", “referer”:"%{Referer}i", “bytes_sent”:"%B", “response_time”:"%T"}, " combined

In Graylog’s Beats input a “simple” JSON extarctor does the trick and all good.

The little problem I’m seeing is that fields such as BYTES, or RESPONSE TIME are strored as string and therefore stats or graphs don’t apply.

Is there any way to “reformat/convert” fields with this setup? Could a pipeline be defined after the message gets processed by the extractor?

Thanks a lot

Juan.

[juaromu (Juan J Romero)]

Yes, a pipeline can sort this out:

rule “WAF Access Logs Convert Fields”
when
true
then
// the following date format assumes there’s no time zone in the string
let new_bytes = to_long($message.bytes_sent);
set_field(“bytes”, new_bytes);
end

The new field “bytes” now stores the value as integer.

Cheers
Juan.