Apache logs - parsing the request

[gerdesj (Jon Gerdes)]

Out of the box, Graylog is very Apache combined log friendly. A common log entry has a request URI with a URL followed by &key1=value1&key2=value2 etc.

You can’t use the usual KV splitter in an input extractor because they don’t quite follow the usual pattern of k1=v1 k2=v2 etc because those are space separated instead of & separated.

Now I’ve put my thoughts into a post it looks like I might mutate the & into a space char or something like that and then KV split.

My question was going to be should I do this in an extractor or a pipeline?

[gsmith]

Hey @gerdesj

pipeline has my vote,

Something like this.

rule "Cisco KV"
when
  has_field("cisco_msg")
then
  set_fields(
    fields:
      key_value(
        value: to_string(to_string($message.cisco_msg)),
        delimiters:",",
        kv_delimiters:":"
  ));
end

[gerdesj (Jon Gerdes)]

It looks like that could be extended to extractor KV splitting too.

Thanks for the heads up. I hadn’t noticed that pipelines have a key_value splitter function.

[ihe]

Don’t go ever for extractors. They will be deprecated in future versions of Graylog, not announced yet when exactly. Always go for Pipelines.

You can customize your logging done by apache - and then have nice JSON to ingest into Graylog. This is the way how I do it with Apache and Squid.

[faen]

I do use an extractor with my Apache input:

%{HOSTNAME:DstURL}:%{POSINT:DstPort} %{COMBINEDAPACHELOG}

Then I use a couple of pipeline rules to further sort out and enrich the data. I guess I could move that into an initial pipeline rule and ditch the extractor. Project for another time, though.

[gerdesj (Jon Gerdes)]

I have this in the pipeline:

rule "Apache combined log"
when
    has_field("message")
then
    set_fields(
        grok(
            pattern: "%{COMBINEDAPACHELOG}", 
            value: to_string($message.message), 
            only_named_captures: true
        )
    );
end

After that I attack the referrer field:

rule "local rule"
when
    contains(
        value: to_string($message.referrer), 
        search: "text_to_search", 
        ignore_case: false
    )
then

   let referrer_fields = grok("%{DATA:left}\\?%{GREEDYDATA:right}", to_string($message.referrer));

    set_fields(
        fields:
            key_value(
                value: urldecode(to_string(referrer_fields.right)),
                delimiters: "&",
                kv_delimiters: "="
            ),
        prefix: "LOCAL_PREFIX_"
    );
    
end