Pipeline rules for apache logs

Graylog Central (peer support)
,
1

What is the best practice for setting up rules to parse different log types? Ex: Apache Error and Apache Access

Currently they are pushed to Graylog as type ‘apache-log’ parsing is done via the following:

rule "apache logs"
when
  has_field("type") AND $message.type == "apache-log"
then
  let message_field = to_string($message.message);
  set_field("full_message", message_field);
  let parsed_fields = grok(pattern: "%{COMBINEDAPACHELOG}", value: message_field, only_named_captures: true);
  set_fields(parsed_fields);
  let parsed_fields2 = grok(pattern: "%{HTTPD_ERRORLOG}", value: message_field, only_named_captures: true);
  set_fields(parsed_fields2);
  set_field("timestamp", flex_parse_date(value: to_string(parsed_fields2.timestamp), timezone: "America/Chicago") );
end

Should this be split into 2 rules instead of handling in 1?

2

do you use filebeat as collector?

That should allow you to check the filename - that might indicate if this error or access log and then build one rule for each log type.

The idea of the pipelines is to run heavy extractors only on the messages that needs that - and GROK Patterns are heavy lifters.

3

I think I answered my own question.

I am using filebeat so I can set the type field based on the inputs. I will then create 2 rules in the pipeline to handle each log type.

I did run into another issue. The pipeline only seems to work on the all messages stream. I have another stream (Move message) setup which moves the message to a different index. The “Move message” stream is working and moving messages to a different index however, if I connect the pipeline to the “Move message” stream the pipeline rules are not running on the messages.

4

On the configurations page, what is your current “Message Processors Configuration”?

5

Here is the current list:

image
image.png766×202 17 KB

This is probably because the pipeline is processing before the message is moved to the index by the “Message Filter Chain”

6

Yep :slight_smile: Exactly right. Messages are routed to streams in the Message Filter Chain so the order won’t allow you to run the pipeline rules against messages in the stream. Moving the processing order will fix it but you need to take into account everything you are doing currently to determine what inadvertant impact it might have.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.