How do I make a number field from an extractor?

Graylog Central (peer support)
1

1. Describe your incident:

I have the following log message:

1ABC2D34e5f6gHI: to=<user@domain.tld>, relay=mail.domain.tld[1.2.3.4]:25, delay=18453, delays=18450/0.01/1.8/0.86, dsn=4.7.500, status=deferred (host mail.domain.tld[1.2.3.4] said: 451 4.7.500 Server busy. Please try again later from [5.6.7.8]. (S77719) [mail.domain.tld] (in reply to end of DATA command))

I created an extractor using a Grok Pattern, using https://github.com/whyscream/postfix-grok-patterns/blob/master/postfix.grok:

%{POSTFIX_QUEUEID:queue_id}: to=<%{EMAILADDRESS:rcpt}>, relay=%{POSTFIX_RELAY_INFO}, delay=%{NUMBER:total_delay;float}, delays=%{POSTFIX_DELAYS}, dsn=%{POSTFIX_STATUS_CODE_ENHANCED:dsn}

The field total_delay is being stored as a string, and I can not do functions like AVERAGE on it.

While retrieving data for this widget, the following error(s) occurred:

    Elasticsearch exception [type=illegal_argument_exception, reason=Field [total_delay] of type [keyword] is not supported for aggregation [avg]].

2. Describe your environment:

  • OS Information: Ubuntu 20.04

  • Package Version:4.2.9

  • Service logs, configurations, and environment variables: N/A

3. What steps have you already taken to try and solve the problem?

Re-read instructions on Extractors multiple times. Attempted to delete and re-create extractor.

4. How can the community help?

I would like to know what I am doing incorrectly. How can I turn these extracted fields into numbers?

2

Hello,

Not sure what’s going on.
First, the link you posted.

image
image1156×901 178 KB

Second [total_delay] field can be corrected by setting the index template to a integer or what ever is needed.

3

Hi Folks,

That´s the Link he Posted for the Grok Pattern he is using: postfix-grok-patterns/postfix.grok at master · whyscream/postfix-grok-patterns · GitHub

You have to convert the Field that has been stored as a String to number, for this you could create a Pipeline Rule.

rule "convert string to numeric: total_delay"
when
    has_field("total_delay")
then
    let var_total_delay = replace( to_string($message.total_delay), "$1");
    set_field("total_delay", to_long(var_total_delay));
end
4

Important to note that Elasticsearch sets the field type on index creation so if you already have a field in your current index called total_delay as a keyword, @Aksel’s rule won’t affect the type until you rotate the index. (System/Inidicies → edit the index, click on Maintenance button and rotate index).

I wrote up a note on historical correction of index types here - it was a while back though with Elastic 6.x so if you have 7.10, there may be some differences.

If you were to use the total_delay field to create a NEW field like this, the new field would start as a long in the current index.

set_field("total_mail_delay", to_long(total_delay));

2 Likes
5

sure but it will affect the new inputs.

1 Like
6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.