GROK Number not being formatted as float/integer

Graylog Central (peer support)
1

Graylog 4.0.16+79eb84c Debian 11.0.14 on Linux 5.10.0-13-amd64

I am attempting to create graphs for various NGINX response times, but despite my attempts at the extractors, the graph keeps saying:
Search type returned error: Expected numeric type on field [NGINXRequestTime], but got [keyword].

I have tried using a Copy input extractor and convert to numeric option, but the graph still complained with the same error.

I have also tried defining the field as NUMBER and BASE10NUM with float after the field name, and it also does not work.

Here is the Grok pattern I am using:
%{NUMBER:NGINXRequestTime:float} (%{NUMBER:NGINXUpstreamConnectTime:float}|-) (%{NUMBER:NGINXUpstreamHeaderTime:float}|-) (%{NUMBER:NGINXUpstreamResponseTime:float}|-)

I have tried changing “:float” to “;float” and it also does not work.

The field itself is correctly populating. The whole decimal number is present in all the correct fields, but its just not numeric for some reason.

I have also ran into the same issue trying to format the packet length number field in iptables logs, using the same format above (int instead of float), and graphs also fail saying the field is not numeric.

What am I doing wrong here?

2

This will work, but the index the data is going to still has it marked as a keyword. If you rotate the index, the first message coming in where you have set to numeric will set the field type. Previous indexes will stay as keyword though. There is a way to reset the old data as numeric that I wrote up a while back in the forum… it’s a bit involved but not too difficult. You can search for it or I can find it if you can’t. :slight_smile:

3

Would it be advisable to use a copy input and set the destination field as the same original field (i.e. overwrite the field?).
Also, does the Grok “;float” option not work at all? Or are you saying once the index is rotated, that will start to work?

4

On the creation of an index Elasticsearch defaults to creating things as keyword unless specifically set otherwise, you can’t change it, the only way is to rotate to a new index and when that field is written specifically as (something other than a keyword) Elasticsearch will change set to that new type. you can use the same field just note that any calculations you have in dashboards and whatnot will fail if you look back in time to previous indexes where that data is fixed as a keyword. (fixable as notes with that other post I put out there.

You could just have it write numeric to a new field name and THAT will start as storing numeric and off you go (with no history until the change)

5

So does this: %{NUMBER:NGINXRequestTime:float} not work? If not, what is the “:float” supposed to do?

I appreciate your help, I am just trying to avoid creating new extractors if I can do it all in one extractor.

6

I am pretty sure that would work fine - I haven’t set type in GROK for Graylog but it makes sense to me that ElasticSearch would pick it up as a float when you rotate the index. One thing to note, I have generally used long instead of float and I know Graylog is particular about field types… it doesn’t use all the ones that Elasticsearch makes available. don’t forget to rotate the index to make the change. If you want to get into changing the historical data as well, this is the post I put up on how to do that:

7

Thanks so much. The rotated index is starting to work. I don’t really need historical data aggregated, but its still good info.

1 Like
8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.