# Changing a field type from string to numeric and mid-setup and past string values **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** pipeline-rules, debuggingpl\ **Created:** [November 27, 2017, 6:11pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297 "2017-11-27T18:11:37Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![texas-bronius](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/texas-bronius/32/1051_2.png) [@texas-bronius](https://community.graylog.org/u/texas-bronius)\ **Post date:** [November 27, 2017, 6:11pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297/1 "2017-11-27T18:11:37Z")
A custom script produces a json message and is logged into my new Graylog instance, but its numeric fields are being interpreted as String and are therefore treated and stored as String. Consequently, I cannot Chart against these values. Ex. Message: ```auto {"level": "6", "details": {"memorypercent": "0.4954", "load_01": "0.47", "load_05": "0.42", "load_15": "0.35", "disksda3percent": "0.64", "disksdc1percent": "0.64"}} ``` I fixed this by changing the source message, removing the quotes surrounding the numeric values. Ex. Message: ```auto {"level": 6, "details": {"memorypercent": 0.4938, "load_01": 0.35, "load_05": 0.30, "load_15": 0.31, "disksda3percent": 0.64, "disksdc1percent": 0.64}} ``` Confirmed by reading a [suggestion to narrow search to a time window](https://community.graylog.org/t/forcing-numeric-fields/1005/4?u=texas-bronius) showing only the recently logged items, even with only the now-numeric fields showing, I still get the “Could not create field graph Field graphs are only available for numeric fields.” message. How can I get Graylog to recognize these fields as numeric? Thanks -Bronius ---
**Author:** ![sachin](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@sachin](https://community.graylog.org/u/sachin)\ **Post date:** [November 27, 2017, 7:05pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297/2 "2017-11-27T19:05:27Z")
Use **Pipeline** rules to convert them to to **numeric** using **to\_double** function. ---
**Author:** ![texas-bronius](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/texas-bronius/32/1051_2.png) [@texas-bronius](https://community.graylog.org/u/texas-bronius)\ **Post date:** [November 27, 2017, 10:20pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297/3 "2017-11-27T22:20:04Z")
Is this really necessary? If json presents the data as numeric, Graylog can’t just know it’s numeric? … and do I really have to do this for _all_ json extracted fields I wish to be treated as numeric? I did create a pipeline rule to convert using `to_double` as suggested, but it doesn’t seem to resolve my issue. Here’s the rule that I’ve applied to the All Messages stream against which I am querying to try to generate a chart (even though eventually I’d like it in the stream I’ve dedicated to this particular message type): ```auto rule "convert string float to numeric" when has_field("application_name") && $message.application_name == "sysmetrics" then // debug(concat("original: ", to_string($message.details_load_01))); set_field("details_load_01", to_double($message.details_load_01)); // debug(concat("converted: ", to_string(to_double($message.details_load_01)))); end ``` I have confirmed this pipeline rule is run by tail’ing /var/log/graylog-server/server.log where I see my debug messages. Here’s the order of processors: ```auto 1 GeoIP Resolver active 2 Message Filter Chain active 3 Pipeline Processor active ``` [edit] Note: Generate Chart does work on a field if I create a new field based on the old like `set_field("details_load_01_converted", to_double($message.details_load_01));` so it’s just not overwriting the original value. ☹ I wonder what else I could try? And I wonder why the json parsed numeric is not automatically interpreted as such… Maybe I can delete the index and start over? [/edit] ---
**Author:** ![texas-bronius](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/texas-bronius/32/1051_2.png) [@texas-bronius](https://community.graylog.org/u/texas-bronius)\ **Post date:** [November 27, 2017, 11:11pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297/4 "2017-11-27T23:11:57Z")
Ok pipeline field rewrite rule is not necessary in my case, because indeed Graylog/Elasticsearch does treat a numeric json value as numeric on store, provided the original value seen is numeric to begin with, and I can do numeric operations on them like Generate Chart. I’ll have to rewrite all my json to not quote numeric fields if I want to most efficiently take advantage of this. Following [this old thread](https://groups.google.com/forum/#!topic/graylog2/s-Nqb3EwRkw) it was suggested to “roll the index” or “cycle the deflector.” I disabled my pipeline rule, and, when I didn’t see these exact StarTrek-like options in 2.3, on a gander I selected Rotate Active Index from the index Maintenance menu, limited my search to a short window after, and I am able to Generate Chart from the now numeric values. Thanks! -Bronius ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [November 28, 2017, 3:35pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297/5 "2017-11-28T15:35:16Z")
just as addition, you can force field type using a template, a custom mapping [http://docs.graylog.org/en/2.3/pages/configuration/elasticsearch.html#custom-index-mappings](http://docs.graylog.org/en/2.3/pages/configuration/elasticsearch.html#custom-index-mappings) That way you do not count on the way elasticsearch recognize the values and if the first message by accident has a string character the field would be - for that index as a string and not a number saved. ---
**Author:** ![texas-bronius](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/texas-bronius/32/1051_2.png) [@texas-bronius](https://community.graylog.org/u/texas-bronius)\ **Post date:** [November 28, 2017, 4:01pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297/6 "2017-11-28T16:01:11Z")
Excellent tip - thanks @jan. I have at least one field coming from apache json that is a valid user id 90% of the time (Long) but sometimes “-” when not set by the application. Will Graylog crap out on “-” and not process the entire message, the field, or …? ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [November 28, 2017, 4:04pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297/7 "2017-11-28T16:04:56Z")
Hej @texas-bronius, it will - as currently already - if the message content did not fit into the field message type. But if you have such known fields, you might want to check those fields/messages in a pipeline and remove those fields not matching the configured/wanted field type. Just to have the messages save and clear. ---
**Author:** ![texas-bronius](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/texas-bronius/32/1051_2.png) [@texas-bronius](https://community.graylog.org/u/texas-bronius)\ **Post date:** [November 28, 2017, 4:11pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297/8 "2017-11-28T16:11:01Z")
Sorry to be dense here (and I am also still looking online for clarification): If a field is known to be, say, Long, and message processor encounters a String like “-”, will the field be set to some empty value, will the field be discarded, or will the whole message get discarded? And when this happens, will there be a new error logged each time in graylog-server/server.log? Thanks -Bronius [update] I think I get it: If a rule has a failing statement (like trying to set\_field(‘remoteIP’, to\_ip($message.remoteIP)) when `remoteIP` is a string and not an IP Address Object), then the whole message is skipped. So to answer my question above, if to\_long() were to fail, the whole message would be dropped. Except that with to\_long(), the default is a `0`. In my use case, since 0 is a valid user ID (“Anonymous user” in Drupal), I can’t have both “-” and “0” mean “Anonymous” but I can override the default for that call with, say, `to_long($message.uid, -1)` and I’m off to the races! [/update] ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [November 29, 2017, 12:36pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297/9 "2017-11-29T12:36:02Z")
I’m glad that you find the answer before I was able to jump in - and shared it! You earn a double star for that! thank you for the active work in the community! ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [November 29, 2017, 1:36pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297/10 "2017-11-29T13:36:49Z")
A post was split to a new topic: [Change elasticsearch field mappings cause error in query](https://community.graylog.org/t/change-elasticsearch-field-mappings-cause-error-in-query/3329) ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [December 13, 2017, 1:37pm UTC](https://community.graylog.org/t/changing-a-field-type-from-string-to-numeric-and-mid-setup-and-past-string-values/3297/11 "2017-12-13T13:37:22Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.