Key_value not detecting tab in pipeline rule

Graylog Central (peer support)
1

I have a message from kubernetes that follows a pattern as below. The \t in the text below is an tab character.

docker:{“container_id”=>“bfd45592a39af0ac9f53861ca4d2003844eab2379750fff0c6e928801884c63a”}\tkubernetes:{“container_name”=>“XXX-XXXX-service”, “namespace_name”=>“XXX-dev”, “pod_name”=>“XXX-audittrail-service-84dd5c74bc-tvzh7”, “container_image”=>“artifactory-docker.YYY.com:443/docker-local/XXX-audittrail-service:integration”, “container_image_id”=>“artifactory-docker.YYY.com:443/docker-local/XXX-audittrail-service@sha256:7b6dcff6f655074bac0110ba97f9965339f52044c399e26ae5025e30f44801f9”}\level:unknown\thostname:ch-d-ops-wrk02.YYY.com

A key_value usage like below fails to detect the tab character and hence doesn’t work in extracting fields.

set_fields(
    fields:
        key_value(
            value: to_string($message.message),
            delimiters: "\t",
           kv_delimiters:":",
            ignore_empty_values: true)
);

When I use a split function such as this, I do see the list is correctly built.
let tokens = split("\t", to_string($message.message));

Could you please suggest

Thank You in advance

Could some one shed some light.

2

HI @gslulu
I’ve tried your example message and pipeline rule and everything worked correctly.

I think, that your problem is not with KV pipeline function, but field level. It contains string unknown, but graylog by default uses numeric value in this field, so it’s collide with it. So either rename field in pipeline rule from level to something different or use own index for kubernetes.

rule "kv kuberneter"
when
  has_field("message")
then
  set_fields(
    fields:
        key_value(
            value: to_string($message.message),
            delimiters: "\t",
            kv_delimiters:":",
            ignore_empty_values: true)
  );
  rename_field("level", "level_k11s");
end
1 Like
3

You hit the bull’s eye…Awesome.
I’m going with just changing the field name.
But, one question… how would I know which fields definition is my index using? I only see a configuration for ‘Field type refresh interval’.

Thank You!

4

Elasticsearch uses dynamic mapping to guest right field data type on field creation. If you want to create own mapping, check this:

https://docs.graylog.org/en/4.1/pages/configuration/elasticsearch.html#custom-index-mappings

5

@shoothub
Thank you so much for that reference.

Since I was not able to remove a key from the Map and rename_field was not working to eliminate the level:unknown entry from the map, I ended up doing something like below:

let fullMessage = replace(to_string($message.message), "level:unknown", "openshift_level:unknown");
set_fields(key_value(
            value: to_string(fullMessage),
            delimiters: "\t",
            kv_delimiters:":",
            ignore_empty_values: true));
6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.