I’m trying to parse json fields in pipeline rule. Im using Graylog 6.1.5.
I trying to use the following rule in a pipeline:
let sJson = to_string($message.fix_1);
let sJson = regex_replace(
pattern: "^\\[|\\]$",
value: sJson,
replacement: ""
);
let rsJson = flatten_json(to_string(sJson), "flatten");
set_fields(to_map(rsJson));
set_field("message", "parsed data");
Die Variable fix_1 enthält:
{"@timestamp":"2025-04-08T08:11:02.336569417Z","app_name":"osc-testdev","facility":"user","hostname":"s113sr197.swk.eu.vesta-group.com","kubernetes":{"annotations":{"k8s.ovn.org/pod-networks":"{\"default\":{\"ip_addresses\":[\"10.47.0.123/23\"],\"mac_address\":\"0a:58:0a:2f:00:7b\",\"gateway_ips\":[\"10.47.0.1\"],\"routes\":[{\"dest\":\"10.44.0.0/14\",\"nextHop\":\"10.47.0.1\"},{\"dest\":\"10.30.206.0/23\",\"nextHop\":\"10.47.0.1\"},{\"dest\":\"169.254.169.5/32\",\"nextHop\":\"10.47.0.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.47.0.1\"}],\"ip_address\":\"10.47.0.123/23\",\"gateway_ip\":\"10.47.0.1\"}}","k8s.v1.cni.cncf.io/network-status":"[{\n \"name\": \"ovn-kubernetes\",\n \"interface\": \"eth0\",\n \"ips\": [\n \"10.47.0.123\"\n ],\n \"mac\": \"0a:58:0a:2f:00:7b\",\n \"default\": true,\n \"dns\": {}\n}]","openshift.io/scc":"restricted-v2","openshift.openshift.io/restartedAt":"2025-04-08T07:50:53.568Z","sez.se.beta.kubernetes.io/pod":"runtime/default"},"container_id":"cri-o://8ce914ad68deb8a1565104a551df41034bc590a5d031b10f4d7efd","container_image":"link.internetseite.de:5080/osc/service/service:latest","container_image_id":"link.internetseite.de:5080/osc/service/service@sha256:cb2e7efe4eb64507515458648424c24d95408b6611f5d67","container_iostream":"stdout","container_name":"container","labels":{"app":"dac","pod-template-hash":"7f6cf7867"},"namespace_id":"0cfsdfsdf-dc7b-4a07-a78fe-cb1ea9e71511","namespace_labels":{"kubernetes_io_metadata_name":"service-test","pod-security_kubernetes_io_audit":"restricted","pod-security_kubernetes_io_audit-version":"v1.24","pod-security_kubernetes_io_warn":"restricted","pod-security_kubernetes_io_warn-version":"v1.24"},"namespace_name":"dac-service-test","pod_id":"d04752341f4-aefef-e3ede7567ac","pod_ip":"10.1.0.1","pod_name":"da-7f75886788c7-mqp9h","pod_owner":"ReplicaSet/da-7f6453f8c7"},"level":"info","log_source":"container","log_type":"application","message":"2025-04-08 10:11:02.336 INFO 1 --- [pool-5-thread-1] d.n.s.d.executor.MonitoringExecutor : ...finished monitoring process, took 262ms.","msg_id":"container","openshift":{"cluster_id":"ce07147272-da00-5543-acfc-2e959f7abeff","sequence":1744099862375954726},"prc_id":"d0ffef92-6d0d-41f4-aded-eefef75e2ef","severity":"debug"}
I’ve tried to use the the rule simulation feature.
In this case, graylog creates a field names kubernetes with all the data above in one field
"kubernetes":{"annotations":{"k8s.ovn.org/pod-networks":"{\"default\":{\"ip_addresses\":[\"10.47.0.123/23\"],\"mac_address\":\"0a:58:0a:2f:00:7b\",\"gateway_ips\":[\"10.47.0.1\"],\"routes\":[{\"dest\":\"10.44.0.0/14\",\"nextHop\":\"10.47.0.1\"},{\"dest\":\"10.30.206.0/23\",\"nextHop\":\"10.47.0.1\"},{\"dest\":\"169.254.169.5/32\",\"nextHop\":\"10.47.0.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.47.0.1\"}],\"ip_address\":\"10.47.0.123/23\",\"gateway_ip\":\"10.47.0.1\"}}","k8s.v1.cni.cncf.io/network-status":"[{\n \"name\": \"ovn-kubernetes\",\n \"interface\": \"eth0\",\n \"ips\": [\n \"10.47.0.123\"\n ],\n \"mac\": \"0a:58:0a:2f:00:7b\",\n \"default\": true,\n \"dns\": {}\n}]","openshift.io/scc":"restricted-v2","openshift.openshift.io/restartedAt":"2025-04-08T07:50:53.568Z","sez.se.beta.kubernetes.io/pod":"runtime/default"},"container_id":"cri-o://8ce914ad68deb8a1565104a551df41034bc590a5d031b10f4d7efd","container_image":"link.internetseite.de:5080/osc/service/service:latest","container_image_id":"link.internetseite.de:5080/osc/service/service@sha256:cb2e7efe4eb64507515458648424c24d95408b6611f5d67","container_iostream":"stdout","container_name":"container","labels":{"app":"dac","pod-template-hash":"7f6cf7867"},"namespace_id":"0cfsdfsdf-dc7b-4a07-a78fe-cb1ea9e71511","namespace_labels":{"kubernetes_io_metadata_name":"service-test","pod-security_kubernetes_io_audit":"restricted","pod-security_kubernetes_io_audit-version":"v1.24","pod-security_kubernetes_io_warn":"restricted","pod-security_kubernetes_io_warn-version":"v1.24"},"namespace_name":"dac-service-test","pod_id":"d04752341f4-aefef-e3ede7567ac","pod_ip":"10.1.0.1","pod_name":"da-7f75886788c7-mqp9h","pod_owner":"ReplicaSet/da-7f6453f8c7"}
AND additional the following fields:
But after set up the pipeline like tested, no parsing at all.
Is the json to complex for this rule o do i habe made any mistakes?
thank you Rene
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]