I want to process into metricbeat data into graylog pipeline to build some rule .
For example i have to create new column name KeyID by concatinating few columns hostname.system.uptime then assign the uptime value.
hostname.system.uptime :1334567
Can you help me how to parse metribeat data and combine two fields . I have done for filebeat and some other json input . But metricbeat i cant parse.
Thanks,
Jay’
when you have done it already - what is your exact problem?
Hi Jan,
I have processed filebeat data . For metricbeat data i want to concatinate any two columns and assign the value.
EX:
in metricbeat i have host name and uptime details. I want to create one column hostname.uptime which contains uptime value.
smartdata.uptime =2000
below is my metricbeat input:
{“host”:{“name”:“smartdata”},“tags”:[“smartdata”,“beats_input_raw_event”],“@version”:“1”,“@timestamp”:“2019-06-26T18:31:21.396Z”,“metricset”:{“name”:“uptime”,“module”:“system”},“system”:{“uptime”:{“duration”:{“ms”:1654908871}}},“beat”:{“name”:“smartdata”,“hostname”:“smartdata”,“version”:“6.3.2”}}
From above input i have to get
smartdata.uptime =2000
ok - you do not know how to make use of the processing pipelines to work with the data right?
Because you have made the impression that you know that in the first postings, but now your latest let me think that you never done message normalization with the processing pipelines.
As what you request is very basic processing pipeline usage. So you have something to search in this community. In addition the Graylog Documentation. How to use the processing pipelines.
Jan,
I used pipeline for filebeat data to process different logs ,they are in plain text i used regex /grok to process into pipeline . But Now i want to process metricbeat data which i posted above. This is in json format thats why i want to know how to process in pipeline rule for my above requirement.
He Jay,
your skill and where you have trouble with wasn’t clear from your question.
The parse-json function is what you are searching:
http://docs.graylog.org/en/3.0/pages/pipelines/functions.html#parse-json
Hi Jan,
Please find the clear details …
Below is my input:
{“tags”:[“smartdata”,“metrics”,“beats_input_raw_event”],“event”:{“duration”:132692,“module”:“system”,“dataset”:“system.network”},“fields”:{“env”:“staging”},“metricset”:{“name”:“network”},“system":{“network”:{“in”:{“errors”:0,“bytes”:4888679808,“dropped”:0,“packets”:6557203},“out”:{“errors”:0,“bytes”:4888679808,“dropped”:0,“packets”:6557203},“name”:“lo”}},“service”:{“type”:“system”},”@version":“1”,“@timestamp”:“2019-06-28T03:54:12.179Z”,“ecs”:{“version”:“1.0.0”},“cloud”:{“instance”:{“name”:“graylog”,“id”:“8712523612341637729”},“machine”:{“type”:“n1-standard-1”},“project”:{“id”:“seventh-vigil-226207”},“availability_zone”:“asia-south1-c”,“provider”:“gcp”},“host”:{“architecture”:“x86_64”,“name”:“graylog”,“id”:“c1a04cd37d5cff9be8d129a366cb7fba”,“containerized”:false,“hostname”:“graylog”,“os”:{“name”:“Ubuntu”,“family”:“debian”,“version”:“16.04.5 LTS (Xenial Xerus)”,“kernel”:“4.15.0-1032-gcp”,“platform”:“ubuntu”,“codename”:“xenial”}},“agent":{“hostname”:"graylog”,“type”:“metricbeat”,“ephemeral_id”:“ea853894-c7c1-42b6-9a05-cfd245ad238d”,“id”:“6f264b40-0604-4564-8f80-199d3beab583”,“version”:“7.2.0”}}
Bold values i am going to fetch and assign to new name in my pipeline
below is my pipeline rule:
rule “metricdata”
when
has_field(“message”)
then
let json = parse_json(to_string($message.message));
let json_fields = select_jsonpath(json, {HOST: “$.agent_hostname”});
set_field(“HOST”, to_string(json_fields.HOST));
let json_fields = select_jsonpath(json, {NetworkData: “$.system_network_in_bytes”});
set_field(“NetworkData”, to_double(json_fields.NetworkData))
end
Expected result is
HOST=graylog
NetworkData = 4888679808
but i am getting no HOST ,in NetworkData value become 0
Can help me to resolve .why i cant get those values from input,where i am doing mistake.
Thanks,
Jay’
I think your syntax may be wrong.
This post may help you clean up your pipeline script or give you a better idea? (replace “class” in the link example with “HOST” - does that solve it for you?)
As for NetworkData I suspect it’s similar (you effectively convert a blank string “” to double = 0) - So again, try using the code in the linked post and see if that resolves your issue.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
