We are running versions of graylog 3.1.1 & Elasticsearch 6.8.3 & mongo 3.6 in our environment. I have installed sidecar in client and configured metricbeat collector in System->sidecars and I started receiving metricbeat data from the client machine and from the data I am trying to find out list of process from the machine and routing it to different stream by using pipeline and below is the sample pipeline rules we have written,
rule "metricbeat_process" when to_string($message.metricset_name) == "process" then let msg = create_message("", ""); set_field("@timestamp", timestamp, "", "", msg); let id = concat(to_string($message.host_name), "::system::"); let id = concat(id, to_string($message.system_process_name)); let host_id = to_string($message.host_name); let type = "PROCESS"; set_field("id", id, "", "", msg); set_field("type", type, "", "", msg); set_field("host_id", host_id, "", "", msg); route_to_stream("Process_Stream,", "", msg); end
After implementing the above rule I am started receiving the data to Process_Stream with the corresponding fields. But I have a problem now…
Let say the metricbeat data is passed to this rule and generated the fields & routed to Process_stream and below is the data will look in stream
@timestamp: 2019-11-13T17:36:07.030Z id: graylog::system::explorer.exe type: "PROCESS" host_id: graylog timestamp: 2019-11-13 17:36:36.970 +00:00
and after sometime I have got the same id to the same stream but timestamp is different since I have specified in metricbeat configuration the interval period in which data will be pushed to graylog and below is the new message,
@timestamp: 2019-11-13T17:50:07.030Z id: graylog::system::explorer.exe type: "PROCESS" host_id: graylog timestamp: 2019-11-13 17:50:36.970 +00:00
I want to avoid the same “id” to be routed to the stream. Is there a way to drop the message before its goes to stream?
How can I check in the stream whether this “id” exists or not? If that id exists in stream drop the message or else sent to the stream.
Is there a way in graylog to handle this scenario?
Please let me know your thoughts and it would very helpful.