Need help for Plugin Development : Logs Similarity

Development
1

Hello,
I’m trying to develop a graylog plugin that searchs for similarity in logs.
I’m really struggling at understanding how implementing such plugin …

The global idea would be :

  • looking at each incoming log try to find out part of the log that are ‘variables’ (such as date, pid of process, ip address …) and other parts that are the same in each occurrence of the log in order to stores counters of similar logs (without the need of writing rules for each of them) .

For example :

'2018-05-04 16:12:26.256 38894 INFO nova.compute.resource_tracker [req-f83de8f6-b87a-4968-b5fd-e74d3d233457 - - - - -] Final resource view: name=compute1 phys_ram=420000MB used_ram=100000MB phys_disk=1000GB used_disk=10GB total_vcpus=22 used_vcpus=10 pci_stats=[]'

Should ideally be incrementing a counter for pattern like following :

%{DATESTAMP}% %{PID}% INFO nova.compute.resource_tracker [ UID - - - - - ] Final resource view: name=XXX phys_ram=%{NUMBER}%MB used_ram=%{NUMBER}%MB phys_disk=%{NUMBER}%GB used_disk=%{NUMBER}%GB total_vcpus=%{NUMBER}% used_vcpus=%{NUMBER}% pci_stats=’

I was first thinking about implementing a MessageProcessor Module …

But i don’t know how can i store theses kind of pattern counters ?

Any help/suggestion/remarks would be greatly appreciated :grinning:

Thanks,

Efflam

2

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.