Detecting Duplicate Log Entries for Sidecar Optimization

[sziegle (Ziegle)]

Hello everyone,

I’m currently configuring Graylog, and I need your assistance with creating a CSV cache table. I have selected all log files under /var/log/ in the sidecars, as I consider all logs relevant for my analysis. Unfortunately, this is leading to a significant amount of traffic, and I want to filter out duplicate log entries without losing important information.

I have already attempted to create a lookup table to identify duplicate entries, but I am encountering some challenges:

1. Creating the CSV File: I have created the CSV file with the required column headers, but I am unsure which keys and values to use for the lookup table. What columns would be most useful in identifying duplicate log entries?
2. CSV Encoding: I have noticed that my CSV file is in us-ascii format. I plan to convert it to UTF-8 to meet Graylog’s requirements. Are there best practices for doing this?
3. Cache Configuration: I have configured a Node-local, in-memory cache, but I’m uncertain how to effectively link the lookup table with the cache. Which adapters would be best suited for this purpose?

I would greatly appreciate any help or pointers that can assist me in optimizing my configuration.

[Joel_Duffield]

I don’t think a CSV lookup table is really going to help with finding duplicates in most cases. What kind of duplicates are you seeing, duplication in the same log file, between log files etc. Why are there duplicates at all is the first step.

[sziegle (Ziegle)]

“Graylog monitors many servers in my case, and since I can’t do without any log files, the entire directory /var/log* is used. This results in a very high data volume. To minimize traffic, I considered eliminating all duplicate log entries. However, there are no specific entries that I could filter.”