Adjusting data in raw message

Graylog Central (peer support)
1

I have a scenario where we have logs (web browsing) containing PII data, and a requirement to forward these logs onto a 3rd party who should not see these fields in their original format.

Currently when logs are processed we hash the PII data using a pipeline rule and store in an additional Graylog field. What I would like to do is replace the original data in a copy of the raw message with the hashed value, so that I can use a stream\Syslog output to forward the modified message to the 3rd party that has the field value “anonymised”, while storing locally the original message and both the true and hashed values as fields.

Is this possible

2

you are able to work that out with processing pipelines.

but the messages my be duplicated and you need to take care of that with different index sets.

3

Jan,

Thanks - what is the mechanism by which we can strip the unwanted field in the message that is forwarded?

I started the process of using pipelines (and to your point on replica messages, was using a index that purged daily for a temporal copy of the edited message) but wasn’t sure how to edit the raw message, not the indexed fields - if that makes sense?

Thanks

4

you would need to “re-model” the message that is saved in Elasticsearch. With Pipelines every available field can be rewritten/adjusted.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.