search for sensitive data on plain txt files

  • 0
  • 1
  • Problem
  • Updated 2 years ago
  • Solved
  • (Edited)


Can Delphix profile unstructured .txt files?

There is documentation about profiling structured files.

In addition there is documentation about virtualize unstructured files.

Maybe does it has to do with profiling unstructured files?


Photo of luigidep


  • 622 Points 500 badge 2x thumb

Posted 2 years ago

  • 0
  • 1
Photo of Hims

Hims, Employee

  • 2,592 Points 2k badge 2x thumb
Official Response
Hi Luigi,
Unstructured files are hard to process, however we can redact them using some creativity.
Approach is below:
1. We will treat the file as delimited, we have to find a character which does not exist in file to be used as field delimiter.
2. This will ensure every row is treated as a whole single field.
3. We will use Multi PHI checkbox on field to keep finding sensitive fields ( the default algorithm to be applied is in a property file), this will find out all patterns existing in the file.
4. We apply the "Free Text Redaction" Algorithm with values and patterns to be redacted to the one field.
5. Execute masking job.

User will have to do a few iterations to get this accurate though.