Question from a partner: Limitations on new masking algorithms

  • 0
  • 1
  • Question
  • Updated 5 months ago
  • Answered
Can we create customized lookup lists for secure lookup, and are there limits on the size of the lookup list?
Photo of Tim Gorman

Tim Gorman, Field Services

  • 2,794 Points 2k badge 2x thumb

Posted 5 months ago

  • 0
  • 1
Photo of Tim Gorman

Tim Gorman, Field Services

  • 2,794 Points 2k badge 2x thumb
The documented steps to do this are in the online and downloadable Delphix documentation for the masking engine.

To find this documentation, go to docs.delphix.com, then click on Masking, then click on either Delphix Masking Documentation in HTML or in downloadable PDF format.

In either display format, open the Delphix Masking Admin Guide.  Within that document, find the section on Managing Algorithm Settings, and then find the section about Adding New Masking Engine Algorithms.  This will include instructions on how to create a new custom algorithm, and how to upload your own customized "lookup file" for that algorithm.

I don't know if there are any limits on the size(i.e. KB, MB, GB, etc) or any limits on the number of entries (i.e. hundreds of lines, thousands of lines, millions of lines, etc) of a custom "lookup file".  Hopefully someone else in the community can share their knowledge on limits?

Please let me know if this helps?

Thanks!

-Tim
Photo of Gary Hallam

Gary Hallam, Official Rep

  • 1,702 Points 1k badge 2x thumb
Hi Tim,
I'd suggest less than a milllion entries for a Secure Lookup.  I've seen as much as 5 million, but this challenges the amount of memory needed and also means that the masking job will take an age to begin masking.  SL is one of the quickest algorithms though.
I notice that there's a couple of members of the community starting to post their lookup tables in github: https://github.com/smouhssine/DELPHIX_MASKING_ALGOS.
Regards,
Gary
Photo of Mouhssine SAIDI

Mouhssine SAIDI

  • 4,682 Points 4k badge 2x thumb
Hi all,

You have just to take care that you will be reading all your lookup file entities into masking memory at algorithm call.

Set adequately the job(s) memory in accordance to that.

Regards,

Mouhssine