Delphix Products

Expand all | Collapse all

Mapping algorithm values

  • 1.  Mapping algorithm values

    Posted 03-18-2019 05:33:00 PM
    Hi there, we have multiple tables with a total of 40K-ish unique values and I have setup a mapping algorithm with 100K unique values but it still fails due to not enough mapping values issue. 
    #Masking


  • 2.  RE: Mapping algorithm values

    Posted 03-18-2019 08:08:00 PM
    How many unique values are there in all the tables that are using this mapping algorithm?  You'll need to make sure you have enough values to cover them all.  


  • 3.  RE: Mapping algorithm values

    Posted 03-19-2019 08:28:00 AM
    Hi there,
    There are about 38K in total across all tables.
    I have 100K unique values in the lookup file.
    Regards


  • 4.  RE: Mapping algorithm values

    Posted 03-19-2019 08:36:00 AM
    Try recreating the algorithm.  I had a similar issue on 5.2.5 and recreating the algorithm fixed it.  I didn't follow it up with support so I never found root cause but it's worth a try.


  • 5.  RE: Mapping algorithm values

    Posted 03-19-2019 08:40:00 AM
    Thanks, I will try that as well, meanwhile I added 1 million values, which are unnecessary for the current set.


  • 6.  RE: Mapping algorithm values

    Posted 03-19-2019 08:46:00 AM

    Hi,

    If you are using lookup file this means that you are not using segmented mapping algorithm but secure lookup one instead.

    Notice that it's more accurate to use segmented mapping for primary / foreign keys columns it will guarantee you the uniqueness value generation.

    Can you show an example of the values you have in your column that we can help.

    Regards,

    Mouhssine



  • 7.  RE: Mapping algorithm values

    Posted 03-19-2019 08:52:00 AM
    Hi Mouhssine,

    I am using mapping algorithm, not segmented mapping or secure lookup.

    regards


  • 8.  RE: Mapping algorithm values

    Posted 03-19-2019 09:00:00 AM
    The downside to that is the startup time of the job will be longer.  Ideally you always want to have a similar number of values in the lookup file to the number of unique rows to be masked.

    https://thedatalobby.kuzodata.com/maximum-performance-data-masking/">https://thedatalobby.kuzodata.com/maximum-performance-data-masking/">https://thedatalobby.kuzodata.com/maximum-performance-data-masking/

    However, it's probably worth testing it to see whether it fixes your issue and how it performs.


  • 9.  RE: Mapping algorithm values

    Posted 03-19-2019 09:09:00 AM
    Yes, thanks, I had this in mind, but it was initially a test: I understand the performance compromise. look forward to get back to a new algorithm with less values. Moreover, it is not suggested to have so many lookup values for unique mapping, should use segmented mapping instead.
    This cannot be the algorithm going forward, just wondering why such an issue came up with the current ample of values. 
    The only thing to add is that, I have multiple copies of the same DB using this algorithm when masked.
    Regards 


  • 10.  RE: Mapping algorithm values

    Posted 03-19-2019 09:52:00 AM

    HI,

    Sorry for the misunderstanding.

    So the rule should be number of unique values in the mapping file have to be equal or superior to the column once to guarantee that uniqueness values.


    Can you confirm pleas if this is the case with you mapping file.

    Regards,

    Mouhssine



  • 11.  RE: Mapping algorithm values

    Posted 03-19-2019 09:54:00 AM
    Thanks, That is true Mouhssine, as I have mentioned before, there are 100K unique values in the lookup file to 38K unique in the DB.


  • 12.  RE: Mapping algorithm values

    Posted 03-19-2019 10:21:00 AM

    Hi,

    Just to complete the answer.

    I'm agree with Math's remarks the making job have to load on memory the 100K entries in mapping file before proceeding with masking and this spins up Performance / Maintenance questions.

    You will have to update the mapping file whenever it's needed to guaranty it have equal or more values than what you have in your columns.

    I still think that the easiest and smart way is to use segmented mapping algorithm, is there any particular reason for not using it in your case

    Regards,

    Mouhssine  




  • 13.  RE: Mapping algorithm values

    Posted 03-19-2019 10:29:00 AM
    Hi Mouhssine,

    The lookup file provides realistic values, while the segmented will be haphazard, random ones which is needed in our case.

    The issue is not which Algorithm being used, it's simply why is it asking for more values while it has more than enough, more than double unique values available.

    I am versed with other provisions and performance issues that exist.

    Regards.


  • 14.  RE: Mapping algorithm values

    Posted 03-19-2019 10:34:00 AM

    Hi,

    I've got the point, could you give it try with SL algorithm you will have to generate a file with >= to the number of entries in the column.

    May be the hick is with how the algorithm is working, and the idea is try with a simple replacement with guarantee of preserving the realistic value meaning.

    Regards,

    Mouhssine



  • 15.  RE: Mapping algorithm values

    Posted 03-19-2019 10:40:00 AM
    Hi,

    Have you defined any ignore character at the algorithm level, may be this influences the results.

    https://support.delphix.com/Unpublished_Articles/KBA1328_Mapping_Algorithm_(MA)_Technical_Overview

    Regards,

    Mouhssine 


  • 16.  RE: Mapping algorithm values

    Posted 03-19-2019 10:43:00 AM
    Thanks Mouhssine,
    There are no ignored chars. Mapping Algorithm ticks all the boxes in our case.
    It's just this anomaly regarding not enough lookup values while there are plenty, we are facing.
    Regards



  • 17.  RE: Mapping algorithm values

    Posted 03-19-2019 01:42:00 PM
    Is it possible that new values are being masked in the columns from one execution to the next?  Once a value from the MA lookup file is assigned to a specific value it is no longer available for new values that could exist on next execution.