Delphix Products

 View Only

  • 1.  nonconforming data that was not masked by algorithm

    Posted 11-05-2019 01:12:00 PM
      |   view attached
    Hi,

    We see the following warnings during our masking process. Please assist.

    Column Telephone1 contained nonconforming data that was not masked by algorithm BH_Telephone_SM_Test

    The top nonconforming data samples were:
    PNNNPZNNNNNNN
    NNNZNNNNNNN
    SNNNNNNNNNNNN
    SNNNNNNNNNNN
    SNZPNNNPZNNNPNNNN
    SNZNNNPNNNPNNNN
    SNNNNNNNNNNNNN
    SNNNNNNNNN
    SNZNNNZNNNZNNNN
    NNNNNZNNNNNN

    Column Telephone2 contained nonconforming data that was not masked by algorithm BH_Telephone_SM_Test

    The top nonconforming data samples were:
    NNNZNNNNNNN
    NNNZNNNZNNNN
    NNNZNNNPNNNN
    PNNNPZNNNPNNNN
    LLLLLLZLLLLLLLLL
    NNNPZNNNPNNNN
    NNNZNNNZNNNNZ
    SNNZNNNNNNNNNN
    SNNNNNNNNNNN
    SNNNNNNNNNNNN

    Column MobilePhone contained nonconforming data that was not masked by algorithm BH_Telephone_SM_Test

    The top nonconforming data samples were:
    NNNZNNNZNNNN
    NNNZNNNNNNN
    NNNPNNNZNNNN
    PNNNPZNNNPNNNN
    NNNZNNNPNNNN
    NNNSNNNPNNNN
    NNNPNNNSNNNN
    NNNPZNNNPNNNN
    SNNZNNNNNNNNNN
    PNNNPNNNPNNNN




    ------------------------------
    Manoj Ravikumar
    IT Systems Engineer
    Bright Horizons Family Solutions
    ------------------------------

    Attachment(s)

    pdf
    masking_46_Tue_Nov_05_19_10_23_UTC_2019.pdf   146 KB 1 version


  • 2.  RE: nonconforming data that was not masked by algorithm

    Posted 11-05-2019 01:49:00 PM
    Hi Manoj,

    Thanks for reaching out. Hope all is well. 

    The non-conforming warnings are for values that are not masked due to the data is not matching the defined rules for the algorithm(s) used. In this case 'BH_Telephone_SM_test' algorithm, which is a Segment Mapping (SM). 

    The Segment Mapping algorithm is only taking A-Z and 0-9 (US Letters and Numbers) and the characters classification are assigned according to and in the following order: 
    1. Letters (L)
    2. Numbers (N)
    3. Marks (M)
    4. Separators (Z)
    5. Punctuation (P)
    6. Symbols (S)
    7. Other (O)
    Note: that there might be Numbers (N) and Letters (L) that are not non-conforming as well (for example extended unicode numbers and letters). 

    For details about the Unicode characters please look here: 
    https://www.compart.com/en/unicode/category/

    To find any non-conforming rows in the data I do recommend using the query below.

    Finding Spacial Characters
    For finding Special Characters, please change the details indicated:

    -- Replace [table]
    -- Replace [mask_col]
    -- Modify 1001 if more sample data is needed.

    SELECT [mask_col] FROM [table]
    WHERE ROWNUM < 1001
    AND regexp_like([mask_col],'[^a-zA-Z0-9]');


    When found, please add these special characters in the Ignore Characters in the SM algorithm.

    Kind regards,

    Anders



    ------------------------------
    Anders Karlsson
    Delphix Support
    Delphix
    ------------------------------

    Original Message


  • 3.  RE: nonconforming data that was not masked by algorithm

    Posted 11-05-2019 02:55:00 PM
    Thanks Anders.
    whats the AND regexp_like([mask_col],'[^a-zA-Z0-9]'); for SQL?

    ------------------------------
    Manoj Ravikumar
    IT Systems Engineer
    Bright Horizons Family Solutions
    ------------------------------

    Original Message


  • 4.  RE: nonconforming data that was not masked by algorithm

    Posted 11-05-2019 03:31:00 PM


    ------------------------------
    Manoj Ravikumar
    IT Systems Engineer
    Bright Horizons Family Solutions
    ------------------------------

    Original Message


  • 5.  RE: nonconforming data that was not masked by algorithm

    Posted 11-05-2019 03:32:00 PM
    We have seperated all the characters by a comma. We don't understand the UI error message.

    ------------------------------
    Manoj Ravikumar
    IT Systems Engineer
    Bright Horizons Family Solutions
    ------------------------------

    Original Message


  • 6.  RE: nonconforming data that was not masked by algorithm

    Posted 11-05-2019 04:43:00 PM
    Edited by System 11-13-2019 07:46:03 PM
    Hi Manoj,

    I can see that the last character in "Ignore Characters" is a comma. Please remove the last comma and try saving again. 

    Kind regards,

    Anders

    ------------------------------
    Anders Karlsson
    Delphix Support
    Delphix
    ------------------------------

    Original Message


  • 7.  RE: nonconforming data that was not masked by algorithm

    Posted 11-06-2019 02:00:00 AM
    Hi Manoj,

    1/ Correct the list of characters:
    a/ list need to have uniq values, you have same characters multiple times (dot, question mark)
    b/ you can't start with comma,
    b/ you can't finish with comma or space.

    2/ If structure of source data is inconsistent, you have 2 options:
    a/ use couple SM algo. for the same column, but prepared for different data structure,
    b/ use other algo like for example integrated telephone_nr masking algo, that will always mask last 5 numbers in the string, despite of a data structure. (that algo is not on the list of algos, you will se it when you select algo for column in inventory).

    Regards
    Karol Kobus

    ------------------------------
    Karol Kobus
    Consultant
    Spica Solutions
    ------------------------------

    Original Message


  • 8.  RE: nonconforming data that was not masked by algorithm

    Posted 11-06-2019 03:17:00 AM

    Hi Manoj,

    I recommande using custom sql at the ruleset level to normalize the phone column to only contain digits.

    This solution provide the advantage that you will not have to worry with the inputs inside the column as they always will be translated to digits.

    Using the ignore character in SM algorithm is somehow tricky, you will not have a 100% guarantee that the masking will always do it's job.

    Because if any column contains a new non ignored character it will fail, this will never happen if you normalize the column content using custom sql as you will always get digits.

    BR,

    Mouhssine 



    ------------------------------
    Mouhssine SAIDI
    Community Member
    Delphix Community Members
    ------------------------------

    Original Message


Related Content