Delphix Products

Expand all | Collapse all

Ignore Comma in SM algorithm

Jump to Best Answer
  • 1.  Ignore Comma in SM algorithm

    Posted 05-13-2020 09:34:00 AM
    Hello All,
      I have  created a new SM algorithm for 30 char length . I want to ignore comma as special charterer , i have selected that check box . but still  it is not scrambling the  value which has comma  in the data. is i am missing anything ? please advise.

    Thanks In advance




    ------------------------------
    GEETHANJALI DEVI
    Business Objects Developer
    Genworth Financial
    ------------------------------


  • 2.  RE: Ignore Comma in SM algorithm

    Posted 05-13-2020 11:09:00 AM
    Hi, what version is your masking engine?

    ------------------------------
    John Leser
    Staff Engineer
    Delphix
    ------------------------------



  • 3.  RE: Ignore Comma in SM algorithm

    Posted 05-13-2020 11:56:00 AM
    Thanks for the response John, I am using 5.3.6.0 version of Delphix Masking.

    ------------------------------
    GEETHANJALI DEVI
    Business Objects Developer
    Genworth Financial
    ------------------------------



  • 4.  RE: Ignore Comma in SM algorithm

    Posted 05-13-2020 12:49:00 PM
    I've checked your case as best I can, and for me, adding ',' ignore character does what I expect:

    First I create SM with 30 alpha-numeric segments, then mask two values:

    123456789012345678901234567890 -> 377949142883334923098463559959
    12345,6789012345678901234,567890 ->12345,6789012345678901234,567890

    Then I added ',' to the list of ignore characters by checking the box in the UI saving the algorithm:

    123456789012345678901234567890 -> 377949142883334923098463559959
    12345,6789012345678901234,567890 -> 37794,9142883334923098463,559959

    The definition of my algorithm looks like this in the api-client:
    {
      "algorithmName": "ALNUM30",
      "algorithmType": "SEGMENT",
      "createdBy": "admin",
      "description": "",
      "algorithmExtension": {
        "ignoreCharacters": [
          44
        ],
        "segments": [
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          },
          {
            "length": 1,
            "minInt": 0,
            "maxInt": 0,
            "minRealInt": 0,
            "maxRealInt": 0
          }
        ],
        "segmentsGroupLength": [
          4,
          4,
          4,
          4,
          4,
          4,
          4,
          2
        ],
        "errorHandlingMode": "DEFAULT"
      }
    }
    
    The block with ignoreCharacters is what changed.

    If you're familiar with the api-client, you could post your algorithm definition, you might check that and see if it contains the similar values to what I have here.  You might also look in the job results to see what non-conforming data is still being reported, for clues about any data formats you may not have accounted for in your ignore characters.  A 30 segment algorithm will not mask a value with 31 characters (no counting ignore characters) or lead/trailing whitespace, which is a common source of problems.

    ------------------------------
    John Leser
    Staff Engineer
    Delphix
    ------------------------------



  • 5.  RE: Ignore Comma in SM algorithm

    Posted 05-13-2020 01:43:00 PM
    Thanks John, I have tried by  added ',' to the list of ignore characters by checking the box in the UI and saved the same but still not working. also  i have checked the size for the   column it is  15 char value which is  not getting scrambled. i am also trying to upload my snap shot it is getting failed to upload too. I don't have option to api-clinet to post the  algorithm definition.

    Thanks
    Geetha




    ------------------------------
    GEETHANJALI DEVI
    Business Objects Developer
    Genworth Financial
    ------------------------------



  • 6.  RE: Ignore Comma in SM algorithm

    Posted 05-13-2020 01:50:00 PM
    Hi Geetha,
    Are you using the image icon shown here 
    to insert your screen capture or the upload file button?

    Thanks,
    Michael

    ------------------------------
    Michael Torok
    Director of Knowledge and Community Management
    Delphix
    ------------------------------



  • 7.  RE: Ignore Comma in SM algorithm

    Posted 05-13-2020 02:07:00 PM
    yes i tried that option , as well as upload file option  in the bottom of the post. also tried with .png,.jepg  .

    Thanks

    ------------------------------
    GEETHANJALI DEVI
    Business Objects Developer
    Genworth Financial
    ------------------------------



  • 8.  RE: Ignore Comma in SM algorithm

    Posted 05-13-2020 02:21:00 PM
    Edited by Michael Torok 05-13-2020 02:21:18 PM
    Attaching the image for Geetha:


    ------------------------------
    Michael Torok
    Director of Knowledge and Community Management
    Delphix
    ------------------------------



  • 9.  RE: Ignore Comma in SM algorithm

    Posted 05-13-2020 03:34:00 PM
    Edited by John Leser 05-13-2020 03:36:58 PM
    Even attempting to recreate exactly the definition you have there, I can't seem to produce a case where the comma checkbox doesn't work as expected.

    Just to confirm, if you open the edit dialog later for this algorithm, the checkbox remains checked?

    Are you able to see non-conforming data reports in your job results?  Can you share the patterns you're seeing there for the values that don't mask?

    ------------------------------
    John Leser
    Staff Engineer
    Delphix
    ------------------------------



  • 10.  RE: Ignore Comma in SM algorithm

    Posted 05-13-2020 03:45:00 PM
    yes, i can see that check box  selected all the time.  here the  pattern  from non confirming data
    PLLLLLLSSLLPLLLLLSSP
    PLLLLLLSSLLPLLLLLSSLLLLP

     my data value  is
    sample,test
    Sample,Test
    Sample,
    sample,test

    Thanks
    Geetha


    ------------------------------
    GEETHANJALI DEVI
    Business Objects Developer
    Genworth Financial
    ------------------------------



  • 11.  RE: Ignore Comma in SM algorithm

    Posted 05-14-2020 01:55:00 AM
    Suggest you check that the BLANK is part of your ignore characters list :  ... , , ...
    SM alfanumeric supports only [0-9] [a-z] [A-Z] .. e.g. any other char that might exist will brake it.
    Also accented characters are not supported. Most often is forgotten to add the blank to ignore list.

    ------------------------------
    Tino Pironti
    Technical Services
    Delphix
    ------------------------------



  • 12.  RE: Ignore Comma in SM algorithm

    Posted 05-14-2020 08:45:00 AM
    Thanks for the update.what do you mean by blank char if that is space  i have added the space in the ignore list. here is the list  of my ignore chars ' ,-,_,;,/,\,?,\,!,|,.,*, ,~,`,' the first one is space. also i don't understand  "Also accented characters are not supported" can you please explain this bit more.

    Thanks
    Geetha


    ------------------------------
    GEETHANJALI DEVI
    Business Objects Developer
    Genworth Financial
    ------------------------------



  • 13.  RE: Ignore Comma in SM algorithm

    Posted 05-14-2020 08:56:00 AM
    The blank cannot be the first argument in ignore characters list. Please put it later .. , , ..
    Further if editing make surety blank is preserved (sometimes if you save > close > reopen the space is lost > so if you save again there is no space anymore).
    Accented characters:  e and ë
    > the e is supported
    > the ë is not supported.

    ------------------------------
    Tino Pironti
    Technical Services
    Delphix
    ------------------------------



  • 14.  RE: Ignore Comma in SM algorithm

    Posted 05-14-2020 10:18:00 AM
    Hello Tino,

      I have removed the  space from the  first argument . and i added in the middle,   "-,_,;,/,\,?,\,!,|,.,*, ,~,`,'" and also there is no accented characters.

    my data value is

    sample,test
    Sample,Test
    Sample,
    sample,test

    I am using the script to  extract the data from the Greenplum table  and then passing that  csv file into the Delphix to do the scrambling. and that  csv file is comma separated file.  is this is causing this ignore comma function not to work as expected?

    Thanks
    Geetha


    ------------------------------
    GEETHANJALI DEVI
    Business Objects Developer
    Genworth Financial
    ------------------------------



  • 15.  RE: Ignore Comma in SM algorithm

    Posted 05-14-2020 10:30:00 AM
    There is a checkbox >> Ignore comma(,)
    You need this checked as your text contains commas.


    ------------------------------
    Tino Pironti
    Technical Services
    Delphix
    ------------------------------



  • 16.  RE: Ignore Comma in SM algorithm

    Posted 05-14-2020 10:34:00 AM
    yes i did that , but not working.

    Thanks
    geetha

    ------------------------------
    GEETHANJALI DEVI
    Business Objects Developer
    Genworth Financial
    ------------------------------



  • 17.  RE: Ignore Comma in SM algorithm

    Posted 05-14-2020 10:42:00 AM
    Edited by Tino Pironti 05-14-2020 10:43:04 AM
    > this is a FILE
    - using FTP or SFTP connector ?
    - is it IN PLACE or ON THE FLY job type ?
    If this is ON THE FLY the file must exist in both SOURCE and TARGET location (can be empty file in target location, gets overwritten).
    - make sure the assignment of FILE-FORMAT to FILE its still configured !
    In monitor you get the LOG > that does that tell ?

    ------------------------------
    Tino Pironti
    Technical Services
    Delphix
    ------------------------------



  • 18.  RE: Ignore Comma in SM algorithm
    Best Answer

    Posted 05-14-2020 11:20:00 AM
    Hello All,
      Actually I have figured the the  problem,  the  issue is with the Delimiter  since we are  using the comma as delimiter the Ignore comma function is not working. i have changed the Delimiter to | and then this ignore comma is working as expected. Thank you all for the wonderful support.

    Thanks
    Geetha

    ------------------------------
    GEETHANJALI DEVI
    Business Objects Developer
    Genworth Financial
    ------------------------------