Delphix Products

I have ID field with 9 digits and some records are '00000000' which I am filtering out in custom sql where clause
Segmented algorithm: has 2 segments masking right 6,7,8,9 digits in 2 segments with 2 digits each
First issue:
Inplace masking job for sql server, i have default values for all parameters and disabling constraints, indexes,and triggers..
1) when i run my inplace masking job, It is running fine until 9999 records and it is failing (I see duplicate key issue in log, but have to ignore that because it is not primary key column and it has 2 duplicates for some ID's and the masking should be consistent for those ID's. I have also tried different fields as logical keys) 
2) I have increased the update threads to 10, it is running till 99990 records and hanging there..
3) If I increase the commit size to more than the records (5450506), it is running till '2XXXXXX' which is 40% in 4-5 mints and performance falls down very badly, it is taking forever to process the remaining records.

Here, I am not able to understand the exact issue, on thing I have in my mind is Transactional log file size in SQL server DB, but mostly I can rule it out as I am able to run masking on other fields on the same table, no issues.
Only Segmented masking? i am not sure exactly what it is doing here..It should be simple straight forward masking the ID's

------------------------------
chandrapalreddy borra
Community Member
Cognizant Technology Solutions
------------------------------

Hi Chandrapalreddy,

It is difficult to diagnose this issue from your description simply because of the vast number of variables involved.  You seem to be suggesting 2 different issues:

- a duplicate entry issue involving a Segment Mapping algorithm
- what appears to be a performance issue related to the same algorithm/column pair

First, Segment Mapping can produce duplicates under certain circumstances.  I.e. if you limit the set of input/output values that can be produced, you can put the algorithm in a position where it masks some values and not others or where there are more input values than valid output values.

Second, while I have some thoughts on the performance, it is hard to understand if the previous issue is influencing this or not.

My suggestion for something this complicated is to open new cases with Delphix Support and provide the details of these issues.  Please upload a support logs bundle so that certain details (such as the system logs and configuration of the SM algorithm) can be extracted.  The specific details that will be most necessary for you to provide in the case are:

- description of what behavior is being seen, paying particular attention to those parts which are considered incorrect
- which job ID or job IDs are involved
- which tables/columns are exhibiting the issue

------------------------------
Tony Slack
Senior Member of Technical Staff
Delphix
------------------------------

Original Message:
Sent: 12-04-2020 02:56:07 PM
From: chandrapalreddy borra
Subject: Segmented masking inplace issue

I have ID field with 9 digits and some records are '00000000' which I am filtering out in custom sql where clause
Segmented algorithm: has 2 segments masking right 6,7,8,9 digits in 2 segments with 2 digits each
First issue:
Inplace masking job for sql server, i have default values for all parameters and disabling constraints, indexes,and triggers..
1) when i run my inplace masking job, It is running fine until 9999 records and it is failing (I see duplicate key issue in log, but have to ignore that because it is not primary key column and it has 2 duplicates for some ID's and the masking should be consistent for those ID's. I have also tried different fields as logical keys)
2) I have increased the update threads to 10, it is running till 99990 records and hanging there..
3) If I increase the commit size to more than the records (5450506), it is running till '2XXXXXX' which is 40% in 4-5 mints and performance falls down very badly, it is taking forever to process the remaining records.

Here, I am not able to understand the exact issue, on thing I have in my mind is Transactional log file size in SQL server DB, but mostly I can rule it out as I am able to run masking on other fields on the same table, no issues.
Only Segmented masking? i am not sure exactly what it is doing here..It should be simple straight forward masking the ID's

------------------------------
chandrapalreddy borra
Community Member
Cognizant Technology Solutions
------------------------------