API/Python/Open-Source Solutions

Expand all | Collapse all

Best setting to mask giant tables?

  • 1.  Best setting to mask giant tables?

    Posted 07-05-2017 07:30:00 PM
    I am trying to mask tables of more than 1 million lines, I tried several configurations, but the masking is very slow, there are some recommended settings for streams, theads .. I tried to use LK but there were no gains.
    #Masking


  • 2.  RE: Best setting to mask giant tables?

    Posted 07-05-2017 07:42:00 PM
    Hi Rodrigo, Tables containing 1 or 2 milions of rows are not giant. We are masking oracle table with hundreds of milions of rows. The mean spead is rough 500k rows per minute. What is your speed, what is your database? Did you disable triggers, indexes? Regards. Gianpiero


  • 3.  RE: Best setting to mask giant tables?

    Posted 07-05-2017 07:49:00 PM
    Hi, Gianpiero
    This Bank I am masking does not contain constraints and also triggers.     Speed ​​is low 3840 rows per min...
    Btree indexes exist in the field I'm masking, I'm using a credit card algorithm from Delphix itself.


    Thanks





  • 4.  RE: Best setting to mask giant tables?

    Posted 07-05-2017 07:51:00 PM
    Try to drop index, then mask, then re-create it.


  • 5.  RE: Best setting to mask giant tables?

    Posted 07-05-2017 09:24:00 PM
    Hi, Is it a partitioned table ? How many streams / threads did you configure for the job and what memory configuration you set for the job ? Regards, Mouhssine


  • 6.  RE: Best setting to mask giant tables?

    Posted 06-16-2018 04:23:00 AM
    Hi,

    I am facing the same issue for partitioned table. though table size is 40 million. Its a partitioned table. and its extremely slow inserting data to destination.


  • 7.  RE: Best setting to mask giant tables?

    Posted 06-19-2018 11:32:00 AM

    Hi Anita,

    My preposition for optimizing masking for your partitioned tables, is to exploit the oracle partition’s key.

    The idea is creating one job per partition (using the key partition as subset condition at job creation) and run theme in parallel using script hooks or masking API if you are in 5.2.

    Hope its clear for you

    Regards,

    Mouhssine



  • 8.  RE: Best setting to mask giant tables?

    Posted 06-19-2018 12:10:00 PM
    Do not forget that the speed is dependent not only on the update time consuming, but also on fetch data from the DB. If you have very large tables in terms of number of columns, you have to keep in mind that data from all columns have to be downloaded from the DB server to the masking engine. In order to reduce this network throughput, you could use a custom SQL to fetch only columns that have to be masked + the logical key (or primary key if you didn't define a Logical Key).

    Regards.
    Gianpiero