Delphix Products

Expand all | Collapse all

dxmc tool to create "on the fly" masking job

  • 1.  dxmc tool to create "on the fly" masking job

    Posted 04-16-2019 02:41:00 PM
    Is anyone tried to create "on the fly" masking job by using dxmc. I am getting connector error, though it is mentioned in the command. For example:

     ./dxmc job add --jobname API-BC_TACCOUNT --envname AwsMask-Target --rulesetname API-BC_TACCOUNT --on_the_fly_source SOURCE-BC_TACCOUNT --max_memory 8192 --min_memory 4096 --num_streams 20 --on_the_fly Y

    This OnTheFly MaskingJob needs a source File Connector specified.

    Not sure what is missing here & if anyone is aware of doing right way.

    Thanks,
    Santosh
    #Masking


  • 2.  RE: dxmc tool to create "on the fly" masking job

    Posted 04-17-2019 02:34:00 PM
    I was looking at the log, seems "on_the_fly_masking_source" or "--on_the_fly_source" parameter is not read at all. I understand its a community driven, but how do get support for these kind of issues if any fix expected?
     

    [2019-04-17 15:17:06,900:DEBUG:DxJob:86 - add()] create job input {'created_by': None,
     'created_time': None,
     'database_masking_options': {'batch_update': True,
                                  'bulk_data': False,
                                  'commit_size': None,
                                  'disable_constraints': False,
                                  'disable_triggers': False,
                                  'drop_indexes': False,
                                  'num_output_threads_per_stream': 1,
                                  'postscript': None,
                                  'prescript': None,
                                  'truncate_tables': False},
     'email': None,
     'feedback_size': None,
     'job_description': None,
     'job_name': u'API-BC_TACCOUNT',
     'masking_job_id': None,
     'max_memory': 8192,
     'min_memory': 4096,
     'multi_tenant': False,
     'num_input_streams': 20,
     'on_the_fly_masking': True,
     'on_the_fly_masking_source': None,
     'ruleset_id': 41}
    [2019-04-17 15:17:06,900:DEBUG:DxJob:88 - add()] API instance created
    [2019-04-17 15:17:06,901:DEBUG:retry:210 - from_int()] Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=0, status=None)
    [2019-04-17 15:17:06,935:DEBUG:connectionpool:393 - _make_request()] http://10.175.27.21:8282 "POST /masking/api/masking-jobs HTTP/1.1" 400 84
    [2019-04-17 15:17:06,935:DEBUG:rest:201 - request()] response body: {"errorMessage":"This OnTheFly MaskingJob needs a source File Connector specified."}
    [2019-04-17 15:17:06,936:ERROR:DxJob:99 - add()] (400)



  • 3.  RE: dxmc tool to create "on the fly" masking job

    Posted 04-17-2019 02:51:00 PM
    Hi Santosh,
    The community aspect of the project does make fixes more sporadic, but let me see what I can find out. 

    Thanks,
    Michael


  • 4.  RE: dxmc tool to create "on the fly" masking job

    Posted 04-23-2019 07:21:00 PM
    Thanks for the info Mike. Please let me know if any fix available or some alternative ways to create the job, other than APIs.

    Thanks,
    Santosh 


  • 5.  RE: dxmc tool to create "on the fly" masking job

    Posted 04-24-2019 05:08:00 PM
    Hi Santosh,
    the concept of "On-the-Fly" requires a source and target data location defined. It looks you have defined the source but not the target. This requires the actual creation of both source and target connectors on the masking engine. Then you need to declare both for the OTF job, for the GUI and/or for the dxm-toolkit.


  • 6.  RE: dxmc tool to create "on the fly" masking job

    Posted 04-24-2019 05:30:00 PM

    Yes, I also looked at the GUI for comparison before posting here. I thought "environment" +  "rulesetname" will take care of mapping it to target connection as they are associated to each other. By looking command usage, I don't see any option to define target connection

    delphix@ip-10-175-18-90:/stagingData/Scripts/API-Automation-ToolKit$ ./dxmc job add --help

    Usage: dxmc job add [OPTIONS] 

    List of required parameters: 

    jobname

    envname

    rulesetname

     

    Return non zero code if there was problem with adding a new job

     Options:

    --jobname TEXT Name of job to add [required]

    --envname TEXT Name of environment where job will be added

    [required]

    --rulesetname TEXT Name of ruleset which will be used for masking

    job [required]

    --jobdesc TEXT Desciption of the job

    --email TEXT e-mail address used for job notification

    --on_the_fly_source TEXT connector name for on the fly job

    --feedback_size INTEGER Feedback size of masking job

    --max_memory INTEGER Maximum size of memory used by masking job

    --min_memory INTEGER Minimum size of memory used by masking job

    --num_streams INTEGER Number of concurrent objects (tables/files)

    being masked by job

    --commit_size INTEGER Number of rows after which commit will be

    issued

    --threads INTEGER Number of threads masking single object. DON'T

    CHANGE UNLESS YOU KNOW WHAT YOU ARE DOING

    --on_the_fly [Y|N] Define job as 'on the fly' masking

    --multi_tenant [Y|N] Define job as multi-tenant

    --batch_update [Y|N] Enable batch update for job.DON'T CHANGE UNLESS

    YOU KNOW WHAT YOU ARE DOING

    --disable_constraints [Y|N] Disable constraints for duration of masking job

    --drop_indexes [Y|N] Drop indexes for duration of masking job

    --disable_triggers [Y|N] Disable triggers for duration of masking job

    --truncate_tables [Y|N] Truncate table before starting load. Use for on

    the fly masking only

    --bulk_data [Y|N] Use bulk update for jobs. It's use by default

    --prescript FILENAME File name and path used as prescript

    --postscript FILENAME File name and path used as postscript

    --format [fixed|csv|json] Output format

    --engine TEXT Engine name

    --debug Enables debug mode.

    --logfile TEXT Logfile path and name

    --help Show this message and exit.