Delphix Products

Is anyone tried to create "on the fly" masking job by using dxmc. I am getting connector error, though it is mentioned in the command. For example:

 ./dxmc job add --jobname API-BC_TACCOUNT --envname AwsMask-Target --rulesetname API-BC_TACCOUNT --on_the_fly_source SOURCE-BC_TACCOUNT --max_memory 8192 --min_memory 4096 --num_streams 20 --on_the_fly Y

This OnTheFly MaskingJob needs a source File Connector specified.

Not sure what is missing here & if anyone is aware of doing right way.

Thanks,
Santosh
#Masking

I was looking at the log, seems "on_the_fly_masking_source" or "--on_the_fly_source" parameter is not read at all. I understand its a community driven, but how do get support for these kind of issues if any fix expected?
 

[2019-04-17 15:17:06,900:DEBUG:DxJob:86 - add()] create job input {'created_by': None,
 'created_time': None,
 'database_masking_options': {'batch_update': True,
                              'bulk_data': False,
                              'commit_size': None,
                              'disable_constraints': False,
                              'disable_triggers': False,
                              'drop_indexes': False,
                              'num_output_threads_per_stream': 1,
                              'postscript': None,
                              'prescript': None,
                              'truncate_tables': False},
 'email': None,
 'feedback_size': None,
 'job_description': None,
 'job_name': u'API-BC_TACCOUNT',
 'masking_job_id': None,
 'max_memory': 8192,
 'min_memory': 4096,
 'multi_tenant': False,
 'num_input_streams': 20,
 'on_the_fly_masking': True,
 'on_the_fly_masking_source': None,
 'ruleset_id': 41}
[2019-04-17 15:17:06,900:DEBUG:DxJob:88 - add()] API instance created
[2019-04-17 15:17:06,901:DEBUG:retry:210 - from_int()] Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=0, status=None)
[2019-04-17 15:17:06,935:DEBUG:connectionpool:393 - _make_request()] http://10.175.27.21:8282 "POST /masking/api/masking-jobs HTTP/1.1" 400 84
[2019-04-17 15:17:06,935:DEBUG:rest:201 - request()] response body: {"errorMessage":"This OnTheFly MaskingJob needs a source File Connector specified."}
[2019-04-17 15:17:06,936:ERROR:DxJob:99 - add()] (400)

Hi Santosh,
The community aspect of the project does make fixes more sporadic, but let me see what I can find out. 

Thanks,
Michael

Thanks for the info Mike. Please let me know if any fix available or some alternative ways to create the job, other than APIs.

Thanks,
Santosh 

Hi Santosh,
the concept of "On-the-Fly" requires a source and target data location defined. It looks you have defined the source but not the target. This requires the actual creation of both source and target connectors on the masking engine. Then you need to declare both for the OTF job, for the GUI and/or for the dxm-toolkit.

Yes, I also looked at the GUI for comparison before posting here. I thought "environment" +  "rulesetname" will take care of mapping it to target connection as they are associated to each other. By looking command usage, I don't see any option to define target connection

delphix@ip-10-175-18-90:/stagingData/Scripts/API-Automation-ToolKit$ ./dxmc job add --help

Usage: dxmc job add [OPTIONS]　

List of required parameters:　

jobname

envname

rulesetname

　

Return non zero code if there was problem with adding a new job

　Options:

--jobname TEXT Name of job to add [required]

--envname TEXT Name of environment where job will be added

[required]

--rulesetname TEXT Name of ruleset which will be used for masking

job [required]

--jobdesc TEXT Desciption of the job

--email TEXT e-mail address used for job notification

--on_the_fly_source TEXT connector name for on the fly job

--feedback_size INTEGER Feedback size of masking job

--max_memory INTEGER Maximum size of memory used by masking job

--min_memory INTEGER Minimum size of memory used by masking job

--num_streams INTEGER Number of concurrent objects (tables/files)

being masked by job

--commit_size INTEGER Number of rows after which commit will be

issued

--threads INTEGER Number of threads masking single object. DON'T

CHANGE UNLESS YOU KNOW WHAT YOU ARE DOING

--on_the_fly [Y|N] Define job as 'on the fly' masking

--multi_tenant [Y|N] Define job as multi-tenant

--batch_update [Y|N] Enable batch update for job.DON'T CHANGE UNLESS

YOU KNOW WHAT YOU ARE DOING

--disable_constraints [Y|N] Disable constraints for duration of masking job

--drop_indexes [Y|N] Drop indexes for duration of masking job

--disable_triggers [Y|N] Disable triggers for duration of masking job

--truncate_tables [Y|N] Truncate table before starting load. Use for on

the fly masking only

--bulk_data [Y|N] Use bulk update for jobs. It's use by default

--prescript FILENAME File name and path used as prescript

--postscript FILENAME File name and path used as postscript

--format [fixed|csv|json] Output format

--engine TEXT Engine name

--debug Enables debug mode.

--logfile TEXT Logfile path and name

--help Show this message and exit.