Delphix Products

 View Only

Migrating Delphix CLI Toolkits: dxtoolkit to dct-toolkit

By Nicholas Mathison posted 07-19-2024 01:56:20 PM

  

Migrating Delphix CLI Toolkits: dxtoolkit to dct-toolkit 

 

Overview 

The Delphix Self-Service (Jet Stream) end-of-support date was announced for September 2025. This date has encouraged many Delphix customers to transition to Data Control Tower (DCT)And while the UI is an important part of the experience, the dxtoolkit, an open-source command line interface (CLI), must be considered and migrated as well. Fortunately, DCT has introduced its own supported CLI, called dct-toolkit In this blog post, we’ll cover the differences between dxtoolkit and dct-toolkit, the common dxtoolkit commands and commands that must be migrated, and various dct-toolkit examples. 


 

What are the differences between the toolkits? 

Before we jump into how you migrate between the dxtoolkit and dct-toolkit, it is important to understand the key differences between the two approaches. In general, the differences largely exist at the architectural product level which then manifest within the CLIs. Therefore, understanding the differences at a higher level will guide your technical implementation. 

No Template or Container Model 

DCT has removed the concept of the template and container model. Therefore, any actions are performed directly against the VDB or dSources - not the container.  

For VDBs that require coordination between one another, such as Oracle E-Business Suite (EBS), we offer the VDB Groups object. Like a traditional Delphix Self-Service container, VDBs within a VDB Group are acted on together, such as refreshing and bookmarking. VDB Groups are not limited by the Delphix Self-Service container model. 

Lastly, the dct-toolkit has different VDB and VDB Group command sets. Therefore, different commands are used based on the type of the object. 

The following table shows a few examples of common template/container dxtoolkit commands and their dct-toolkit equivalents. 

Use Case 

dxtoolkit Command Example 

dct-toolkit Command Equivalents 

Refresh VDB to latest dSource snapshot 

dx_refresh_db 

refresh_vdb_by_snapshot 

Refresh Single VDB container to latest dSource snapshot 

dx_ctl_js_container refresh (Single VDB) 

(See above) 

Restore multi-VDB container to chosen bookmark 

dxt_ctl_js_container restore –bookmark <NAME> (Multi VDB) 

refresh_vdb_group 

Manage bookmarks for a container 

dx_get_js_bookmarks 

dx_ctl_js_bookmarks 

get_bookmarks_by_vdb 
create_bookmark 

Create a container 

dx_ctl_js_container create 

N/A - Identical to creating a VDB or VDB Group. 

Create a template 

dx_ctl_js_template 

N/A - Identical to creating a dSource, VDB, or VDB Group. 

Grant user access 

dx_ctl_js_container addowner 

Tag management or add_objects_to_access_group_scope 

  

 

Refresh Time Operation Consolidation 

In DCT, all refresh, restore, rollback, and rewind operations have been consolidated into the singular “refresh” operation. Now, the operation is simplified as refreshing from what: a snapshot, timestamp, or bookmark. This flow follows the UI wizards. 

The following table shows a few examples of common refresh and restore dxtoolkit commands and their dct-toolkit equivalents. 

dxtoolkit Command Examples 

dct-toolkit Command Equivalents 

dx_refresh_db 

refresh_vdb_by_snapshot 

dx_ctl_js_container refresh (Single VDB) 

See above 

dx_ctl_js_container restore (Single VDB) 

refresh_vdb_by_snapshot <snapshot_id> 

refresh_vdb_by_bookmark <bookmark_id> 

refresh_vdb_by_timestamp <timestamp> 

dxt_ctl_js_container refresh (Multi VDBs) 

Coming Q3 2024! 

dxt_ctl_js_container restore (Multi VDBs) 

refresh_vdb_group <bookmark_id> 

Snapshot command coming Q3 2024! 

 

Single Authorization & Access Control 

In dxtoolkit, the user was forced to manage the connection and authorization to each engine. In addition, there were multiple calls to establish an authorized connection. This has been drastically simplified in the dct-toolkit. 

DCT is a centralized gateway to all Continuous Data and Compliance engines. Therefore, if a connection is authorized to DCT, then all granted engine actions are also permitted. No multi-credential management is required. And thanks to the DCT RESTful API, the authentication handshake has been greatly simplified and embedded within every command. 

 

dxtoolkit Authentication 

dct-toolkit Authentication 

https://github.com/delphix/dxtoolkit/wiki#configuration  
(Repeat N times for each engine) 

 

In addition to the management of multiple connections, the engines were limited in their access control model. They had a limited number of immutable roles that often missed the nuances of customer organizations. In DCT, we’ve built a robust Attribute-based Access Control mechanism that allows you to build your roles.  

The dct-toolkit takes advantage of these permissions. Therefore, the account’s provided API Key will follow the same permission model defined manually within DCT.  

Please read the Data Control Tower (DCT) - Access Control Implementation blog for further explanation and examples. 

 

Delphix Support 

One of the most common requests we heard was the need for Delphix to support a command line option. Thus, we have taken the extra step to release the dct-toolkit as a fully support offering. 

Unlike the dxtoolkit, you can file a ticket on the Delphix Support Portal for any dct-toolkit issues, and we’ll respond shortly!  

 

There are a few other minor differences between DCT and Delphix Continuous Data Engines, but they are outside the scope of the Delphix Self Service CLI migration. If you would like to learn more, please see the full set of differences in our documentation.  

 

When do I have to migrate off the dxtoolkit? 

If your dxtoolkit script depends on one of the following commands, you will need to migrate off the dxtoolkit by the September 2025 release of Continuous Data. 

  • dx_get_js_bookmarks - List Self Service bookmarks. 
  • dx_ctl_js_bookmarks - Add or remove Self Service bookmarks. 

  • dx_get_js_snapshots - List snapshot for a Self Service container. 

  • dx_get_js_containers - List Self Service containers. 

  • dx_ctl_js_container - Create, remove, refresh, or restore Self Service containers.

  • dx_ctl_js_branch - Create, delete, or activate a Self Service container branch.

  • dx_get_js_branches - List available branches for a container.

  • dx_get_js_datasources - List available Self-Service data sources.

  • dx_ctl_js_template - Create or delete a Self-Service template.

  • dx_get_js_templates - List available Self-Service templates. 

 

If a script references any of the above dxtoolkit commands, we recommend the entire script is re-written using the dct-toolkit. However, you could also take the opportunity to migrate to DCT, CD, or CC's APIs or a DCT integration, such as Terraform, ServiceNow, or Jenkins.  

These commands will continue to work until you upgrade your Continuous Data engine to the September 2025 release. If required, you can optionally leave an older version of Continuous Data running to solve any legacy scenarios. 

There are some edge cases where other commands in the dxtoolkit can be used to avoid a complete migration.  

 

Use Case 

Requires dct-toolkit Migration? 

Comments 

Single VDB – Refresh 

No 

This functionality can easily be reproduced with the dxtoolkit’s `dx_refresh_db` command. 

Single VDB – Create and Share Bookmark 

Yes 

Engine bookmarks can be managed through `dx_ctl_bookmarks`. However, unlike DCT, Self-Service bookmarks cannot be migrated to Engine bookmarks 

Multi VDB - Refresh 

Yes 

This functionality can be built through individual management of the VDBs, but it can be challenging if there are strong dependencies (such as Oracle EBS). 

Multi VDB - Create and Share Bookmark 

Yes 

See Single VDB – Bookmark Restore and Share 

Account Creation  

No 

`dxt_ctl_users` will continue to function 

Account Access Management  

No 

No equivalent to allow targeted VDB access. 

 

Whether you choose to move over to dct-toolkit or stay with dxtoolkit, all VDBs must be disassociated with their Self Service templates and containers by Continuous Data’s September 2025 release. 

 

What are the common dct-toolkit examples? 

Now that you have a good handle on the differences between dxtoolkit vs dct-toolkit and the scenarios that will require migration, let’s jump into various examples. Beware, not all commands can be copied directly into your terminal. Many will require some modification locally to specify valid VDBs, bookmarks, etc. 

 

Installation, Configuration, and Help 

Previously, dxtoolkit users would need to define connection and authorization information for each engine. With the dct-toolkit, only the credentials to DCT are required and thus, connectivity is drastically simplified.  

Next, to see the available commands, simply run `. /dct-toolkit –h`. This will list every available command. Then, to learn more about a specific command, run ./dct-toolkit <COMMAND> -h  to see full helpful documentation.

For example: ./dct-toolkit.exe refresh_vdb_by_snapshot –h

 

We recommend you use the -h argument frequently! All text is populated based on your installed version of DCT. Therefore, you can see new commands or documentation updates when DCT is upgraded. 

This blog post was written against DCT v17 (May 2024’s release). 

 

VDB Provision  

Create a new virtual dataset 

Create a new VDB. The various provision sub-commands are dependent on the database type.  

  1. Provision an Oracle VDB 
     
    ./dct-toolkit.exe provision_vdb_by_snapshot oracle name=<NEW VDB NAME> source_data_id=<SOURCE VDB OR DSOURCE NAME OR ID>... 
     
    Inserting image... 
     

  1. Provision a Microsoft SQL Server VDB 
     
    ./dct-toolkit.exe provision_vdb_by_snapshot mssql name=<NEW VDB NAME> source_data_id=<SOURCE VDB OR DSOURCE NAME OR ID>...
     

  1. Provision a PostgreSQL (or another AppData) VDB 
     
    ./dct-toolkit.exe provision_vdb_by_snapshot appdata name=<NEW VDB NAME> source_data_id=<SOURCE VDB OR DSOURCE NAME OR ID>... 

Note: AppData is the generic term that refers to all database types that are managed through the connector’s plugin ecosystem. 

 

We recommend using the provision_vdb_by_snapshot –h argument to review all available input parameters. Every dataset type requires a different set of properties.  

 

Once provisioned, you can use this VDB in any of the following commands or instead, simply use a VDB that already exists. 

 

Single VDB Management 

Refresh from Parent’s Latest Snapshot 

Refresh a VDB with the parent’s latest snapshot. 

  1. List all VDBs and record the VDB Name or ID to update. 
     
    ./dct-toolkit.exe get_vdbs
     
     

 

  1. Refresh data from its parent’s latest snapshot.

    ./dct-toolkit.exe 
    refresh_vdb_by_snapshot vdb_id=<THE CHILD VDB’S NAME OR ID>

 

 

By default, the `refresh_vdb_by_snapshot` command will grab the latest snapshot from the VDB’s parent. Follow the example below to specify a specific snapshot. 

 

Refresh from a Snapshot 

Refresh the VDB with a chosen dSource snapshot. (Formally known as Restore.) 

  1. List a dSource snapshots and record the ID. 
     

./dct-toolkit.exe get_dsource_snapshots dsource_id=<DSOURCE NAME OR ID>
 
 

  1. List all VDBs and record a compatible VDB Name or ID. 
     
    ./dct-toolkit.exe get_vdbs
     
     
     

Note: A dSource and VDBs are compatible” with one another if they originate from the same dSource. 

  

  1. Refresh the VDB to the chosen snapshot. 
     
    ./dct-toolkit.exe refresh_vdb_by_snapshot vdb_id="<VDB NAME OR ID>” snapshot_id=”<DSOURCE’s SNAPSHOT ID>
     
     

 

You will receive the following error if the snapshot is incompatible with the specified VDB: 
“Refresh to an object with a different root container is not supported.” 
 

Refresh from a Timestamp 

Refresh the VDB with a selected timestamp from within itself. (Formally known as Restore.) 

  1. List the available snapshots on the VDB to identify a rough range. 
     
    ./dct-toolkit.exe get_vdb_snapshots vdb_id=<VDB NAME OR ID>
     
     
     

  1. Get the details from one or more snapshots to target an available range.  
     
    ./dct-toolkit.exe get_snapshot_timeflow_range snapshot_id=<SNAPSHOT ID OR NAME>
     
     
     

  1. Refresh the VDB using a timestamp. 
     
    ./dct-toolkit.exe refresh_vdb_by_timestamp vdb_id=<VDB NAME OR ID> timestamp=<TIMESTAMP>
     
     

 

 
Note 1: The Timestamp is provided in ISO 8601 format. (Example: yyyy-MM-dd’T’HH:mm:ssZ) 

Note 2: By default, every snapshot has a timestamp. However, to enable a continuous range of timestamps, LogSync (or Point-in-Time Recovery) must be enabled for your selected VDB or dSource dataset. 

You will receive the following error if the timestamp is unavailable to the selected VDB: 
Cannot find provisionable point <timestamp> for source <VDB ID>

 

Refresh from a Bookmark 

Restore a VDB with any compatible bookmark. 

  1. List compatible bookmarks and record the name or ID. 
     
    ./dct-toolkit.exe get_bookmarks_by_vdb vdb_id="<VDB NAME OR ID>"
     
     
     

  1. Refresh a VDB to the specified bookmark. 
     
    ./dct-toolkit.exe refresh_vdb_from_bookmark vdb_id="<VDB NAME OR ID>" bookmark_id="<BOOKMARK NAME OR ID>" 
     
     

 

Similar functionality can be achieved for “snapshots” with equivalent get_snapshot_by_id and refresh_vdb_by_snapshot commands. 

 

Create, Share, and Delete a Bookmark 

Create a new bookmark on a VDB and then share it with compatible VDBs. 

  1. Create a new VDB bookmark.

    ./dct-toolkit.exe create_bookmark name="<NEW BOOKMARK NAME>" vdb_ids="<VDB ID OR NAME>"
     
     
     

  1. Confirm the bookmarks’ details. 
     
    ./dct-toolkit.exe get_bookmark_by_id bookmark_id="<BOOKMARK NAME OR ID>" 

     
     

  1. Share the bookmark with other compatible VDBs. 
     
    ./dct-toolkit.exe update_bookmark bookmarkId="<BOOKMARK NAME OR ID>" bookmark_type=PUBLIC
     
     
     

  1. Delete the bookmark. 
     
    ./dct-toolkit.exe delete_bookmark bookmark_id="<BOOKMARK NAME OR ID>"
     
     
     
     

Multi VDB Management 

Create a VDB Group 

Combine multiple existing VDBs into a new VDB Group. 

  1. Create the VDB Group from two VDBs. 
     
    ./dct-toolkit.exe create_vdb_group vdb_ids= "<VDB 1 NAME OR ID>" vdb_ids="<VDB 2 NAME OR ID>" name="<NEW VDB GROUP NAME>" 
     
     
     

  1. Add Tags to the VDB Group to assist with organization or access control.  
     
    ./dct-toolkit.exe create_vdb_groups_tags vdb_group_id="<VDB GROUP NAME OR ID>" tags='<KEY1>=<VALUE2>' '<KEY2>=<VALUE2>'
     
     
     

  1. Confirm your VDB Group is configured correctly. 
     
    ./dct-toolkit.exe get_vdb_groups
     
     
     

Create, Share, and Refresh Bookmark 

Pardon our dust! 
 
We are expanding heavily on our VDB Group refresh and bookmark management in the second half of 2024. This section will be revamped once implemented. 

Account and Access Management

Full user management is a detailed topic that requires comprehension of our Attribute-Based Access Control model. If you are unfamiliar with it, we recommend first reading our Data Control Tower (DCT) - Access Control Implementation blog post. Once you grasp how accounts, access groups, roles, and scopes interact, the following commands will make more sense. 

We do not recommend heavy access management through the dct-toolkit. The DCT user interface provides a better customization experience. Once set up, modifications are rarely needed outside of tag additions 

The below examples provide ways to create accounts and perform minor permission management. 

 

Account Creation 

Create a new account with an email, password, API Key, and automatically assign the user to an Access Group. 
 

  1. Create the user with the required attributes. 
     
    ./dct-toolkit.exe create_account email=<EMAIL> username=<USERNAME> password="<PASSWORD>" tags=<KEY>=<VALUE> --generate_api_key
     

 

This example takes advantage of Access Group assignment by using tags.  

Accounts can be manually assigned to Access Groups by using the add_access_group_accounts_ids command. 

 

Access 

There are primarily three different ways in which object access, such as VDBs and dSources, can be granted to a specific user. Follow the examples below to learn more. 

 

Owner 

By default, if a user creates an object, that user will automatically have the `owner` access to that object. This is useful if your users create their own objects or an API Key is shared across a team. 

This can be confirmed by creating any object and then navigating to the Account’s Individual access group and reviewing its contents. 

 

Direct Assignment 

Due to DCT’s attribute-based access control model, this is considered an advanced assignment mechanism. It is not recommended for early DCT configurations. This strategy leverages the “Scoped - Objects Mapping” or “Advanced Scoping” assignment modes.  

We recommend implementing some form of Direct Assignment via the UI first before managing access via the CLI.  

Tags (see below) is a much simpler model. 

 

Tags  

We recommend tags as the primary mechanism to manage wide scale object access control. This leverages the “Scoped - Tag Mapping” assignment mode. This typically requires configuration outside of the CLI to set up the framework, but then you can easily add new object tags through the following commands. 

  1. Identify a VDB, or other object, and review its contents. 
     
    ./dct-toolkit.exe get_vdb_by_id vdb_id="<VDB NAME OR ID>
     
     

This VDB does not have any tags. 

 

  1. Add tags to the object. 
     
    ./dct-toolkit.exe create_vdb_tags vdb_id=" VDB NAME OR ID> " tags=<KEY 1>=<VALUE 1>' '<KEY 2>=<VALUE 2>'
     

 

In this example, if access is controlled by the “Team=Alpha” or “Geo=US” tags, the users within that access group will now have access to those resources defined by the role’s scope. It’s often best to attempt this workflow through the UI first and then automate it second. 

 

Closing 

We hope you found this blog post valuable when migrating from the dxtoolkit and hitting the September 2025 end of life date! While we only covered a subset of common Self-Service dct-toolkit commands, there are over 200+ others to explore. New features are being added frequently, so be sure to keep an eye on the DCT release notes to learn about new commands and use cases. 

If you have any questions or need some pointers on suggested commands, please leave a comment below. 

0 comments
26 views

Permalink