Delphix Products

 View Only

Delphix Self-Service to DCT: Bookmark, VDB, and User Migration

By Nicholas Mathison posted 05-31-2024 02:03:41 PM

  

October 2024 Update

When this blog post was originally written, we had not yet introduced support for multi-VDB container migration and its replacement functionality, the VDB Group, was minimal. In September's v22 release, that all changed! If your Delphix Self-Service configuration takes advantage of a single container with multiple VDBs, we highly recommend DCT v22+ before continuing. 

Learn more about the DCT v22 release here! 
 

Summary 

In March 2024, Delphix Self-Service (Jet Stream) was announced to have end-of-life and to be replaced with the Data Control Tower (DCT) Developer Experience offering. The popular refresh, bookmark, and share feature set is available in DCT along with new possibilities, such as expanded data restores and DevOps Integrations. However, it is important to understand how to replicate the permission model, container structure, and bookmarks end users rely on in Delphix Self-Service. This blog post aims to teach Delphix Administrators who have successfully installed DCT but need guidance on migrating their virtual databases (VDBs), bookmarks, and end users to minimize downtime.  

If you are new to DCT, we recommend watching the Adopting Data Control Tower for Self-Service webinar. This guide assumes that you have successfully installed Data Control Tower and can navigate through the user interface and swagger API. These prerequisites alongside this blog will help migrate your Self-Service end users. 

Scroll to the bottom of this blog to find a video of what the Delphix Self-Service migration process looks like.

Component Overview 

In DCT v17, we introduced new capabilities to assist in the migration for self-service. They fall under three categories: 

  • Automatic import of the Delphix Self-Service template and container bookmarks
  • Generation of a Delphix Self-Service permission model
  • Controlled deletion of Delphix Self-Service containers 

It is important to understand how these three features function as they can have major effects across your Delphix DevOps Platform.  

 

Automatic import of the Delphix Self-Service container bookmarks 

DCT is a data governance solution that communicates constantly with its connected Delphix Continuous Data Engines. Once DCT establishes a connection with an engine or after an upgrade, DCT will start to poll the existing Delphix Self-Service container bookmarks and re-creating them within DCT. This can be confirmed by an administrative user when navigating to the Data > Bookmarks tab and reviewing the presented table 

 

However, these are not “full” DCT bookmarks because they are still under the management of the Delphix Continuous Data Engine. You can easily discern the bookmark’s management type by navigating to the Data > Bookmarks table, selecting the column modifier option, and enabling the “Managed By column.  

 

 

You will see a list of bookmarks with the “DCT orEngine label.  

  • “DCT” labeled bookmarks were created directly through the DCT UI or API.  

  • “Engine” labeled bookmarks were created within Delphix Self-Service.  

 

Engine bookmarks can be used to provision new or refresh other VDBs. However, because they are Engine-managed, modification in the Self-Service portal is still possible. Any modifications to these bookmarks on the engine will be reflected in DCT on the next telemetry run. These bookmarks will be converted to DCT bookmarks by completing the below workflow. 

 

The automated bookmark telemetry is initiated on the following triggers: 

  • Initial engine connection 

  • On DCT Start 

  • Every four hours 

  • Manually through the /bookmarks/import-engine-bookmarks/{engineId} API. 

 

Generation of the Delphix Self-Service permission model 

One of the most commonly requested Delphix feature requests has been customization of the Access Control model. DCT has successfully implemented a customizable attribute-based solution. However, to help customers to onboard quickly, DCT v17 implemented a mechanism to automate the recreation of the Delphix Self-Service permission model.  

 

In this model, we automatically create one Access Group for every Delphix Self-Service container with the <engine_name>-<container_name> name format. Each Access Group will have two scoped Roles, “self-service-data-operator-system-role” and “self-service-data-operator-system-role”. These scoped roles will then have the necessary VDB and dSource permissions to emulate the known Delphix Self-Service permission model. Optionally, every known user (identified by their email address) can be automatically added to the Access Group to simplify the rollout further. 

 

This creates a permission model as shown in the following table: 

 

 

 

 

Scoped Role Permissions 

Access Group 

Role 

Scope Name 

Scope 

VDBs 

dSources 

Environments 

Jobs 

<engine_name>-<container_name> 

self-service-data-operator-system-role 

self-service-data-operator-system-role 

Objects Scoped Manually 

Create Bookmark, Delete Bookmark, Disable, Enable, Lock, Read, Read Bookmark, Refresh, Refresh from Bookmark, Refresh from Location, Refresh from Snapshot, Refresh from Timestamp, Start, Stop, Switch Timeline, Unlock, Update Bookmark 

 

Read 

Read 

<engine_name>-<container_name> 

self-service-data-viewer-system-role 

self-service-data-viewer-system-role 

Objects Scoped Manually 

Read, Read Bookmark, Refresh from Bookmark, Refresh from Location, Refresh from Snapshot, Refresh from Timestamp 

Read, Read Bookmark, Refresh from Bookmark, Refresh from Location, Refresh from Snapshot, Refresh from Timestamp 

 

 

 

The default permission model should be perfect for initial usage. Unlike a typical System Role, these self-service roles can have their permissions updated. Beware, any modification of these default roles will affect all resulting Access Groups. 

 

As your administrative team’s understanding of end-user requirements grows, we encourage you to build custom roles as our default System Roles cannot be modified. If you are unfamiliar DCT Access Control Model, then we recommend you review the Data Control Tower (DCT) - Access Control Implementation blog post. 

 

Controlled deletion of Delphix Self-Service containers 

Alongside the construction of the Delphix Self-Service permission model, we will also begin dropping the Virtual Databases (VDBs) and deleting the Self-Service containers. The controlled deletion through the DCT API will ensure the VDB is dropped (not deleted), the container is deleted, and all bookmarks are migrated over correctly. This is a final action. 

 

Ignoring this process and dropping containers directly through Delphix Self-Service will result in unrecoverable bookmarks. 

 

If you have previously attempted a DCT action on a VDB that is still managed by a Self-Service container, you might have seen an error message that prevents its manipulation. This process will resolve that problem. 

Migration Procedure 

The below process will help you migrate your Delphix Self-Service users, permissions, and bookmarks into DCT. As some of these steps cannot be reversed, we highly recommend you attempt the migration on a low priority or test the Self-Service container before you try more critical VDBs. In addition, we recommend you review our DCT for Self-Service End Users training material and ensure all end users first understand DCT. 

 

  1. Install Data Control Tower v17+ and establish a connection with your Delphix Continuous Data Engines. On connection, DCT will automatically begin the import of all Self-Service VDBs and their bookmarks through its telemetry. Depending on how many bookmarks you have, it might take an hour or two to synchronize all of them after the engine’s connection. 

    1. If you are upgrading DCT to v17+, then the VDBs should already exist and only the bookmarks will be created.
    2. As explained above, these Self-Service bookmarks will be labeled in the “Managed By” field with “Engine”. You will need to enable the hidden “Managed By” column to see these values. 
    3. If you wish to perform some light testing, these “Engine” managed bookmarks can refresh non-Service-Self managed VDBs or provision new VDBs.

  1. Next, identify the list of Self-Service VDBs to migrate to DCT and the users you will need to enable by container or template. This is performed by logging into the Delphix Self-Service UI, reviewing the presented templates and their containers, scanning the User list, and confirming they are all ready for migration.  


At a minimum, you should confirm the following details:

    1. The users who have access to these containers must be notified.

    2. The bookmarks exist within DCT.

    3. The VDBs exist within DCT. (In DCT, users will interact with the VDBs directly, not the containers.) 

Once confirmed, record the Template or Container name(s). This will be used in the next step. As you work through this process, you will slowly formulate a plan to migrate all VDBs over.  

 

  1. Next, navigate to the DCT Swagger API (https://<dct-hostname>/static/index.html), find, and run the following API, /data-layouts. Match the Container and Template names retrieved in the prior step with the IDs presented here. This is the Data Layout ID (dataLayoutId). 

 

The format for the Data Layout Id is the <engine-id>-<jetstream container id>. For example, if the id is “1-JS_DATA_CONTAINER-68”, then `1` is the DCT-given engine ID and `JS_DATA_CONTAINER-68` is the container id in Delphix Self-Service. 
  

Note: If you are receiving 401 unauthorized errors when running the API calls, make sure you specify your API Key with the following format: apk 1.123… 

 

  1. Still within the DCT Swagger API, find, and run the following API, /data-layouts/{dataLayoutId}/convert-and-drop where the dataLayoutId is supplied with the Data Layout Id identified in the prior step.  
     
    This is a one-way action that will perform the following:

    1. Imports the bookmarks (i.e marks the bookmarks as DCT managed)

    2. Imports any identified Accounts from the Delphix Continuous Data Engine based on matching email addresses.

    3. Creates Access Groups named after each Container.

    4. And upon completion, delete the Self-Service Template and Containers. 

Afterward, if you navigate to the DCT Admin > Access Group page, you will see the working Access Group with all Accounts correctly assigned and permissions defined. 

 

  1. At this point, the Delphix Self-Service VDBs have been disconnected from Delphix Self-Service containers. Inform your end users that they can log in to the DCT interface and perform their Self-Service VDB activities within DCT! 

Based on the number of Delphix Self-Service end users, containers, and templates you will need to migrate, this process might need to be repeated a handful of times. We highly recommend it is done purposefully with the lowest priority VDBs migrated first and then the highest priority VDBs last. This will ensure you feel confident from an administrative perspective and your organization can help one another as they learn the DCT UI and API. 

 

Video Walkthrough

If you wish to automate this process, we recommend you grab a full list of the container and template IDs. Identify the batches of containers and templates you want to migrate and then feed them into a script using the DCT Toolkit or APIs. All containers within a template must be migrated before the template itself.

I hope this blog was informative. If you have questions, please comment below or contact your Delphix Account Team for additional guidance. 

 

 


#DCT
#self-service
#self_service
#developer
#jetstream

0 comments
62 views

Permalink