In March 2024, Delphix Self-Service (Jet Stream) was announced to have end-of-life and to be replaced with the Data Control Tower (DCT) Developer Experience offering. The popular refresh, bookmark, and share feature set is available in DCT along with new possibilities, such as expanded data restores and DevOps Integrations. However, it is important to understand how to replicate the permission model, container structure, and bookmarks end users rely on in Delphix Self-Service. This blog post aims to teach Delphix Administrators who have successfully installed DCT but need guidance on migrating their virtual databases (VDBs), bookmarks, and end users to minimize downtime.
If you are new to DCT, we recommend watching the Adopting Data Control Tower for Self-Service webinar. This guide assumes that you have successfully installed Data Control Tower and can navigate through the user interface and swagger API. These prerequisites alongside this blog will help migrate your Self-Service end users.
Scroll to the bottom of this blog to find a video of what the Delphix Self-Service migration process looks like.
In DCT v17, we introduced new capabilities to assist in the migration for self-service. They fall under three categories:
It is important to understand how these three features function as they can have major effects across your Delphix DevOps Platform.
DCT is a data governance solution that communicates constantly with its connected Delphix Continuous Data Engines. Once DCT establishes a connection with an engine or after an upgrade, DCT will start to poll the existing Delphix Self-Service container bookmarks and re-creating them within DCT. This can be confirmed by an administrative user when navigating to the Data > Bookmarks tab and reviewing the presented table.
However, these are not “full” DCT bookmarks because they are still under the management of the Delphix Continuous Data Engine. You can easily discern the bookmark’s management type by navigating to the Data > Bookmarks table, selecting the column modifier option, and enabling the “Managed By” column.
You will see a list of bookmarks with the “DCT” or “Engine” label.
“DCT” labeled bookmarks were created directly through the DCT UI or API.
“Engine” labeled bookmarks were created within Delphix Self-Service.
Engine bookmarks can be used to provision new or refresh other VDBs. However, because they are Engine-managed, modification in the Self-Service portal is still possible. Any modifications to these bookmarks on the engine will be reflected in DCT on the next telemetry run. These bookmarks will be converted to DCT bookmarks by completing the below workflow.
The automated bookmark telemetry is initiated on the following triggers:
Initial engine connection
On DCT Start
Every four hours
Manually through the /bookmarks/import-engine-bookmarks/{engineId} API.
Note: Migration of template or container bookmarks with multiple VDBs, called a VDB Group in DCT, is not supported today. Support is coming soon! In the meantime, we still recommend you migrate single VDB bookmarks.
One of the most commonly requested Delphix feature requests has been customization of the Access Control model. DCT has successfully implemented a customizable attribute-based solution. However, to help customers to onboard quickly, DCT v17 implemented a mechanism to automate the recreation of the Delphix Self-Service permission model.
In this model, we automatically create one Access Group for every Delphix Self-Service container with the <engine_name>-<container_name> name format. Each Access Group will have two scoped Roles, “self-service-data-operator-system-role” and “self-service-data-operator-system-role”. These scoped roles will then have the necessary VDB and dSource permissions to emulate the known Delphix Self-Service permission model. Optionally, every known user (identified by their email address) can be automatically added to the Access Group to simplify the rollout further.
This creates a permission model as shown in the following table:
Scoped Role Permissions
Access Group
Role
Scope Name
Scope
VDBs
dSources
Environments
Jobs
<engine_name>-<container_name>
self-service-data-operator-system-role
Objects Scoped Manually
Create Bookmark, Delete Bookmark, Disable, Enable, Lock, Read, Read Bookmark, Refresh, Refresh from Bookmark, Refresh from Location, Refresh from Snapshot, Refresh from Timestamp, Start, Stop, Switch Timeline, Unlock, Update Bookmark
Read
self-service-data-viewer-system-role
Read, Read Bookmark, Refresh from Bookmark, Refresh from Location, Refresh from Snapshot, Refresh from Timestamp
The default permission model should be perfect for initial usage. Unlike a typical System Role, these self-service roles can have their permissions updated. Beware, any modification of these default roles will affect all resulting Access Groups.
As your administrative team’s understanding of end-user requirements grows, we encourage you to build custom roles as our default System Roles cannot be modified. If you are unfamiliar DCT Access Control Model, then we recommend you review the Data Control Tower (DCT) - Access Control Implementation blog post.
Alongside the construction of the Delphix Self-Service permission model, we will also begin dropping the Virtual Databases (VDBs) and deleting the Self-Service containers. The controlled deletion through the DCT API will ensure the VDB is dropped (not deleted), the container is deleted, and all bookmarks are migrated over correctly. This is a final action.
Ignoring this process and dropping containers directly through Delphix Self-Service will result in unrecoverable bookmarks.
If you have previously attempted a DCT action on a VDB that is still managed by a Self-Service container, you might have seen an error message that prevents its manipulation. This process will resolve that problem.
The below process will help you migrate your Delphix Self-Service users, permissions, and bookmarks into DCT. As some of these steps cannot be reversed, we highly recommend you attempt the migration on a low priority or test the Self-Service container before you try more critical VDBs. In addition, we recommend you review our DCT for Self-Service End Users training material and ensure all end users first understand DCT.
Install Data Control Tower v17+ and establish a connection with your Delphix Continuous Data Engines. On connection, DCT will automatically begin the import of all Self-Service VDBs and their bookmarks through its telemetry. Depending on how many bookmarks you have, it might take an hour or two to synchronize all of them after the engine’s connection.
At a minimum, you should confirm the following details:
The users who have access to these containers must be notified.
The bookmarks exist within DCT.
The VDBs exist within DCT. (In DCT, users will interact with the VDBs directly, not the containers.)
Once confirmed, record the Template or Container name(s). This will be used in the next step. As you work through this process, you will slowly formulate a plan to migrate all VDBs over.
Next, navigate to the DCT Swagger API (https://<dct-hostname>/static/index.html), find, and run the following API, /data-layouts. Match the Container and Template names retrieved in the prior step with the IDs presented here. This is the Data Layout ID (dataLayoutId).
The format for the Data Layout Id is the <engine-id>-<jetstream container id>. For example, if the id is “1-JS_DATA_CONTAINER-68”, then `1` is the DCT-given engine ID and `JS_DATA_CONTAINER-68` is the container id in Delphix Self-Service.
Note: If you are receiving 401 unauthorized errors when running the API calls, make sure you specify your API Key with the following format: apk 1.123…
apk 1.123…
Still within the DCT Swagger API, find, and run the following API, /data-layouts/{dataLayoutId}/convert-and-drop where the dataLayoutId is supplied with the Data Layout Id identified in the prior step. This is a one-way action that will perform the following:
Imports the bookmarks (i.e marks the bookmarks as DCT managed)
Imports any identified Accounts from the Delphix Continuous Data Engine based on matching email addresses.
Creates Access Groups named after each Container.
And upon completion, delete the Self-Service Template and Containers.
Afterward, if you navigate to the DCT Admin > Access Group page, you will see the working Access Group with all Accounts correctly assigned and permissions defined.
At this point, the Delphix Self-Service VDBs have been disconnected from Delphix Self-Service containers. Inform your end users that they can log in to the DCT interface and perform their Self-Service VDB activities within DCT!
Based on the number of Delphix Self-Service end users, containers, and templates you will need to migrate, this process might need to be repeated a handful of times. We highly recommend it is done purposefully with the lowest priority VDBs migrated first and then the highest priority VDBs last. This will ensure you feel confident from an administrative perspective and your organization can help one another as they learn the DCT UI and API.
I hope this blog was informative. If you have questions, please comment below or contact your Delphix Account Team for additional guidance.
#DCT#self-service#self_service#developer#jetstream