DCT Self-Service Migration

Summary

Data Control Tower (DCT) v7 offers new self-service capabilities! This is an alternative offering for the existing Delphix Continuous Data Self-Service (Jetstream) approach. Due to DCT's central gateway model, you are able to leverage the same self-service capabilities, but now across your entire Delphix Platform. For example, you may now provision a VDB on Engine A, create a bookmark, and provision a VDB on Engine B using that same Bookmark. In addition, DCT's newer functionality, such as VDB Groups and Integrations, enables previously challenging or impossible scenarios.

While organizations can simultaneously take advantage of DCT and the Delphix Continuous Data Self-Service solutions, existing VDBs and Bookmarks cannot be shared. Therefore, organizations much discontinue their usage of Jetstream before they can fully migrate. The below directions are intended to help customers perform the migration from Engine to DCT in a safe and orderly manner. 

Pre-requisites

• [Required] Installation of Data Control Tower with one or more connected Continuous Data engines.

• [Required] Download the DCT Self-Service Migration scripts onto a Linux environment. (Delphix Support account required)

• Ensure the jq package (Linux 64-bit) is available at runtime.

• The Linux runtime environment location should have network access to both the DCT server and CD engines.

• [Optional] Configuration of Access Controls.

Directions

The following steps will recreate Engine self-service bookmarks in DCT and ensure your self-service environment is ready. This will break the self-service experience in the Engine. These steps should be repeated once per engine. We recommend following these steps to completion once on a test engine, before parallelizing across multiple engines.

Step 1: Record Engine Bookmarks

Summary: Identify the list of the Continuous Data Engine’s Self-Service Container Bookmarks and their underlying snapshots

The first step includes running the “Extract Snapshot Details” Bash script. A sample command looks like the following:

sh extract_ss_boookmark_details.sh  <Delphix Continuous Data Engine IP> <Admin Username> <Admin Password>

On submission, the script will scan your entire Continuous Data Engine and return a CSV file called <engine>_snapshot_details_<timestamp>.csv.This file contains a list of every Self-Service bookmark and its associated snapshots. We recommend scanning through this list to ensure all bookmarks are available and none are missing. We will use this file in the subsequent step.

Manual Validation [Optional]

We can quickly validate the CSV’s contents manually by navigating to the Engine’s Self-Service portal, identifying the “SS_BMK_TEMPLATENAME” value, such as “PostgresFS Demo”, and selecting it.

Next, match a Container Name with the “SS_BMK_CONTAINERNAME” values, such as “PostgresFS QA”, and select “Data Operations”.

Lastly, navigate to the Bookmarks tab and match the bookmarks presented here with those available in the snapshot_details_<timestamp>.csv file. 

If you happen to see duplicate bookmarks to different sources, then confirm by navigating to the “Sources” tab. You should have one set of bookmarks per source.

Step 2: Create DCT Bookmarks

Summary: Create equivalent Self-Service Bookmarks in DCT

Next, we create equivalent DCT bookmarks. We do this by using the second “DCT Migrate Bookmarks” Bash script. Run it as follows:

sh dct_migrate_ss_bookmarks.sh <DCT Host IP Address> <DCT Admin APK Key> <ENGINE_snapshot_details_TIMESTAMP.csv> <Delphix Continuous Data Engine IP>

On submission, this script will scan the provided snapshot_details CSV file, iterate through each bookmark, and create a new DCT bookmark based on the mapped snapshot. The expiration date will be set based on the original Bookmark’s expiration date.

If successful, the DCT’s Data > Bookmark tab will look like the following:

Note 1: If you want new bookmark names, update the CSV file or within DCT’s UI.

Note 2: The original bookmark’s timestamp will not be preserved. All DCT bookmark timestamps will match the timestamp at runtime.

Note 3: At this time, the DCT Migrate Bookmarks script will not maintain the grouped relationship of containers with multiple sources. This entity is called a “VDB Group” in DCT. It is not visible in the UI as of DCT v7. However, if you wish to leverage this capability, then it can be managed through the API.

After this point, you can leverage these bookmarks to create new VDBs. However, they cannot be used to refresh the source VDBs that are attached to self-service containers. We will tackle this in the next step.

Manual Bookmark Creation [Optional]

Navigate to DCT’s Swagger UI (https://<DCT_URL>/static/) and call DCT’s GET /snapshots/ endpoint. This will return a full list of all known DCT snapshots. Next, iterate through Step 1’s <engine>_snapshot_details_<timestamp>.csv file, and match the bookmark’s “SNAPSHOT_NAME”  with the DCT Snapshot’s “name” value. Once identified, record the DCT Snapshot’s id. To maintain this information, I recommend a separate table that looks like the following:

Note: The DCT Snapshot ID is similar to the engine’s Snapshot ID. However, the `1-`, `2-`, `3-`, etc value at the front indicates the DCT’s assigned engine ID. DCT does this to prevent conflicting values.

Next, navigate to DCT’s POST /bookmarks endpoint. You will want to run this command for each Self-Service bookmark you wish to recreate. As a part of this process, we recommend adding appropriate Tags if they have been configured. The JSON payload might look like the following:

{

  "name": "<SS_BMK_NAME>",

  "snapshot_ids": [ "<DCT SNAPSHOT ID>" ],

  "expiration": "<SS_BMK_TIMESTAMP AS YYYY-MM-DD>",

  "retain_forever": false,

  "tags": [ { "key": "Team", "value": "Alpha" } ]

}

On creation, the bookmark will be available through the DCT UI.

Step 3: Drop Engine Containers

Summary: Drop the Self-Service Container, so they can be refreshed within DCT. Do not delete VDBs and vFiles.

Warning: This step is final. There is no going back.

In order to make use of these newly created Bookmarks on their associated VDBs, you will need to drop the VDB from its Container. Please recognize that by doing so, it is an unrecoverable act and all associated bookmarks will be deleted. However, by creating the bookmarks in the prior step, you will preserve the underlying snapshots.

You can drop the VDBs from the Container by following the below steps. Repeat the steps for every Container that contains a VDB you wish to manage through DCT. We recommend you attempt it with a test container first and validate that refreshing is possible before all containers are dropped. 

First, identify a data template and select it:

Next, navigate to the Containers tab, select the Container you wish to delete, and select the trashcan icon. As a reminder, do not select the “Delete associated VDBs and vFiles.”

Finally, navigate to the VDB in DCT and validate refreshing the bookmark completes successfully. Application teams must now go through the DCT UI to create bookmarks and refresh their VDBs. Functionality through the Engine directly is still possible.

Step 4: Confirm DCT Refresh

Summary: Validate VDB refreshes are possible through DCT’s user experience.

Once the container has been deleted, navigate back to DCT, identify the VDB that was dropped from the engine container, select the “Bookmarks” tab, and run a Refresh action through one of the buttons shown below.

For those who are familiar with Delphix Continuous Data, the following list might help translate the engine’s Self-Service actions into DCT Self-Service equivalents:

¹  Please note that DCT’s “Share” functionality is implemented through the role-based access model. Therefore, to share with your team members, you must have permission on that bookmark to update the Tags accordingly which in turn grants permission for other users. Tags can be assigned on creation or afterward. Optionally, Administrators can directly add available Bookmarks through the Access Group > Roles > Edit. 

Delphix Engine Self-Service
Data Control Tower Refresh Self-Service

Congratulations! At this point, your Application Teams are able to switch their Self-Service approach from the Continuous Data engine to DCT! We encourage you to leverage our various other resources to enable your teams. 

For advanced users, you can now leverage DCT APIs or integrations and expand DevOps practices easily across the organization.

Frequently Asked Questions

Question: As an Admin I can see the newly created bookmarks and provision VDBs, but my users cannot. Where are they?

Answer: Today, the migration scripts do not assign Tags because the Engine Self-Service model did not use tags in the same way. Therefore, you must manually assign Tags to each bookmark and VDB. This can be automated if desired using the DCT APIs. If Tagging is new, then I recommend you check out my other Access Model blog post.

Questions: Which version of the Delphix engine have these scripts been validated on? 

Answer: We built these scripts using v6.0.17.2 which is API v1.11.17. Source Before you migrate to DCT, I recommend you are at least on this engine version. 

Question: I am having trouble running your scripts. Will you help me? 

Answer: I will try my best to answer questions in the comments selection below and add new FAQs as needed. The support team is available to answer general DCT questions, but not specifically regarding these scripts. Delphix services team is always available.