Delphix Products

 View Only

Data Control Tower (DCT) - Self Service Migration Steps

By Nicholas Mathison posted 05-17-2023 04:21:38 PM

  

DCT Self-Service Migration

Summary

Data Control Tower (DCT) v7 offers new self-service capabilities! This is an alternative offering for the existing Delphix Continuous Data Self-Service (Jet Stream) approach. Due to DCT's central gateway model, you can leverage the same self-service capabilities, but now across your entire Delphix Platform. In addition, DCT's newer functionality, such as VDB Groups and Integrations, enables previously challenging or impossible scenarios. 

While organizations can simultaneously take advantage of DCT and the Delphix Continuous Data Self-Service solutions, existing VDBs and Bookmarks cannot be shared. Therefore, organizations much discontinue their usage of Jetstream before they can fully migrate. The below directions are intended to help customers perform the migration from Engine to DCT in a safe and orderly manner. 

Pre-requisites

 

Directions

 

The following steps will recreate Engine self-service bookmarks in DCT and ensure your self-service environment is ready. This will break the self-service experience in the Engine. These steps should be repeated once per engine. We recommend following these steps to completion once on a test engine, before parallelizing across multiple engines.

 

Step 1: Record Engine Bookmarks

Summary: Identify the list of the Continuous Data Engine’s Self-Service Container Bookmarks and their underlying snapshots


The first step includes running the “Extract Snapshot Details” Bash script. A sample command looks like the following:

sh extract_ss_boookmark_details.sh  <Delphix Continuous Data Engine IP> <Admin Username> <Admin Password>

On submission, the script will scan your entire Continuous Data Engine and return a CSV file called <engine>_snapshot_details_<timestamp>.csv.This file contains a list of every Self-Service bookmark and its associated snapshots. We recommend scanning through this list to ensure all bookmarks are available and none are missing. We will use this file in the subsequent step.


Manual Validation [Optional]

We can quickly validate the CSV’s contents manually by navigating to the Engine’s Self-Service portal, identifying the “SS_BMK_TEMPLATENAME” value, such as “PostgresFS Demo”, and selecting it.

Next, match a Container Name with the “SS_BMK_CONTAINERNAME” values, such as “PostgresFS QA”, and select “Data Operations”.

Lastly, navigate to the Bookmarks tab and match the bookmarks presented here with those available in the snapshot_details_<timestamp>.csv file. 

If you happen to see duplicate bookmarks to different sources, then confirm by navigating to the “Sources” tab. You should have one set of bookmarks per source.

Step 2: Create DCT Bookmarks

Summary: Create equivalent Self-Service Bookmarks in DCT


Next, we create equivalent DCT bookmarks. We do this by using the second “DCT Migrate Bookmarks” Bash script. Run it as follows:

sh dct_migrate_ss_bookmarks.sh <DCT Host IP Address> <DCT Admin APK Key> <ENGINE_snapshot_details_TIMESTAMP.csv> <Delphix Continuous Data Engine IP>

On submission, this script will scan the provided snapshot_details CSV file, iterate through each bookmark, and create a new DCT bookmark based on the mapped snapshot. The expiration date will be set based on the original Bookmark’s expiration date.

 

If successful, the DCT’s Data > Bookmark tab will look like the following:

Note 1: If you want new bookmark names, update the CSV file or within DCT’s UI.

Note 2: The original bookmark’s timestamp will not be preserved. All DCT bookmark timestamps will match the timestamp at runtime.

Note 3: At this time, the DCT Migrate Bookmarks script will not maintain the grouped relationship of containers with multiple sources. This entity is called a “VDB Group” in DCT. It is not visible in the UI as of DCT v7. However, if you wish to leverage this capability, then it can be managed through the API.

 

After this point, you can leverage these bookmarks to create new VDBs. However, they cannot be used to refresh the source VDBs that are attached to self-service containers. We will tackle this in the next step.


Manual Bookmark Creation [Optional]

Navigate to DCT’s Swagger UI (https://<DCT_URL>/static/) and call DCT’s GET /snapshots/ endpoint. This will return a full list of all known DCT snapshots. Next, iterate through Step 1’s <engine>_snapshot_details_<timestamp>.csv file, and match the bookmark’s “SNAPSHOT_NAME”  with the DCT Snapshot’s “name” value. Once identified, record the DCT Snapshot’s id. To maintain this information, I recommend a separate table that looks like the following:

 

Bookmark Name
(aka SS_BMK_NAME)

Bookmark Timestamp
(aka SS_BMK_TIMESTAMP)

Snapshot Name
(aka SNAPSHOT_NAME)

DCT Snapshot ID

Pre-test

2023-01-20T11:09:34.509Z

@2023-01-20T12:07:06.232

1-ORACLE_SNAPSHOT-1234

QA Error # 12345

2022-11-10T11:26:39.802Z

@2022-11-10T06:37:55.321

2-APPDATA_SNAPSHOT-2085

<From CSV>

<From CSV>

<From CSV>

<From DCT API output>

 

Note: The DCT Snapshot ID is similar to the engine’s Snapshot ID. However, the `1-`, `2-`, `3-`, etc value at the front indicates the DCT’s assigned engine ID. DCT does this to prevent conflicting values.

Next, navigate to DCT’s POST /bookmarks endpoint. You will want to run this command for each Self-Service bookmark you wish to recreate. As a part of this process, we recommend adding appropriate Tags if they have been configured. The JSON payload might look like the following:

 

{

  "name": "<SS_BMK_NAME>",

  "snapshot_ids": [ "<DCT SNAPSHOT ID>" ],

  "expiration": "<SS_BMK_TIMESTAMP AS YYYY-MM-DD>",

  "retain_forever": false,

  "tags": [ { "key": "Team", "value": "Alpha" } ]

}

 

On creation, the bookmark will be available through the DCT UI.

 

Step 3: Drop Engine Containers

Summary: Drop the Self-Service Container, so they can be refreshed within DCT. Do not delete VDBs and vFiles.

Warning: This step is final. There is no going back.


In order to make use of these newly created Bookmarks on their associated VDBs, you will need to drop the VDB from its Container. Please recognize that by doing so, it is an unrecoverable act and all associated bookmarks will be deleted. However, by creating the bookmarks in the prior step, you will preserve the underlying snapshots.

 

You can drop the VDBs from the Container by following the below steps. Repeat the steps for every Container that contains a VDB you wish to manage through DCT. We recommend you attempt it with a test container first and validate that refreshing is possible before all containers are dropped. 

 

First, identify a data template and select it:

Next, navigate to the Containers tab, select the Container you wish to delete, and select the trashcan icon. As a reminder, do not select the “Delete associated VDBs and vFiles.”

 

Finally, navigate to the VDB in DCT and validate refreshing the bookmark completes successfully. Application teams must now go through the DCT UI to create bookmarks and refresh their VDBs. Functionality through the Engine directly is still possible.



Step 4: Confirm DCT Refresh

Summary: Validate VDB refreshes are possible through DCT’s user experience.

Once the container has been deleted, navigate back to DCT, identify the VDB that was dropped from the engine container, select the “Bookmarks” tab, and run a Refresh action through one of the buttons shown below.




 


For those who are familiar with Delphix Continuous Data, the following list might help translate the engine’s Self-Service actions into DCT Self-Service equivalents:

Delphix Engine Self-Service Model

Data Control Tower Model

Data Control Tower APIs

Description

Refresh

VDB > Action > Refresh > Parent (Latest)

/vdbs/{vdbId}/refresh_by_timestamp
(No timestamp specified)

Sync with the parent’s (aka data template’s) latest data.

Restore (aka Rollback)

VDB > Action > Refresh > Self

/vdbs/{vdbId}/rollback_by_X

Update data to any point in time from itself. See other DCT options for other restore sources.

Reset (aka Rewind)

VDB > Action > Refresh >
Self (Latest)

/vdbs/{vdbId}/rollback_by_timestamp
(No timestamp specified)

Update data to the last marked point on itself.

N/A

VDB > Action > Refresh > Parent

/vdbs/{vdbId}/refresh_by_X

Update data to any available point in time from the parent.

N/A

VDB > Action > Refresh > Relative

/vdbs/{vdbId}/refresh_by_X

Update data to any defined point in time from any sibling.

Bookmark

VDB > Action > Create Bookmark

/bookmarks

Create a bookmark based on the VDB’s current state.

Share1

VDB’s Bookmark > Tags > Add Tag

/bookmarks/{bookmarkId}

(Specify Tag value(s))

Share the bookmark with other members who have sufficient permissions.

Branch

VDB >
Provision VDB

/vdbs/provision_by_X

Create a new VDB based on the selected source (Timestamp, Snapshot, Bookmark, etc)

 

1  Please note that DCT’s “Share” functionality is implemented through the role-based access model. Therefore, to share with your team members, you must have permission on that bookmark to update the Tags accordingly which in turn grants permission for other users. Tags can be assigned on creation or afterward. Optionally, Administrators can directly add available Bookmarks through the Access Group > Roles > Edit. 

 

Delphix Engine Self-Service
Data Control Tower Refresh Self-Service



Congratulations! At this point, your Application Teams are able to switch their Self-Service approach from the Continuous Data engine to DCT! We encourage you to leverage our various other resources to enable your teams. 

For advanced users, you can now leverage DCT APIs or integrations and expand DevOps practices easily across the organization.

 

Frequently Asked Questions

 

Question: As an Admin I can see the newly created bookmarks and provision VDBs, but my users cannot. Where are they?

Answer: Today, the migration scripts do not assign Tags because the Engine Self-Service model did not use tags in the same way. Therefore, you must manually assign Tags to each bookmark and VDB. This can be automated if desired using the DCT APIs. If Tagging is new, then I recommend you check out my other Access Model blog post.

 

Questions: Which version of the Delphix engine have these scripts been validated on? 

Answer: We built these scripts using v6.0.17.2 which is API v1.11.17. Source Before you migrate to DCT, I recommend you are at least on this engine version. 

 

Question: I am having trouble running your scripts. Will you help me? 

Answer: I will try my best to answer questions in the comments selection below and add new FAQs as needed. The support team is available to answer general DCT questions, but not specifically regarding these scripts. Delphix services team is always available.

Edit 5/17/2024: This blog previously stated that users can use a bookmark managed on Engine A and provision a VDB with that same bookmark on Engine B. This is not possible as of DCT v17. We hope to build this feature into a future release of DCT.

0 comments
74 views

Permalink