Refresh a Self Service Container from a Template Snapshot to reduce refresh time

By Diego Loureda posted 21 days ago

  
By default Self Service Refreshes on a container will refresh the container database or databases to the latest point in time available. We have seen that some of these refreshes may take some time. In the Oracle case, all archivelogs created after the backup have to be applied.


Our clients have been requested a way to refresh a container to the point in time where the snapshot happened in the template, but unfortunately the snapshots are not visible on the template timeline.


Here we present a workaround to be able to see snapshots in the template timeline on the Self Service screen.


On this example, the template has only one datasource.

The solution is very simple and based on the fact that bookmarks on templates are always shared, never private. So what I did is just create a post sync hook on the source (the template), that every time we do a snapsync, it will create a bookmark in the template that will be visible for all Self Service users. The hook is very simple and I used DxToolkit for it:



So every time a snapsync runs, this post sync hook will create a bookmark on the template timeline called snapsync + date:



With this simple hook, all Self Service users will have visibility to all the snapsyncs that are running on the source.

If you are not familiar with DxToolkit you must check it out. It's available here: https://github.com/delphix/dxtoolkit


If you don't want to use the DxToolkit script, as you will have to put it in all the targets, we can use a script like the one I pasted below. The good news about this approach is that all the logic to create the bookmark is stored in Delphix. This script has been tested in Red Hat Linux, may require minor changes for other Linux flavors or Unix.



On my example you can see delphix_admin is being used. If this is just not good enough in terms of security, we can just create a user with ownership on the dsource.



## This hook will create a bookmark in the template that contains this dsource

# Below Variables may change, make sure you have the proper values:
ENGINEURL=http://192.168.92.220
export URL
USR=delphix_admin
export USR
PASS=delphix
export PASS
BRANCHNAME=JS_BRANCH-1
export BRANCHNAME
SOURCEDATALAYOUT=JS_DATA_TEMPLATE-1
export SOURCEDATALAYOUT




#Below variables are fixed, no need to modify
BNAME=snapsyc-$(date +%Y-%m-%d-%H-%M-%S)
export BNAME




## Connect to the Delphix Engine
echo
echo
echo "Create session"
curl -s -X POST -k --data @- ${ENGINEURL}/resources/json/delphix/session \
    -c ~/cookies.txt -H "Content-Type: application/json" <
{
    "type": "APISession",
    "version": {
        "type": "APIVersion",
        "major": 1,
        "minor": 4,
        "micro": 0
    }
}
EOF
echo
echo
echo "Logon as:" ${USR}/${PASS}
curl -s -X POST -k --data @- ${ENGINEURL}/resources/json/delphix/login \
    -b ~/cookies.txt -H "Content-Type: application/json" <
{
    "type": "LoginRequest",
    "username": "${USR}",
    "password": "${PASS}"
}
EOF1




# Create Bookmark on Template
echo
echo
echo
echo
curl -s -X  POST -k --data @- ${ENGINEURL}/resources/json/delphix/jetstream/bookmark \
 -b ~/cookies.txt -H "Content-Type: application/json" <
{
    "type": "JSBookmarkCreateParameters",
    "bookmark": {
        "type": "JSBookmark",
        "name": "$BNAME",
        "branch": "$BRANCHNAME"
    },
    "timelinePointParameters": {
        "type": "JSTimelinePointLatestTimeInput",
        "sourceDataLayout": "$SOURCEDATALAYOUT"
    }
}
EOF1


2 comments
20 views

Comments

01-30-2019 07:31:26 PM

Hello all,

This is an updated bash script version that can be used as a post sync hook.

You only have to enter the Delphix Admin credentials and it will automatically get the DB name and will look for the SelfService templates on which this dSource is part of and will create a bookmark on the template timeline.

It will also set the expiration date 2 days from the time/date where it's taken.


Script:

#!/bin/bash
#================================================================================
# File:         create_bookmark_template.sh
# Type:         bash-shell script
# Date:         14-Jan 2019
# Author:       v2- Carlos Cuellar - Delphix Professional Services
#             v1- Diego Loureda  - Delphix Professional Services
# Ownership:    This script is owned and maintained by the user, not by Delphix
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Copyright (c) 2019 by Delphix. All rights reserved.
#
# Description:
#
# Script to be used as a PostSync hook on a dSource to create a new bookmark
# on selfservice/jetstream templates where dSource is the Source.
#
#================================================================================
#
# Please set the following variables to suit your purposes.
# Set this to the Delphix admin user name
DELPHIX_ADMIN=admin
# Set this to the password for the Delphix admin user
DELPHIX_PASS=delphix1
# Delphix Engine name
DelphixEngine='echo $SSH_CLIENT|awk '{ print $1}''
# Reference of dSource

if [ -z "$DELPHIX_DATABASE_NAME" ]
then
    export DSOURCE_NAME=${DELPHIX_PDB_NAME}
elif [ -z "$DELPHIX_PDB_NAME" ]
then
    export DSOURCE_NAME=${DELPHIX_DATABASE_NAME}
else
    export DSOURCE_NAME=${DELPHIX_PDB_NAME}
fi

#Below variables are fixed, no need to modify
BNAME=snapsyc-$(date +%Y-%m-%d-%H-%M-%S)
export BNAME
export DSOURCE_NAME_CONTAINER
export DSOURCE_NAME_TEMPLATE
export DSOURCE_NAME_BRANCH
export API_PATH

# Create API session
create_api_session () {
curl -s -X POST -k --data @- http://${DelphixEngine}/resources/json/delphix/session \
    -c ~/cookies.txt -H "Content-Type: application/json" <<EOF
{
    "type": "APISession",
    "version": {
        "type": "APIVersion",
        "major": 1,
        "minor": 9,
        "micro": 0
    }
}
EOF
echo
}


# Authenticate to the DelphixEngine
authenticate_api_session () {
curl -s -X POST -k --data @- http://${DelphixEngine}/resources/json/delphix/login \
    -b ~/cookies.txt -c ~/cookies.txt -H "Content-Type: application/json" <<EOF1
{
    "type": "LoginRequest",
    "username": "${DELPHIX_ADMIN}",
    "password": "${DELPHIX_PASS}"
}
EOF1
echo
}

# Get API information for Containers, Templates and Branches
get_api_info () {
api_action=$1
case $api_action in
                "source")
                        export ddsource='curl --silent -X GET -k http://${DelphixEngine}/resources/json/delphix/source -b ~/cookies.txt -H "Content-Type: application/json"'
                        IFS=',' read -a delphixdsourcearray <<< "${ddsource}"
                        ;;
                "sourceconfig")
                        export ddsource='curl --silent -X GET -k http://${DelphixEngine}/resources/json/delphix/sourceconfig -b ~/cookies.txt -H "Content-Type: application/json"'
                        IFS=',' read -a delphixdsourceconfigarray <<< "${ddsource}"
                        ;;
                "template")
                        export ddatasource='curl --silent -X GET -k http://${DelphixEngine}/resources/json/delphix/${API_PATH}/datasource -b ~/cookies.txt -H "Content-Type: application/json"'
                        IFS=',' read -a delphixtemplatearray <<< "${ddatasource}"
                        ;;
                "branch")
                        export dbranch='curl --silent -X GET -k http://${DelphixEngine}/resources/json/delphix/${API_PATH}/branch -b ~/cookies.txt -H "Content-Type: application/json"'
                        IFS=',' read -a delphixbrancharray <<< "${dbranch}"
                        ;;
                "version")
                        export dversion='curl --silent -X GET -k http://${DelphixEngine}/resources/json/delphix/system -b ~/cookies.txt -H "Content-Type: application/json"'
                        IFS=',' read -a delphixversionarray <<< "${dversion}"
                        ;;
esac

case $api_action in
        "source")
                counter=0
                for i in "${delphixdsourcearray[@]}"
                do
                     export js_col='echo $i|awk -F":" '{ print $1}''
                     export js_val='echo $i|awk -F\":\" '{ print $2}''
                     if [ "$js_col" == "\"config\"" ]||[ "$js_col" == "\"container\"" ]; then
                             export js_val_f4='echo $js_val|sed 's/"//g''
                             reference_dsource[$counter]="$js_val_f4"
                             counter='expr $counter + 1'
                     fi
                 done
                 ;;
        "sourceconfig")
                counter=0
                for i in "${delphixdsourceconfigarray[@]}"
                do
                     export js_col='echo $i|awk -F":" '{ print $1}''
                     export js_val='echo $i|awk -F\":\" '{ print $2}''
                     if [ "$js_col" == "\"name\"" ]||[ "$js_col" == "\"reference\"" ]; then
                             export js_val_f5='echo $js_val|sed 's/"//g''
                             reference_dsourceconfig[$counter]="$js_val_f5"
                             counter='expr $counter + 1'
                     fi
                 done
                 ;;
         "template")
                counter=0
                for i in "${delphixtemplatearray[@]}"
                do
                     export js_col='echo $i|awk -F":" '{ print $1}''
                     export js_val='echo $i|awk -F\":\" '{ print $2}''
                     if [ "$js_col" == "\"name\"" ] ||[ "$js_col" == "\"databaseName\"" ] ||[ "$js_col" == "\"dataLayout\"" ] ||[ "$js_col" == "\"container\"" ]; then
                             export js_val_f2='echo $js_val|sed 's/"//g''
                             reference_template[$counter]="$js_val_f2"
                             counter='expr $counter + 1'
                     fi
                 done
                 ;;
         "branch")
                counter=0
                for i in "${delphixbrancharray[@]}"
                do
                     export js_col='echo $i|awk -F":" '{ print $1}''
                     export js_val='echo $i|awk -F\":\" '{ print $2}''
                     if [ "$js_col" == "\"name\"" ]||[ "$js_col" == "\"reference\"" ]||[ "$js_col" == "\"dataLayout\"" ]; then
                             export js_val_f3='echo $js_val|sed 's/"//g''
                             reference_branch[$counter]="$js_val_f3"
                             counter='expr $counter + 1'
                     fi
                done
                 ;;
        "version")
        counter=0
                for i in "${delphixversionarray[@]}"
                do
                     export js_col='echo $i|awk -F":" '{ print $1}''
                     export js_val='echo $i|awk -F\":\" '{ print $2}''
                     if [ "$js_col" == "\"buildTitle\"" ]; then
                             export js_val_f='echo $js_val|sed 's/"//g''
                             reference_version[$counter]="$js_val_f"
                     fi
                 done
        if [[ ${reference_version[${counter}]} =~ .*Engine.* ]]
        then
            export DE_VERSION='echo ${reference_version[${counter}]}|awk '{ print $3}''
        else
            export DE_VERSION='echo ${reference_version[${counter}]}|awk '{ print $4}''
        fi
         echo "Delphix Engine version is "  $DE_VERSION
         DE_V1='echo $DE_VERSION|awk -F. '{ print $1}''
         DE_V2='echo $DE_VERSION|awk -F. '{ print $2}''
         if [ $DE_V1 -gt 4 ]; then
                if [ $DE_V2 -gt 2 ]; then
                    export API_PATH=selfservice
                else
                    export API_PATH=jetstream
                fi
         else
            export API_PATH=jetstream
         fi
                 ;;
esac
}

#Get Config name to match DB name to dSource name in Delphix engine
get_config_name () {
counter=0
for j in "${reference_dsourceconfig[@]}"
do
        if [ "$j" == "$DSOURCE_NAME" ]; then
                pointer='expr $counter - 1'
                export DSOURCE_CONFIG_CONTAINER=${reference_dsourceconfig[${pointer}]}
                echo "Source Config name is : " $DSOURCE_CONFIG_CONTAINER
        fi
        counter='expr $counter + 1'
done
}

#Get the Container name to create bookmarks
get_container_name () {
counter=0
for j in "${reference_dsource[@]}"
do
        if [ "$j" == "$DSOURCE_CONFIG_CONTAINER" ]; then
                pointer='expr $counter - 1'
                DSOURCE_NAME_CONTAINER=${reference_dsource[${pointer}]}
                echo "Container name is : " $DSOURCE_NAME_CONTAINER
        fi
    counter='expr $counter + 1'
done
}


#Get Template reference and with it create bookmarks as needed
get_temp_branch_create_bookmark () {
export BMDT=$(date '+%Y-%m-%dT%H:%M:%S.999Z' --date="+2 days");
for j in "${reference_template[@]}"
do
        if [ "$j" == "$DSOURCE_NAME_CONTAINER" ] ; then
                DSOURCE_NAME_TEMPLATE=${k}
                echo "Template name is : " $DSOURCE_NAME_TEMPLATE
len=${#reference_branch[@]}
for (( i=0; i<$len; i++ ))
do
k=${reference_branch[$i]}
        if [ "$k" == "$DSOURCE_NAME_TEMPLATE" ] ; then
                pointer='expr $i - 2'
                DSOURCE_NAME_BRANCH=${reference_branch[$pointer]}
                echo "Branch name is : " $DSOURCE_NAME_BRANCH

11-05-2018 08:48:38 PM

This Powershell script can be used as a post hook to create bookmark after every snapshot.

############################### Delphix Masking and Snapshot API Calls ################################
# File: ss_bookmark_template.ps1
# Type: Powershell script
# Author: Delphix Professional Services (Created by - Jatinder Luthra)
# Date: 17-Oct-2018
#
# Copyright and license:
#
#       Licensed under the Apache License, Version 2.0 (the "License"); you may
#       not use this file except in compliance with the License.
#
#       You may obtain a copy of the License at
#     
#               http://www.apache.org/licenses/LICENSE-2.0
#
#       Unless required by applicable law or agreed to in writing, software
#       distributed under the License is distributed on an "AS IS" basis,
#       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#
#       See the License for the specific language governing permissions and
#       limitations under the License.
#     
#       Copyright (c) 2018 by Delphix.  All rights reserved.
#
# Description:
#
# Powershell script to create bookmark on Self Service Template
#   
#   Parent Script: ss_bookmark_template.ps1
#   
#   Notes: This script can be called as a hook which will create a bookmark in the template that contains this dSource                           
#________________________________________________________________________________________
######################################################################################################
#########################################################
#                   DELPHIX CORP                        #
#########################################################
########################################################################################
#########################       SET THE SCRIPT PARAMETERS        #######################
########################################################################################
$SS_TEMPLATE=$args[0]
If ([string]::IsNullOrEmpty($SS_TEMPLATE)){
echo 'Please pass the Self-Service Template Name'
exit 1
}

$VIRTUALIZATION_ENGINE="http://172.16.98.25/resources/json/delphix";
$VIRT_USERNAME="delphix_admin"
$VIRT_PASSWORD="delphix"
#$SS_TEMPLATE = "PerformanceMasking"
$BOOKMARK_TIME=(Get-Date -Format yyyy-MM-dd-HH-mm-ss).ToString()
$SS_BOOKMARK = 'snapsync-'+$BOOKMARK_TIME
#####################################################################################
################         NO CHANGES REQUIRED BELOW THIS POINT        ################
#####################################################################################
#####################################################################################
####################       FUNCTION TO PARSE JSON OUTPUT         ####################
#####################################################################################
function ConvertFrom-Json20([object] $item){ 
    add-type -assembly system.web.extensions
    $ps_js=new-object system.web.script.serialization.javascriptSerializer
    #The comma operator is the array construction operator in PowerShell
    return ,$ps_js.DeserializeObject($item)
}
#########################################################
#
######## Creating Session to Virtualization Engine #######
$cookies = "cookies.txt"
$json = @"
{
    "type": "APISession",
    "version": {
        "type": "APIVersion",
        "major": 1,
        "minor": 7,
        "micro": 0
    }
}
"@
write-output $json | Out-File "session.json" -encoding utf8
$Session = $(C:\Users\dtully\Downloads\curl-7.58.0\src\curl.exe -c $cookies -sX POST --header 'Content-Type: application/json' -d "@session.json" -k "$VIRTUALIZATION_ENGINE/session")
$getsesstat = ConvertFrom-Json20 $Session
$sesstatus=$getsesstat.status
echo "##### Session Status ##### - $sesstatus"
if ("${sesstatus}" -ne "OK") {
   echo "Session to DELPHIX Virtualization Engine failed with ${sesstatus}"
   exit 1
}

################# Login to Virtualization Engine #################### 
echo " "
echo "########### Login into Virtualization Engine ############"
echo " "
$json = @"
{
    "type": "LoginRequest",
    "username": "${VIRT_USERNAME}",
    "password": "${VIRT_PASSWORD}"
}
"@
write-output $json | Out-File "login.json" -encoding utf8
$Login = $(C:\Users\dtully\Downloads\curl-7.58.0\src\curl.exe -b $cookies -sX POST --header 'Content-Type: application/json' -d "@login.json" -k "$VIRTUALIZATION_ENGINE/login")
$getloginstat = ConvertFrom-Json20 $Login
$loginstatus=$getloginstat.status
echo "##### Login Status ##### - $loginstatus"
if ("${loginstatus}" -ne "OK") {
   echo "Login to DELPHIX Virtualization Engine failed with ${loginstatus}"
   exit 1
}
#####################################################################################
################        GET Self Service TEMPLATE REFERENCE        ##################
#####################################################################################

$fetch_templates = $(C:\Users\dtully\Downloads\curl-7.58.0\src\curl.exe -b $cookies -sX GET --header 'Content-Type: application/json' -k "$VIRTUALIZATION_ENGINE/jetstream/template")
# write-output "Databases API Results: ${dbsList}"
$get_temp_list = ConvertFrom-Json20 $fetch_templates
$temp_status=$get_temp_list.status
if ("${temp_status}" -ne "OK") {
   echo "Fetch Jet Stream Templates from DELPHIX Virtualization Engine failed with ${temp_status}"
   exit 1
}
$templist = $get_temp_list.result
#### Extract Template #####
$extractemplate = $templist | where { $_.name -eq "${SS_TEMPLATE}" -and $_.type -eq "JSDataTemplate"} | Select-Object
$TEMPLATE_REFERENCE=$extractemplate.reference
echo "#### TEMPLATE REFERENCE FOR ${JS_TEMPLATE} IS #### " $TEMPLATE_REFERENCE
#####################################################################################
################        GET Self Service BRANCH REFERENCE        ##################
#####################################################################################
$fetch_branches = $(C:\Users\dtully\Downloads\curl-7.58.0\src\curl.exe -b $cookies -sX GET --header 'Content-Type: application/json' -k "$VIRTUALIZATION_ENGINE/jetstream/branch")
$get_br_list = ConvertFrom-Json20 $fetch_branches
$branch_status=$get_br_list.status
if ("${branch_status}" -ne "OK") {
   echo "Fetch Jet Stream Branches from DELPHIX Virtualization Engine failed with ${branch_status}"
   exit 1
}
$branchlist= $get_br_list.result
#### Extract Branch #####
$extractbranch = $branchlist | where { $_.name -eq 'master' -and $_.dataLayout -eq ${TEMPLATE_REFERENCE}} | Select-Object
$BRANCH_REFERENCE=$extractbranch.reference
echo "#### BRANCH REFERENCE FOR ${SS_BRANCH} IS #### " $BRANCH_REFERENCE
#####################################################################################
################         Bookmark Template and Track Progress        ################
#####################################################################################
echo " "
echo "########### Creating Bookmark on Self Service Template - ${SS_TEMPLATE} ############"
echo " "
$bookmark_param = @"
{
    "type": "JSBookmarkCreateParameters",
    "bookmark": {
        "type": "JSBookmark",
        "name": "$SS_BOOKMARK",
        "branch": "$BRANCH_REFERENCE"
    },
    "timelinePointParameters": {
        "type": "JSTimelinePointLatestTimeInput",
        "sourceDataLayout": "$TEMPLATE_REFERENCE"
    }
}
"@
write-output $bookmark_param | Out-File "bookmark.json" -encoding utf8
$bookmark_template = $(C:\Users\dtully\Downloads\curl-7.58.0\src\curl.exe -b $cookies -sX POST --header 'Content-Type: application/json' -d "@bookmark.json" -k "$VIRTUALIZATION_ENGINE/jetstream/bookmark")
$getbmstat = ConvertFrom-Json20 $bookmark_template
$bm_status=$getbmstat.status
echo "##### GET Bookmark Status ##### - $bm_status"
if ("${bm_status}" -ne "OK") {
   echo "Bookmark of Template "${SS_TEMPLATE}" failed with ${bm_status}"
   exit 1
}
$bm_jobID = $getbmstat.job
echo " "
echo "###### Bookmark Creation JOBID ##### - $bm_jobID "
echo " "
################ Tracking Bookmark Creation Progress ################

echo "############### Tracking Bookmark Status of Template - $SS_TEMPLATE ####################"
$sleepTimer=1
DO {
   Start-Sleep $sleepTimer
$bm_job = $(C:\Users\dtully\Downloads\curl-7.58.0\src\curl.exe -b $cookies -sX GET --header 'Content-Type: application/json' -k "$VIRTUALIZATION_ENGINE/job/$bm_jobID")
$getjobstat = ConvertFrom-Json20 $bm_job
$jobState=$getjobstat.result.jobState
echo "####### Current Status of Bookmark ${SS_BOOKMARK} Creation ####### - $jobState"
} WHILE ($jobState -eq'RUNNING')
echo " "
echo "####### Self Service Bookmark, ${SS_BOOKMARK} created on Template, ${SS_TEMPLATE} ####### - $jobState"
exit 0

################ END OF SCRIPT ###############

~Thanks
Jatinder Luthra