API/Python/Open-Source Solutions

Expand all | Collapse all

how to expire vdbs

Jump to Best Answer
  • 1.  how to expire vdbs

    Posted 03-05-2018 06:11:00 PM
    We are moving into VM servers and planning to virtualize all our databases. In this case, we need to be able to provision vdbs but also expire them after a certain time because we dont want them to be tracked forever. What is the best way to achieve this ? Is there a way we can set the vdb retention when it is getting provisioned ?


  • 2.  RE: how to expire vdbs

    Posted 03-05-2018 07:42:00 PM
    Hi Srividya,
    I'd be interested to hear a little more detail on your use case, however I'm not aware of a way of setting an expiry/delete time for a VDB.

    We do have a REST API which will allow you to automate the VDB deletion after a specified period.  There is also an open source Perl wrapper to the API here (remove VDB Perl script here).

    Web Service and CLI guides are here">https://docs.delphix.com/docs/reference">here.

    Regards,
    Gary


  • 3.  RE: how to expire vdbs

    Posted 03-06-2018 01:12:00 PM

    The use case is this – lets say we have a demand to provision a vdb dbA and the application team tells us they need this database for 15 days. We want to be able to set that timeframe during creation of the vdb because we will have a number of these databases and don’t want to be housekeeping them. does that help ?



  • 4.  RE: how to expire vdbs
    Best Answer

    Posted 03-06-2018 01:27:00 PM
    Hi Srividya, One question in return to you. What happens if your dev didn’t end up his project in the window of 15 days and need more time. Regards, Mouhssine


  • 5.  RE: how to expire vdbs

    Posted 03-06-2018 01:44:00 PM
    Thanks for the clarification, I understand the use case and also Mouhssine's response which hints at some of the issues that might be faced.
    The way forward is to use the API.
    In fact you're probably better to disable VDBs in the first instance - send out a warning and then subsequently delete the VDB after a further time period.  Or just run a periodic job that whacks aged VDBs after a prescribed period.
    This is probably a good enhancement request for a UI for managing VDBs.
    It would be good to hear from the community how others have managed this scenario using the API or some other method.  Perhaps there's scope for a dxToolkit executable where you enter a number of days and the job disables or deletes all VDBs older than the entry.  Rather a dangerous utility though.
    Regards,
    Gary


  • 6.  RE: how to expire vdbs

    Posted 03-06-2018 01:56:00 PM
    Mouhssine - I was thinking if there is a way to specify a retention period then if any of the VDBs need to be retained longer (which should be a minimal number) then we can extend the retention period. I am not sure if that is possible but that is how I am envisioning.


  • 7.  RE: how to expire vdbs

    Posted 03-06-2018 02:01:00 PM
    Gary - We don't mind disabling the VDBs and then deleting them. Are you suggesting we do this manually ? If so, the issue is we will have a huge number (more than 50) of virtual databases and each of them would be from different time periods. I thought it will be cumbersome to keep track of each of them manually.


  • 8.  RE: how to expire vdbs
    Best Answer

    Posted 03-06-2018 02:13:00 PM
    No I was figuring that you would need to write a small script, perhaps using the dxtoolkit to expire aged VDBs, perhaps taking a parameter for a list of VDBs you might want to keep. 


  • 9.  RE: how to expire vdbs

    Posted 03-06-2018 02:18:00 PM
    Hi Srividya, Gary is right. I think you have run a job on scheduled time (every morning) that compares the creation time of vdb with the actual timestamp if it exceeds 15 days it will disable/drop vdb. Regards, Mouhssine


  • 10.  RE: how to expire vdbs

    Posted 03-06-2018 06:40:00 PM
    Ok that makes sense. I will have to dig in more. to see how to get details on the vdb. will reach out if I need more info. thanks a lot


  • 11.  RE: how to expire vdbs

    Posted 03-06-2018 07:59:00 PM
    I was referencing the CLI cookbook and I am able to connect to the delphix engine and run manual commands to list the vdb creationtime but how do we automate and get that value in a script. Are there examples of something similar ?


  • 12.  RE: how to expire vdbs

    Posted 03-06-2018 09:17:00 PM
    Srividya,
    You could use the dxtoolkit to create a script. [dxtoolkit is here]

    You just need to download the executable version of the files (there's a zip file for Windows and one for Linux/Mac).

    One you download and unzip the executables you then need to update the dxtools.config file with the details of your Delphix Engine.

    After that it's a matter of checking the documentation and executing a command with appropriate parameters.  Or you can download the source and modify to your needs.

    For instance I can use the following command:

    ./dx_ctl_db -engine LandsharkEngine -name vdb21 -type vdb -action disable

    to disable a VDB called "vdb21" on my Delphix Engine called "LandsharkEngine"

    You can use the executable "dx_remove_db" to delete the VDB.  See description below.

    ./dx_remove_db -help

    SYNOPSIS

         dx_remove_db.pl [ -engine|d <delphix identifier> | -all ] < -group group_name | -name db_name | -host host_name | -type dsource|vdb > 

         [-force] [-skip] [ --help|? ] [ -debug ] [-parallel p]

    DESCRIPTION

        Delete all database(s) selected by filter on selected engine(s)

    ARGUMENTS

      Delphix Engine selection - if not specified a default host(s) from dxtools.conf will be used.

        -engine|d Specify Delphix Engine name from dxtools.conf file

        -all Display databases on all Delphix appliance

      Filters

        -group Group Name

        -name Database Name

        -host Host Name

        -type Type (dsource|vdb)

    OPTIONS

        -skip Skip confirmation

        -force Force mode

        -help Print this screen

        -debug Turn on debugging

    Regards,
    Gary


  • 13.  RE: how to expire vdbs

    Posted 03-06-2018 09:22:00 PM

    Thanks Gary. I get that part. But I have to first check if the vdb can be disabled or deleted based on the timestamp. For eg., I see this on my vdb

    delphengine database 'vtestdb'> get creationTime
        2018-03-05T18:28:59.722Z

    Is there a way to get this creationtime through a script so I can compare and see if it is eligible to be deleted




  • 14.  RE: how to expire vdbs

    Posted 03-07-2018 02:05:00 AM
    Okay, Srividya
    You can create a script something like:

    ssh delphix_admin@LandsharkEngine "cd database; select vdb21; get creationTime"

    Or this will get you a list of creation times (unless your DB names have spaces):

    DELPHIX_ENGINE=LandsharkEngine
    SSH_CMD="ssh delphix_admin@${DELPHIX_ENGINE}"
    env_array=('${SSH_CMD}  "cd database; list display=name" | grep -v NAME')
    for i in "${env_array[@]}"
    do
      ${SSH_CMD} "cd database; select $i; get creationTime; back"
    echo $i
    done

    In my system this script did not work because my dataset names contain spaces.  In any case you may want  to use the REST API.

    To get around the fact that you will be prompted for a password when you use ssh you'll need to create a key pair on your machine.  To do that on a Mac you simply type the following: 

    ssh-keygen -t rsa

    This puts a public and private key in the folder ~/.ssh
    There may already be one there, in which case use it.
    Open the public key (usually named id_rsa.pub) with an editor and copy the text.
    You then have to place that public key into the Delphix Engine against the user that you want to log in as from your client machine - let's say delphix_admin.  This is done by logging onto the engine as delphix_admin and using the following commands:

    delphix> user current
    delphix user "delphix_admin"> update
    delphix user "delphix_admin" update *> set publicKey
    Enter publicKey: AAAAB3NzaC1yc2EYYYYDAQABAAABAQCt6Gkq3sxtv//OzIl1tPTbHNblo7y+rLMwcPOkpra6iwmcIL+7pVbSSWLGYOXuW/keLOCey4r3LqsTdpwL2F2vvngafCIP9yEhqfzsZ4zhTOaDioBnZ1jc/gQiM9lk8bEO6qpg8Snk/OZ71BxJK1tszuleOBd6uHfD36P6d3DjjvAOGi0C9A/+QnWNW2+SRUzfJZryijuyfYfe46FziVM0HwAPxJiB8h8n8DBN8d1GOuvBljPNYSEu5wpphiK/QvW7c8d0LW2jonbJ3X/FJaOA4d1xG8Fz43SJHCMq9by5Czp6jPdXtVEmQRoCIY7SBAr+2/2LNxDtw2YzR srividya@Srividyas-MacBook-Pro.local

    Some documentation around the CLI:
    CLI Automation
    Generating SSH Key

    Regards,
    Gary



  • 15.  RE: how to expire vdbs

    Posted 03-07-2018 02:10:00 AM
    We do have a feature request for functionality similar to this, however it is not yet on the roadmap at this point in time.  I have added your request to the list - if you are working on behalf of a specific customer then I can add that customer name also.
    Regards,
    Gary