Delphix Toolkits (dxToolkit and dxmToolkit)

 View Only
Expand all | Collapse all

Issue with dx_provision_vdb

  • 1.  Issue with dx_provision_vdb

    Posted 07-09-2020 11:45:00 PM

    We recently upgraded our delphix engine to 5.3.8.1. After the upgrade I ran one of our provisioning jobs and it failed. Then I downloaded the latest dxtoolkit but it still keeps failing.

    Old command
    ----------------
    dx_provision_vdb -type oracle -group "<groupname>" -sourcename <srcname> -targetname <tgtname> -dbname <tgtname> -srcgroup "<groupname>" -timestamp LATEST_SNAPSHOT -environment <envname> -template <template> -envinst <oraclehome> -instname <instname> -uniqname <name> -mntpoint <mnt> -archivelog no

    When using the above command, got this error -
    command output:
    Problem with starting job
    Error: source->allowAutoVDBRestartOnHostReboot->The field "{0}" is required.
    Job wasn't defined. If there is no error on the screen, try with -debug flag to find a root cause.
    ===============================
    command exit code: 1

    New command
    ------------------
    dx_provision_vdb -type oracle -group "<groupname>" -sourcename <srcname> -targetname <tgtname> -dbname <tgtname> -srcgroup "<groupname>" -timestamp LATEST_SNAPSHOT -environment <envname> -template <template> -envinst <oraclehome> -instname <instname> -uniqname <name> -mntpoint <mnt> -archivelog no -autostart yes -allowAutoVDBRestartOnHostReboot true

    With the new command, I get the following error

    Unknown option: allowAutoVDBRestartOnHostReboot



    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------


  • 2.  RE: Issue with dx_provision_vdb

    Posted 07-10-2020 11:08:00 AM
    Hi,

    What version of Dxtoolkit are you using ? 
    You may need to upgrade it as well, if you have a new version of Delphix Engine.

    regards,
    Marcin

    ------------------------------
    Marcin Przepiorowski
    Technical Principal
    Delphix
    ------------------------------



  • 3.  RE: Issue with dx_provision_vdb

    Posted 07-10-2020 11:44:00 AM
    dxtoolkit2-2.3.8.1-redhat7.tar

    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------



  • 4.  RE: Issue with dx_provision_vdb

    Posted 07-10-2020 12:23:00 PM
    Hi,

    Could you please upgrade to the latest version and let me know if this is working OK ?

    Here is a link:
    https://github.com/delphix/dxtoolkit/releases/tag/v2.4.7

    regards,
    Marcin

    ------------------------------
    Marcin Przepiorowski
    Technical Principal
    Delphix
    ------------------------------



  • 5.  RE: Issue with dx_provision_vdb

    Posted 07-10-2020 04:28:00 PM
    Downloaded version 2.4.7 and got the same error
    Error: source->allowAutoVDBRestartOnHostReboot->The field "{0}" is required.
    Job wasn't defined. If there is no error on the screen, try with -debug flag to find a root cause.


    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------



  • 6.  RE: Issue with dx_provision_vdb

    Posted 07-13-2020 03:14:00 AM
    Hello,

    Could you please run this command with -debug 3 and zip a debug_xxxx directory and attach here ? 
    xxxx is a name of your Delphix Engine and this subdirectory will be created your current working directory.

    If you want to define auto start option in the provisioning command you can use this option and choose one yes or no

    dx_provision_db -autostart [yes|no] ........


    regards,
    Marcin

    ------------------------------
    Marcin Przepiorowski
    Technical Principal
    Delphix
    ------------------------------



  • 7.  RE: Issue with dx_provision_vdb

    Posted 07-13-2020 07:01:00 AM
    I have the zip file. how do I attach here ?

    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------



  • 8.  RE: Issue with dx_provision_vdb

    Posted 07-13-2020 09:34:00 AM
    Hi Srividya,
    When you reply, there are is a button at the bottom right of the screen which says "Upload file". Click that button and you will be guided through uploading your zip file. Please see the image below.
    If you have any issue doing this, please let me know. 

    Thank you,
    Michael


    ------------------------------
    Michael Torok
    Director of Knowledge and Community Management
    Delphix
    ------------------------------



  • 9.  RE: Issue with dx_provision_vdb

    Posted 07-13-2020 09:56:00 AM
    Yes I tried to click on the Upload File earlier but no window opened up. So wanted to see if I was missing something. Is there another way I can send the zip file ?

    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------



  • 10.  RE: Issue with dx_provision_vdb

    Posted 07-13-2020 10:06:00 AM
    Please feel free to send it to me directly and I will attach it. Michael.torok@delphix.com

    The upload file button pops up a new window, so it could be stopped by popup blockers.

    Thanks,
    Michael

    ------------------------------
    Michael Torok
    Director of Knowledge and Community Management
    Delphix
    ------------------------------



  • 11.  RE: Issue with dx_provision_vdb

    Posted 07-13-2020 12:49:00 PM
      |   view attached
    I emailed it directly but after seeing your message about the popup I tried from another browser and was able to upload here as well. Thanks for the help

    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------

    Attachment(s)

    zip
    debug_filidelph1vapp.zip   192 B 1 version


  • 12.  RE: Issue with dx_provision_vdb

    Posted 07-13-2020 12:51:00 PM
    Perfect!

    ------------------------------
    Michael Torok
    Director of Knowledge and Community Management
    Delphix
    ------------------------------



  • 13.  RE: Issue with dx_provision_vdb

    Posted 07-14-2020 07:37:00 AM
    Hi - any updates ?

    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------



  • 14.  RE: Issue with dx_provision_vdb

    Posted 07-14-2020 08:54:00 AM
    Hi Srividya,

    Your zip file contain an empty directory.
    Could you please check if there are any files generated ?

    regards,
    Marcin

    ------------------------------
    Marcin Przepiorowski
    Technical Principal
    Delphix
    ------------------------------



  • 15.  RE: Issue with dx_provision_vdb

    Posted 07-14-2020 11:27:00 AM
      |   view attached
    Sorry about that. Reattaching

    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------

    Attachment(s)

    zip
    debug_filidelph1vapp.zip   303 KB 1 version


  • 16.  RE: Issue with dx_provision_vdb

    Posted 07-15-2020 03:22:00 AM
    Hello,

    I can see that job has been started, so new Dxtoolkit is working OK 
    but now you have a problem with Delphix job itself and this is an Oracle error

    ORA-00603: ORACLE server session terminated by fatal error
    ORA-01092: ORACLE instance terminated. Disconnection forced
    ORA-00704: bootstrap process failure
    ORA-00604: error occurred at recursive SQL level 1
    ORA-00904: \"ACDRROWTSINTCOL#\": invalid identifier
    Process ID: 23558\nSession ID: 1144 Serial number: 46852
    ORA-24324: service handle not initialized
    ORA-01041: internal error. hostdef extension doesn't exist\nORACLE_RECOVERY_ERROR\n",

    Are you provisioning this VDB to a new server ? Do you have a name of this server in /etc/hosts 
    and name is not pointing to 127.0.0.1 ? 

    regards,
    Marcin

    ------------------------------
    Marcin Przepiorowski
    Technical Principal
    Delphix
    ------------------------------



  • 17.  RE: Issue with dx_provision_vdb

    Posted 07-15-2020 03:58:00 PM
    We have other VDBs running on this server. I have this on /etc/hosts file
    127.0.0.1 localhost.localdomain localhost

    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------



  • 18.  RE: Issue with dx_provision_vdb

    Posted 07-16-2020 03:04:00 AM
    Hi,

    It's is outside of scope of Dxtoolkit support but I would suggest to add server name and IP address to /etc/hosts.
    Are you able to create VDB on that server using GUI ?

    regards,
    Marcin

    ------------------------------
    Marcin Przepiorowski
    Technical Principal
    Delphix
    ------------------------------



  • 19.  RE: Issue with dx_provision_vdb

    Posted 07-16-2020 05:31:00 AM
    Yes I am able to create VDB on that server using GUI and script. For this database I can create using GUI. Only with script it is not working. Also I got this error if i see on the last page of the debug
    -Entering Engine::getEngineName
    Problem with starting job
    Error: source->allowAutoVDBRestartOnHostReboot->The field "{0}" is required.
    -Can't submit job for operation resources/json/delphix/database/provision
    error $VAR1 = {
      'source' => {
      'allowAutoVDBRestartOnHostReboot' => {
      'action' => undef,
      'id' => 'exception.validation.field.missing',
      'details' => 'The field "{0}" is required.',
      'commandOutput' => undef,
      'diagnoses' => undef
      }
      }
      };
    $VAR2 = 1;
     
    -Check your input parameters and try again.


    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------



  • 20.  RE: Issue with dx_provision_vdb

    Posted 07-16-2020 05:43:00 AM
    Hi,

    In the zip file you send me I can see that:

    {"timeflowPointParameters":{"type":"TimeflowPointSemantic","container":"ORACLE_DB_CONTAINER-973","location":"LATEST_SNAPSHOT"},"container":{"type":"OracleDatabaseContainer","name":"vtrinlwc","group":"GROUP-42","sourcingPolicy":{"logsyncMode":"UNDEFINED","logsyncInterval":5,"logsyncEnabled":false,"type":"OracleSourcingPolicy","loadFromBackup":false}},"source":{"archivelogMode":false,"type":"OracleVirtualSource","mountBase":"/delphix/provision","nodeListenerList":[],"allowAutoVDBRestartOnHostReboot":false,"operations":{"configureClone":[],"type":"VirtualSourceOperations","preRefresh":[],"postRefresh":[]},"configTemplate":"DATABASE_TEMPLATE-12"},"sourceConfig":{"services":[],"linkingEnabled":false,"repository":"ORACLE_INSTALL-183","databaseName":"vtrinlwc","instance":{"type":"OracleInstance","instanceName":"vtrinlwc","instanceNumber":1},"type":"OracleSIConfig","uniqueName":"vtrinlwc"},"type":"OracleProvisionParameters"}

    and response from Delphix Engine

    {
    "action" : "ACTION-179057",
    "type" : "OKResult",
    "job" : "JOB-96064",
    "result" : "ORACLE_DB_CONTAINER-1215",
    "status" : "OK"
    }

    so job was started and it failed with this error

    {
    "percentComplete" : 36,
    "timestamp" : "2020-07-13T11:48:07.764Z",
    "messageAction" : "Review the Oracle alert log for more details.",
    "diagnoses" : [],
    "messageCode" : "exception.oracle.targetscripts.db.recovery",
    "messageCommandOutput" : "command: umask 027; . $DB_SCRIPT_DIR/setup-oraenv.sh; $DLPX_SHELL $DB_SCRIPT_DIR/doRecoveryWithBackupCF.sh $DB_SCRIPT_DIR \"$LOGON_STR\"\noutput:END_OF_SETUP\nEND_OF_SETUP\nMOUNT_DATABASE_COMPLETED\nLOG_MODE=NOARCHIVELOG\n\nPL/SQL procedure successfully completed.\nLASTEST_CHECKPOINT_CHANGE_NUMBER=\n alter database open\n*\nERROR at line 1:\nORA-00603: ORACLE server session terminated by fatal error\nORA-01092: ORACLE instance terminated. Disconnection forced\nORA-00704: bootstrap process failure\nORA-00604: error occurred at recursive SQL level 1\nORA-00904: \"ACDRROWTSINTCOL#\": invalid identifier\nProcess ID: 23558\nSession ID: 1144 Serial number: 46852\n\n\nORA-24324: service handle not initialized\nORA-01041: internal error. hostdef extension doesn't exist\nORACLE_RECOVERY_ERROR\n",
    "eventType" : "ERROR",
    "messageDetails" : "Failed to apply logs in database recovery.",
    "type" : "JobEvent",
    "state" : null
    }


    Could you please confirm that ZIP file you sent me is an output from your run ? It is for this VDB - vtrinlwc

    regards,
    Marcin 



    ------------------------------
    Marcin Przepiorowski
    Technical Principal
    Delphix
    ------------------------------



  • 21.  RE: Issue with dx_provision_vdb

    Posted 07-16-2020 06:13:00 AM
    yes vtrinlwc is the vdb that I have been trying to create. so this is what happened. maybe thats the difference. we usually have jobs creating vdbs running from jenkins -> udeploy -> delphix for automation purposes. When I ran from this setup I saw the error I mentioned above but I couldnt find the debug logs anywhere. So then I ended up running from the server itself directly calling the dx_provision_vdb script which is when I did see the debug files generated and thats the one I zipped up and sent you. So should we just proceed along the lines that you are mentioning ?

    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------



  • 22.  RE: Issue with dx_provision_vdb

    Posted 07-16-2020 06:37:00 AM
    Also just realized only the first line from /etc/hosts was attached before in my response. These are all the lines from /etc/hosts

    # created during provisioning
    #
    127.0.0.1 localhost.localdomain localhost
    10.211.38.130 vlfilid01oma2a.fmr.com vlfilid01oma2a loghost
    10.253.219.58 vlfilid01oma2a-priv.fmr.com vlfilid01oma2a-priv
    10.253.219.59 vlfilid02oma2b-priv.fmr.com vlfilid02oma2b-priv

    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------



  • 23.  RE: Issue with dx_provision_vdb

    Posted 07-16-2020 06:44:00 AM
    Hi,

    So maybe the problem is with different versions of Dxtoolkit being used manually 
    and via automation. This error Error: source->allowAutoVDBRestartOnHostReboot from automation should be fixed by new version of dxtoolkit
    and I can see from debug logs that it moved forward.

    Could you please double check that same Dxtoolkit is used via your automation scripts ? 
    I would also propose to setup a call with your Delphix contacts to speed up resolution of this issue.

    regards,
    Marcin

    ------------------------------
    Marcin Przepiorowski
    Technical Principal
    Delphix
    ------------------------------



  • 24.  RE: Issue with dx_provision_vdb

    Posted 07-16-2020 07:02:00 AM
    Ok I will check that. Not sure what you mean by Delphix contacts. I opened a case with Delphix support but they said I have to ask the question here because it is to do with dxtoolkit

    ------------------------------
    Srividya Srinivasan
    Community Member
    Delphix Community Members
    ------------------------------



  • 25.  RE: Issue with dx_provision_vdb

    Posted 07-16-2020 08:47:00 AM
    Hi,

    Thanks - let me know output from your checks.
    By Delphix contact I mean a Customer Success Director or someone from Delphix Professional team.
    Let me check internally and someone will contact you.

    regards,
    Marcin

    ------------------------------
    Marcin Przepiorowski
    Technical Principal
    Delphix
    ------------------------------