Delphix Products

  • 1.  Delphix Provisioning is stuck at Recovering Oracle Database (40% for 24hours and counting)

    Posted 08-01-2015 02:40:00 AM
    What do I need to check to know that is still is ongoing, or if it hanged/encountered an error. There are no messages in the Delphix Engine GUI.


  • 2.  RE: Delphix Provisioning is stuck at Recovering Oracle Database (40% for 24hours and counting)
    Best Answer

    Posted 08-01-2015 10:35:00 PM
    Hi Mark - It's difficult to say what's happening on your Engine without logs... at this point in the process (40-50%) it should be mounting the filesystems from Delphix, then starting the Oracle instance for the VDB, before altering settings like disable Flashback, etc, in preparation for rename of datafiles.  All of these preparatory steps are usually not that time consuming; any real work would be expected in the processing of datafiles, not mounting filesystems.

    Out of curiosity, a few questions: 
    - What version of Delphix are you using? 
    - Is this the first provision job after some kind of environmental event, Engine or host reboot, etc?

    If the process appears to be stuck, you can check the mount state from the target environment, and look for the Delphix Engine IP address; here's an example below:

    delphix_os@rhel62:/home/delphix_os-> mount
    /dev/mapper/vg_rhel62-lv_root on / type ext4 (rw)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
    /dev/sda1 on /boot type ext4 (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
    10.43.7.20:/domain0/group-2/oracle_db_container-16/oracle_timeflow-16 on /mnt/provision/Vdb1_14F type nfs (rw,nosuid,bg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,tcp,noacl,port=2049,addr=10.43.7.20)10.43.7.20:/domain0/group-2/oracle_db_container-16/oracle_timeflow-16/datafile on /mnt/provision/Vdb1_14F/datafile type nfs (rw,nosuid,bg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,tcp,noacl,port=2049,addr=10.43.7.20)
    10.43.7.20:/domain0/group-2/oracle_db_container-16/oracle_timeflow-16/archive on /mnt/provision/Vdb1_14F/archive type nfs (rw,nosuid,bg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,tcp,noacl,port=2049,addr=10.43.7.20)
    10.43.7.20:/domain0/group-2/oracle_db_container-16/oracle_timeflow-16/external on /mnt/provision/Vdb1_14F/external type nfs (rw,nosuid,bg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,tcp,noacl,port=2049,addr=10.43.7.20)
    10.43.7.20:/domain0/group-2/oracle_db_container-16/oracle_timeflow-16/temp on /mnt/provision/Vdb1_14F/temp type nfs (rw,nosuid,bg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,tcp,noacl,port=2049,addr=10.43.7.20)

    If you don't see this, it might also be worth exercising the NFS mount function manually - you can do this by creating a directory then mounting the /public NFS share from Delphix:

    delphix_os@rhel62:/home/delphix_os-> mkdir /tmp/sean
    delphix_os@rhel62:/home/delphix_os-> sudo mount sean4230.dcenter.delphix.com:/public /tmp/sean
    delphix_os@rhel62:/home/delphix_os-> mount
    /dev/mapper/vg_rhel62-lv_root on / type ext4 (rw)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
    /dev/sda1 on /boot type ext4 (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
    sean4230.dcenter.delphix.com:/public on /tmp/sean type nfs (rw,vers=4,addr=10.43.7.20,clientaddr=10.43.4.179)
    If the public mount functions properly, it would be worth stopping the job and attempting to provision again; perhaps we encountered a transient issue. If, however, the public mount also fails, this indicates another issue.  In either instance you may be best served by engaging Delphix support for further guidance, as a job hang isn't expected behavior.
    
                                                    


  • 3.  RE: Delphix Provisioning is stuck at Recovering Oracle Database (40% for 24hours and counting)

    Posted 08-03-2015 02:28:00 AM
    Hi Sean,

    Good day. Thank you for your reply. Here is the our version: Delphix Engine - Demo 4.2.4.0. The source database was activated from being a standby database for this activity, so it might be a contributing event.

    This is actually our POC for Delphix and we have contacted the Support, but I was trying to troubleshoot it over the weekend. We were advised to stop the provisioning job so I was not able to try your suggestions while it was running. I will just update the thread of what we did, if possible.

    Regards,
    Mark


  • 4.  RE: Delphix Provisioning is stuck at Recovering Oracle Database (40% for 24hours and counting)

    Posted 08-05-2015 02:07:00 AM
    Hi,

    In case others encounter the same problem. We found out that the cause was that we were provisioning a VDB with a 62GB memory target but the server only had 32GB of RAM. Our job hanged because this error caused our swap space to increase to 100%. After editing the default values and memory target below the maximum, our job continued.

    Regards,
    Mark