Delphix Provisioning is stuck at Recovering Oracle Database (40% for 24hours and counting)

  • 0
  • 1
  • Question
  • Updated 3 years ago
  • Answered
What do I need to check to know that is still is ongoing, or if it hanged/encountered an error. There are no messages in the Delphix Engine GUI.
Photo of Mark

Mark

  • 160 Points 100 badge 2x thumb

Posted 3 years ago

  • 0
  • 1
Photo of Sean Nothdurft

Sean Nothdurft, Technical Support

  • 860 Points 500 badge 2x thumb
Official Response
Hi Mark - It's difficult to say what's happening on your Engine without logs... at this point in the process (40-50%) it should be mounting the filesystems from Delphix, then starting the Oracle instance for the VDB, before altering settings like disable Flashback, etc, in preparation for rename of datafiles.  All of these preparatory steps are usually not that time consuming; any real work would be expected in the processing of datafiles, not mounting filesystems.

Out of curiosity, a few questions: 
- What version of Delphix are you using? 
- Is this the first provision job after some kind of environmental event, Engine or host reboot, etc?

If the process appears to be stuck, you can check the mount state from the target environment, and look for the Delphix Engine IP address; here's an example below:

delphix_os@rhel62:/home/delphix_os-> mount
/dev/mapper/vg_rhel62-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
10.43.7.20:/domain0/group-2/oracle_db_container-16/oracle_timeflow-16 on /mnt/provision/Vdb1_14F type nfs (rw,nosuid,bg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,tcp,noacl,port=2049,addr=10.43.7.20)10.43.7.20:/domain0/group-2/oracle_db_container-16/oracle_timeflow-16/datafile on /mnt/provision/Vdb1_14F/datafile type nfs (rw,nosuid,bg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,tcp,noacl,port=2049,addr=10.43.7.20)
10.43.7.20:/domain0/group-2/oracle_db_container-16/oracle_timeflow-16/archive on /mnt/provision/Vdb1_14F/archive type nfs (rw,nosuid,bg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,tcp,noacl,port=2049,addr=10.43.7.20)
10.43.7.20:/domain0/group-2/oracle_db_container-16/oracle_timeflow-16/external on /mnt/provision/Vdb1_14F/external type nfs (rw,nosuid,bg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,tcp,noacl,port=2049,addr=10.43.7.20)
10.43.7.20:/domain0/group-2/oracle_db_container-16/oracle_timeflow-16/temp on /mnt/provision/Vdb1_14F/temp type nfs (rw,nosuid,bg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,tcp,noacl,port=2049,addr=10.43.7.20)

If you don't see this, it might also be worth exercising the NFS mount function manually - you can do this by creating a directory then mounting the /public NFS share from Delphix:

delphix_os@rhel62:/home/delphix_os-> mkdir /tmp/sean
delphix_os@rhel62:/home/delphix_os-> sudo mount sean4230.dcenter.delphix.com:/public /tmp/sean
delphix_os@rhel62:/home/delphix_os-> mount
/dev/mapper/vg_rhel62-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
sean4230.dcenter.delphix.com:/public on /tmp/sean type nfs (rw,vers=4,addr=10.43.7.20,clientaddr=10.43.4.179)
If the public mount functions properly, it would be worth stopping the job and attempting to provision again; perhaps we encountered a transient issue. If, however, the public mount also fails, this indicates another issue.  In either instance you may be best served by engaging Delphix support for further guidance, as a job hang isn't expected behavior.
(Edited)