Hi Raj,
Thanks for your question! In terms of functionality, the Delphix Engine would work in an environment when using mixed vmfs3 and vmfs5 filesystems, as it will see both as regular virtual disks. That being said, we would not recommend this sort of configuration as there may be performance differences between vmfs3 and vmfs5 that when used in a mixed mode could create inconsistencies in the performance of your Delphix Engine
For details on vmfs5 vs. vmfs3:
http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.storage.doc_50%2FGUID-7552DA...Please note the first line: "VMFS5 provides many improvements in scalability and performance over the previous version."
Thanks,
Srini