Delphix Products

We already granted all necessary rights to account "yyyy" to both source and stagning hosts.

Why we still got the following error:

Error
Failed to mount "xxx" because the mount command executed as user "yyyy" on host "160.88.112.66" failed for mount point.

Hi Danny,

The debug logs in the support bundle should have more detail on the failure. Sometimes the failure message in the GUI displays a little information icon "(i)". If you click on it, it may give you more information.

Usually, this happens because the user's sudo profile is not setup correctly or sometimes because the NFS client services are not running or installed on the machine where the VDB is being provisioned to.

Can you provide the output from "sudo -l" from the Delphix operating system user on host "160.88.112.66" ?

What operating system is host "160.88.112.66" running on?

The Delphix engine has a very small public share that you can test mounting against. Can you try a test mount against it? First make a directory:

mkdir /tmp/pub_test

Then depending on your operating system:
Solaris NFS

mount -F nfs -o rw,fg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,proto=tcp,sec=sys <delphix_engine_ip>:/public /tmp/pub_test

Linux NFS

mount -t nfs -o rw,fg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,tcp <delphix_engine_ip>:/public /tmp/pub_test

AIX NFS

mount -o rw,fg,hard,rsize=65536,wsize=65536,vers=3,nointr,timeo=600,proto=tcp <delphix_engine_ip>:/public /tmp/pub_test

Thanks,
  Neal

Where can I find the debug logs?

[sybase@cdcsrv83]/dbms/maintain/home/sybase/delphix_toolkits> sudo -l
Matching Defaults entries for sybase on this host:
    logfile=/var/adm/sudo.log, !env_reset, timestampdir=/var/adm/sudo, umask=0022, umask_override

User sybase may run the following commands on this host:
    (root) /usr/bin/smit, /etc/cfgmgr, /usr/bin/rm, /usr/bin/acledit, /usr/bin/rm, /usr/bin/vi, /usr/bin/ls,
    /usr/bin/chmod, /usr/bin/chown, /usr/bin/chgrp, /usr/bin/crontab, /usr/bin/mv, /usr/bin/cp, /usr/bin/tar,
    /usr/bin/kill, /usr/sbin/mount, /usr/sbin/umount, /etc/shutdown, /etc/accton, /usr/bin/startsrc,
    /usr/bin/stopsrc, /usr/bin/cancel, /usr/bin/lp, /usr/bin/lpstat, /usr/bin/enq, /usr/bin/enable,
    /usr/bin/disable, /usr/bin/lpr, /usr/bin/qchk, (root) NOPASSWD: /usr/sbin/mount, /usr/sbin/umount,
    /usr/sbin/nfso, /usr/bin/ps

It is a AIX 7.1 platform.

Those commands were working fine.

# sudo mount -o rw,fg,hard,rsize=65536,wsize=65536,vers=3,nointr,timeo=600,proto=tcp 160.81.64.49:/public /tmp/pub_test

# sudo umount /tmp/pub_test

I found this NFS file system was exported from Delphix Engine and mount process was executed on the Staging host. Wasn't it correct?  But, I have no idea why it didn't work.

Hi Danny,

If you download a support bundle from the engine, you can see the debug logs in a directory "log/mgmt_log".

The sudo test you have illustrated above indicates to me that you tested it from the root prompt (I see the "#" sign). Delphix does not "su" to root and then execute commands. It executes the commands as the environment user that was specified when you added the environment to Delphix so you should login as that user and try the commands I suggested.

The only thing I see missing from the sudo is "!requiretty". You will want to have that added.

After you add "!requiretty", could you try the mount commands I suggested while logged in as the user "sybase" (it appears that was the user your provided the "sudo -l" output for)?

If that doesn't help, please try the operation again from the GUI with the new sudo setting in place, generate a support bundle and upload it to our site https://upload.delphix.com

https://support.delphix.com/Delphix_Virtualization_Engine/Delphix_Admin/How_to_Create_and_Upload_Sup...

Let me know the name of the file or use a dummy case number "88888".

Thanks,
  Neal

File has been uploaded as follows:

Your file, 564d78aa-58da-a6cf-221d-f27c7dc1e579-20170915-5-42-43.tar, has completed processing successfully.

 

Support Ticket #88888 was updated with this upload.

I tried to NFS mount on cdcsrv83 and it worked as follows:

[sybase@cdcsrv83]/dbms/maintain/home/sybase> sudo mount -o rw,fg,hard,rsize=65536,wsize=65536,vers=3,nointr,timeo=600,proto=tcp 160.81.64.49:/public /dbms/maintain/home/sybase/delphix_toolkits/564d78aa-58da-a6cf-221d-f27c7dc1e579-staging-10/datafile
[sybase@cdcsrv83]/dbms/maintain/home/sybase> df
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4         5242880   5066920    4%     5990     2% /
/dev/hd2        10485760   4708488   56%    45177     8% /usr
/dev/hd9var      6291456   5601976   11%     5022     1% /var
/dev/hd3         6291456   5939120    6%      310     1% /tmp
/dev/hd1         2097152   1871976   11%      874     1% /home
/proc                  -         -    -         -     -  /proc
/dev/hd10opt     1048576    720136   32%     3628     5% /opt
/dev/vartmplv    4194304   4192928    1%       21     1% /var/tmp
/dev/acctfile   10485760  10236152    3%        7     1% /var/adm/acf
/dev/nsr          524288    523048    1%       81     1% /nsr
/dev/nimbackup   20971520  12669456   40%        5     1% /nimbackup
/dev/syscfg     12582912  11842816    6%       82     1% /appl/toss/syscfg
/dev/backuplog     524288    523552    1%        4     1% /var/backuplog
/dev/haunixcp   12582912  12016304    5%     3428     1% /appl/haunixcp
/dev/high_avail     524288    523304    1%       26     1% /high_availability
/dev/security    4194304   4191608    1%      125     1% /appl/toss/security
/dev/tossdump   10485760   9906256    6%      755     1% /appl/toss/dump
/dev/san_home     524288    495672    6%      996     2% /appl/san/home
/dev/hd11admin     524288    523488    1%       13     1% /admin
/dev/livedump     524288    523552    1%        4     1% /var/adm/ras/livedump
/dev/dbms        4194304   3439592   18%     2060     1% /dbms
/dev/product     3145728   3010280    5%      162     1% /product
/dev/symantec_esm65    2097152   1488520   30%      145     1% /product/symantec/esm65
/dev/symantec_scsp52    1048576    898824   15%      154     1% /product/symantec/scsp52
/dev/patrol_home_831   10485760  10431696    1%      986     1% /appl/patrol/home
/dev/rs_home_831     262144    169616   36%     1671     9% /appl/rs/home
/dev/rs_rsmdata_831     262144    261368    1%       10     1% /appl/rs/rsmdata
/dev/oc_job_831      65536     64432    2%       40     1% /appl/oc/job
/dev/oc_stag_831      65536     64864    2%        4     1% /appl/oc/staging
/dev/san_data_831    2097152   2057824    2%     1639     1% /appl/san/data
/dev/atl_home_831    2097152    836368   61%      130     1% /appl/atl/home
/dev/sybase15    4194304   4168840    1%      106     1% /dbms/sybase15
/dev/ebf23008   12582912   7271968   43%     7027     1% /dbms/sybase15/ebf23008
/dev/ebf21337   10485760   5245400   50%     8880     2% /dbms/sybase15/ebf21337
/dev/rs1260      4194304   4123176    2%       14     1% /dbms/rs1260
/dev/rs1570      4194304   2532016   40%     3886     2% /dbms/rs1570
/dev/sybasetar   20971520   9007912   58%      625     1% /dbms/sybasetar
/dev/ebf11659    4194304   2727464   35%     3995     2% /dbms/sybase1250/ebf11659
/dev/ctrlm8_home_831    3145728   2078816   34%     6339     3% /appl/ctrlm8/home
/dev/addm_home_831      65536     64624    2%        9     1% /appl/addm/home
/dev/bppm        6291456   5013760   21%     8669     2% /product/bppm
/dev/biem        4194304   3231584   23%     8076     3% /product/biem
/dev/ctrlm8_sysout    7340032   7338256    1%        4     1% /product/ctrlm8_sysout
/dev/hrps_home_831    4194304   4192936    1%        7     1% /appl/hrps/home
/dev/cmsmoe_home_831      65536     61480    7%       41     1% /appl/cmsmoe/home
/dev/eis_home_831   10420224   5337088   49%    24534     4% /appl/eis/home
/dev/eis_audit_831    8388608   8386184    1%        5     1% /appl/eis/audit_iq
/dev/ebf15597    6291456   4874080   23%     5930     2% /dbms/sybaseiq127/ebf15597
/dev/ebf21214    6291456   4872880   23%     5842     2% /dbms/sybaseiq127/ebf21214
/dev/sybaseiq16    4194304   4180464    1%       60     1% /dbms/sybaseiq16
/dev/ebf24073   12582912   6360872   50%     5130     1% /dbms/sybaseiq16/ebf24073
/dev/eis_dump_831   52428800   7490040   86%       28     1% /appl/eis/dump
/dev/eis_dump_832   29818880   2503760   92%       20     1% /appl/eis/dump2ase
/dev/eis_dump_833    3145728   3144592    1%        4     1% /appl/eis/dump3ase
/dev/eis_data_831   10485760   3555592   67%     3709     1% /appl/eis/data
/dev/eis_data_832    4194304   4193008    1%        4     1% /appl/eis/dataase
/dev/eis_data_834    1048576   1047760    1%        4     1% /appl/eis/dumpase
/dev/ebf25543   12582912   7538200   41%     3905     1% /dbms/sybaseiq16/ebf25543
/dev/cdcsrv83  419430400 302251008   28%       44     1% /appl/eis/iqdumptest
/dev/hkpmi_home_831     524288    523480    1%        7     1% /appl/hkpmi/home
/dev/emp_home_831    2097152   1749672   17%     3100     2% /appl/emp/home
/dev/lis_home_834    1048576   1047528    1%       28     1% /appl/lis/home6
/dev/lis_home_832    4194304   4192520    1%       50     1% /appl/lis/home2
/dev/lis_home_831   10485760  10286344    2%      778     1% /appl/lis/home
/dev/lis_home_833    1048576   1047528    1%       28     1% /appl/lis/home3
/dev/epr_home_831      65536     64824    2%        6     1% /appl/epr/home
/dev/lis_tempdb_831   25165824  25161328    1%        4     1% /appl/lis/db
/dev/lis_tempdb_832    2097152   2096176    1%        4     1% /appl/lis/db2
/dev/lis_dtd_831     131072    130392    1%        4     1% /appl/lis/dtd
/dev/lis_home_837    1048576   1047760    1%        4     1% /appl/lis/home7
/dev/lis_data_831    4194304   4193008    1%        4     1% /appl/lis/lstore
/dev/lis_utils_831     131072     76424   42%      100     2% /appl/lis/utils
/dev/lis_dump_831   41943040  41154440    2%       14     1% /appl/lis/dumpx
/dev/lis_test_831    4194304   4193008    1%        4     1% /appl/lis/test
/dev/lis_dump_832    4194304   4193008    1%        4     1% /appl/lis/dump
/dev/lis_eaisto_831    4194304   4193008    1%        4     1% /appl/lis/eaistore
/dev/lisver_83lis   10485760  10483504    1%        4     1% /appl/lis/verctrl
/dev/lispdf_83lis    4194304   4193008    1%        4     1% /appl/lis/pdfstore
/dev/cmsmsgbc_hom831    2097152   2076584    1%     1000     1% /appl/cmsmsgbc/home
/dev/cmsarch_home831    4194304   4141224    2%     2224     1% /appl/cmsarch/home
/dev/sam_home_831    4194304   2586736   39%    11861     4% /appl/sam/home
/dev/lrskwc_home_831    8388608   7667800    9%     5981     1% /appl/lrskwc/home
/dev/hafm_home_831    2097152   2093344    1%       40     1% /appl/hafm/home
/dev/adt_home_831   20971520  16237584   23%       35     1% /appl/adt/home
/dev/cmscorp_home831    4194304   3700144   12%       46     1% /appl/cmscorp/home
/dev/truesight10   20971520  20967664    1%        4     1% /product/truesight10
/dev/sybase16    4194304   4192976    1%        6     1% /dbms/sybase16
/dev/ebf26923   14680064   3822160   74%     7662     2% /dbms/sybase16/ebf26923
/dev/ebf26709   12582912  12580336    1%        4     1% /dbms/sybase15/ebf26709
/dev/dbs_backup_831   20971520  20967664    1%        4     1% /appl/dbs/backup
/dev/fsbackup   62914560  62904296    1%        4     1% /fsbackup
160.81.64.49:/public  546718120 546718072    1%        7     1% /dbms/maintain/home/sybase/delphix_toolkits/564d78aa-58da-a6cf-221d-f27c7dc1e579-staging-10/datafile
[sybase@cdcsrv83]/dbms/maintain/home/sybase>

I have tested below command on 160.88.112.66 and they all are success.

export DLPX_BIN=/dbms/maintain/home/sybase/delphix_toolkits/Delphix_COMMON_564d78aa_58da_a6cf_221d_f27c7dc1e579_sybase_host/scripts/bin
/dbms/maintain/home/sybase/delphix_toolkits/Delphix_COMMON_564d78aa_58da_a6cf_221d_f27c7dc1e579_sybase_host/scripts/bin/dlpx_mount -F nfs -o rw,fg,hard,rsize=1048576,wsize=1048576,vers=3,nointr,timeo=600,proto=tcp,sec=sys 160.81.64.49:/domain0/group-2/ase_db_container-53/ase_timeflow-54/datafile /dbms/maintain/home/sybase/delphix_toolkits/564d78aa-58da-a6cf-221d-f27c7dc1e579-staging-10/datafile

/dbms/maintain/home/sybase/delphix_toolkits/Delphix_COMMON_564d78aa_58da_a6cf_221d_f27c7dc1e579_sybase_host/scripts/bin/dlpx_umount  /dbms/maintain/home/sybase/delphix_toolkits/564d78aa-58da-a6cf-221d-f27c7dc1e579-staging-10/datafile

Hi Danny,

Thanks for uploading the support bundle. It certainly shows that the user "sybase" is being denied sudo permission to execute the "mount" command:

[2017-09-21 10:15:40,194][DEBUG][util.mount.UnixMounter#mount:97][Worker-624|JOB-539|DB_SYNC(ASE_DB_CONTAINER-10)][ACTION-740] Executed mount script with result RemoteResult{status=1, stdout=ERROR: No sudo privileges or incorrect password
, stderr=nfsmnthelp: 1831-006 NFS mounts/unmounts are only allowed by
        superuser and by members of the system group
stty: tcgetattr: A specified file does not support the ioctl system call.
stty: tcgetattr: A specified file does not support the ioctl system call.
mount.sh: MOUNT_FAILED
}

Since you have illustrated that the user "sybase" has the correct sudo permissions to execute "/usr/sbin/mount" during an interactive login, I suspect that there might be a different "mount" executable in another directory that is found in the user's PATH environment variable. You may be able to review the AIX system messages to see why it is denying us permission to the command.

Alternatively, you could also try this the following to see if there are multiple "mount" executables found. Delphix executes the commands as a non-interactive login and I have seen (especially on AIX) that the environment settings can be different for non-interactive versus interactive logins. Could you try the following and replace the AIX host with yours and let me know the result (do not login directly to the host to issue the "whereis" command unless you wish to compare the results)?

$ ssh sybase@aix101-14.delphix.com "whereis mount"
sybase@aix101-14.delphix.com's password: 
mount: /usr/sbin/mount

Next, if neither checking the AIX system logs nor the "whereis" provide any insight, you could edit the file "/dbms/maintain/home/sybase/delphix_toolkits/Delphix_COMMON_564d78aa_58da_a6cf_221d_f27c7dc1e579_sybase_host/scripts/bin/dlpx_pfexec" and add a "-x" to the very top like this:

#!/bin/sh -x

Then execute the "dlpx_mount" command just as you illustrated above. You should see more verbose output that shows Delphix does not hardcode the path to the commands (mount/umount/mkdir/rmdir) it is executing. You should see something like this:

+ [ 212 -eq 0 ]
+ sudo -l -S
+ 1>> /dev/null 2>& 1 + [ 0 -ne 0 ]
+ sudo -p  -S mount -o rw,fg,hard,rsize=524288,wsize=524288,vers=3,nointr,timeo=600,proto=tcp 10.43.3.176:/public /work/nstack/toolkit/test
+ 2>& 1 0<<

And maybe we can spot where the permission is being denied?

Thanks,
  Neal