Virtualization Plugins

The SDK cannot make changes to files/directories under the NFS vFile when mounted on a target.

The linked function pre_snapshot can write to the staging mount no problem.
Note that the permissions of the staging mount is 755 with ownership delphix_os:mysql, and the delphix_os user is in the mysql group.

But when the virtual functions (configure, pre_snapshot, post_snapshot) try to make changes to the target vFiles mount it fails with permission denied.  I tried creating files, removing files, and doing chmod, but they all fail.
Note that the permissions of the target vFile mount is 750 with ownership mysql:mysql (set using ownership_specification), and the delphix_os user is in the mysql group.
The issue looks to be that target mount is 750, which is not letting the delphix_os user make any changes.

I tried to chmod 775 on the mount from virtual.pre_snapshot and virtual.post_snapshot, but it fails with 'chmod: changing permissions of '/target/vdbA/mysql': Operation not permitted.   I try at the command line when logged in as delphix_os and get the same thing.  I tried as root and got the same thing.
Is there a way to change the permission on the mount point?  If not the run_host command, ran as delphix_os, cannot make any changes to the files/directories on the mount.

Also root, even though it is part of the mysql group, cannot cd to the target mount point or look at anything under it?  The delphix_os user can at least cd/read.  This is an issue because root needs access in order to start the service as the mysql user.

Let me know if you have any questions, suggestions, or would like to discuss.

This is a major issue for us and needs to be resolved before we can deploy the SDK anywhere.

Thanks,  Paul


------------------------------
Paul Jauquet
Senior Data Services Consultant
Axis Technology, LLC
------------------------------

Hey Paul,

The reason that root does not work is that by default we have "root squash" enabled for all NFS mounts. What this does is that root is mapped to an anonymous user by the NFS server which is the Delphix Engine in this case. It is a security feature of NFS servers, however it doesn't always provide any additional security in the context of plugins. You can turn this off if you choose to as part of the plugin config by setting "rootSquashEnabled: false":
https://developer.delphix.com/References/Plugin_Config/
"This dictates whether "root squash" is enabled on NFS mounts for the plugin (i.e. whether the root user on remote hosts has access to the NFS mounts). Setting this to false allows processes usually run as root, like Docker daemons, access to the NFS mounts. The default value is true. This field only applies to Unix hosts."

Once you disable "root squash", you would also be able to chmod the files when running as root, and root would have all the privileges for the files it usually would on the target host itself.

To allow delphix_os user write access to the files, I think we might have a couple of options but it would be great to set up some time to talk through the exact use cases and what you are trying to write to the mysql directory, so that we can suggest something appropriate. We should discuss more offline.


------------------------------
Ankur Sarin
Senior Engineering Manager
Delphix
------------------------------

Original Message:
Sent: 01-08-2020 10:13:10 AM
From: Paul Jauquet
Subject: SDK VDB mount permissions issues

The SDK cannot make changes to files/directories under the NFS vFile when mounted on a target.

The linked function pre_snapshot can write to the staging mount no problem.
Note that the permissions of the staging mount is 755 with ownership delphix_os:mysql, and the delphix_os user is in the mysql group.

But when the virtual functions (configure, pre_snapshot, post_snapshot) try to make changes to the target vFiles mount it fails with permission denied.  I tried creating files, removing files, and doing chmod, but they all fail.
Note that the permissions of the target vFile mount is 750 with ownership mysql:mysql (set using ownership_specification), and the delphix_os user is in the mysql group.
The issue looks to be that target mount is 750, which is not letting the delphix_os user make any changes.

I tried to chmod 775 on the mount from virtual.pre_snapshot and virtual.post_snapshot, but it fails with 'chmod: changing permissions of '/target/vdbA/mysql': Operation not permitted.   I try at the command line when logged in as delphix_os and get the same thing.  I tried as root and got the same thing.
Is there a way to change the permission on the mount point?  If not the run_host command, ran as delphix_os, cannot make any changes to the files/directories on the mount.

Also root, even though it is part of the mysql group, cannot cd to the target mount point or look at anything under it?  The delphix_os user can at least cd/read.  This is an issue because root needs access in order to start the service as the mysql user.

Let me know if you have any questions, suggestions, or would like to discuss.

This is a major issue for us and needs to be resolved before we can deploy the SDK anywhere.

Thanks,  Paul


------------------------------
Paul Jauquet
Senior Data Services Consultant
Axis Technology, LLC
------------------------------

Setting rootSquashEnabled: false made worked.  The root user can now see the mount and underlying directories/files.  Thanks!

We still need to look into how the SDK virtual functions can make changes to the files/directories under the NFS mount though.
Use cases would be to customize files under the mount to get the database to start, removing unnecessary databases, etc.  We could probably handle this using a configure clone hook script, but would be cleaner if we could handle it using the schema and associated SDK logic.  Also, if we find that we need to make changes to files under the mount point to start the VDB it would be need to be done in the SDK, as opposed to a clone script.  I should know more scenarios in the next few days now that we are past the root issue.

Thanks,  Paul





------------------------------
Paul Jauquet
Senior Data Services Consultant
Axis Technology, LLC
------------------------------

Original Message:
Sent: 01-08-2020 09:28:33 PM
From: Ankur Sarin
Subject: SDK VDB mount permissions issues

Hey Paul,

The reason that root does not work is that by default we have "root squash" enabled for all NFS mounts. What this does is that root is mapped to an anonymous user by the NFS server which is the Delphix Engine in this case. It is a security feature of NFS servers, however it doesn't always provide any additional security in the context of plugins. You can turn this off if you choose to as part of the plugin config by setting "rootSquashEnabled: false":
https://developer.delphix.com/References/Plugin_Config/
"This dictates whether "root squash" is enabled on NFS mounts for the plugin (i.e. whether the root user on remote hosts has access to the NFS mounts). Setting this to false allows processes usually run as root, like Docker daemons, access to the NFS mounts. The default value is true. This field only applies to Unix hosts."

Once you disable "root squash", you would also be able to chmod the files when running as root, and root would have all the privileges for the files it usually would on the target host itself.

To allow delphix_os user write access to the files, I think we might have a couple of options but it would be great to set up some time to talk through the exact use cases and what you are trying to write to the mysql directory, so that we can suggest something appropriate. We should discuss more offline.


------------------------------
Ankur Sarin
Senior Engineering Manager
Delphix

Original Message:
Sent: 01-08-2020 10:13:10 AM
From: Paul Jauquet
Subject: SDK VDB mount permissions issues

The SDK cannot make changes to files/directories under the NFS vFile when mounted on a target.

The linked function pre_snapshot can write to the staging mount no problem.
Note that the permissions of the staging mount is 755 with ownership delphix_os:mysql, and the delphix_os user is in the mysql group.

But when the virtual functions (configure, pre_snapshot, post_snapshot) try to make changes to the target vFiles mount it fails with permission denied.  I tried creating files, removing files, and doing chmod, but they all fail.
Note that the permissions of the target vFile mount is 750 with ownership mysql:mysql (set using ownership_specification), and the delphix_os user is in the mysql group.
The issue looks to be that target mount is 750, which is not letting the delphix_os user make any changes.

I tried to chmod 775 on the mount from virtual.pre_snapshot and virtual.post_snapshot, but it fails with 'chmod: changing permissions of '/target/vdbA/mysql': Operation not permitted.   I try at the command line when logged in as delphix_os and get the same thing.  I tried as root and got the same thing.
Is there a way to change the permission on the mount point?  If not the run_host command, ran as delphix_os, cannot make any changes to the files/directories on the mount.

Also root, even though it is part of the mysql group, cannot cd to the target mount point or look at anything under it?  The delphix_os user can at least cd/read.  This is an issue because root needs access in order to start the service as the mysql user.

Let me know if you have any questions, suggestions, or would like to discuss.

This is a major issue for us and needs to be resolved before we can deploy the SDK anywhere.

Thanks,  Paul


------------------------------
Paul Jauquet
Senior Data Services Consultant
Axis Technology, LLC
------------------------------