Virtualization Plugins

 View Only
  • 1.  Questions on using direct linking

    Posted 01-21-2020 08:17:00 PM
    Edited by Paul Jauquet 01-22-2020 11:21:48 AM
    I am trying to use direct linking and have a few questions.

    1. Why does run_sync() not return anything?  I'd think it should return like run_bash so that we can check exit code and error.

    2. I am calling run_sync() from the linked_pre_snapshot operation.  I assume this is where it should be called from?  When I call it once it works fine.  But when I call it twice it gives me the below error:
    DirectLinkingErrorWhenRunSyncIsRanMoreThanOnce

    I want to be able to loop to call run_sync() until the duration is less than a certain threshold.  When the directory being synchronized is large the initial sync could take quite a while (i.e. hours).  I'd like to run it, and then check how long it took.  If it took more than x seconds I'd like to run it again to get closer.  I'd like to loop thru this until the rsync takes less than x seconds, at which point I'll call run_bash to flush the database, and then run the rsync one last time so as to get a read consistent version of the files, and then call run_bash to unlock the database.   But when I try to run it more than once it is failing?


    3. Is there a way to remove files that are on the target that are no longer on the source?  I assume run_sync will sync files from the source dir to the engine, but how to remove anything that is on the engine that is no longer on the source (i.e. they deleted from source)?
    When I used staged linking I was able to run the following:
    i.e. rsync -r --delete --existing --ignore-existing --ignore-errors --progress <src> <tgt>

    4. I tried to pass in rsync_user to run rsync as a user that has permissions on the underlying files.  When I don't send the default delphix user does not have access since the files are 600 and not owned by delphix.  With staged linking I was able to run as root and worked fine.  When I pass in rsync_user="root" it seems to be ignoring it and running as delphix?   Does this need to be a user defined for the environment in Delphix?  Or can we setup trusted ssh on the source so that delphix can remote ssh as root?  If so where would I get the delphix key to add to the authorized_hosts file?  Please explain how to properly use rsync_user?

    5. I am not seeing the 'Resynchronize dsource' option for the dSource?  All I see is 'Disable' and 'Delete'.  I see it when the SDK is staged linking, but not for my direct linking SDK.  Am I missing an operation or something?   

    I have everything working with staged linking, but would like to get it working with direct linking since I'd think rsync would be quicker since it would be going directly from the source to the engine where it is ingested, as opposed to going to a target and then back to the engine via NFS.  Initial testing looks to be quicker, but I still need to figure out how to accomplish the same (i.e. looping rsync, deleting files, etc) to make it a workable option.

    Thanks!  Paul

    ------------------------------
    Paul Jauquet
    Senior Data Services Consultant
    Axis Technology, LLC
    ------------------------------


  • 2.  RE: Questions on using direct linking
    Best Answer

    Posted 01-22-2020 01:55:00 PM

    Hi Paul 

    1. That's a good question. Currently, run_sync() operation follows a little bit different semantics than run_bash() and it throws a user exception when something goes wrong. The exception would be displayed in the UI. It is currently not possible to react to a failed run_sync() call from the plugin context.

    2. This might be a bug in the Delphix engine code. We're investigating the issue and will get back to you soon.

    3. The files on the engine will reflect the state on the source as we run rsync with --delete flag. Here are all the flags we pass to rsync:

    "-a, --delete, --delete-excluded, --ignore-errors, --inplace, --relative, --delete-missing-args, --partial, --no-inc-recursive, --info=progress2"

    1. Rsync user doesn't need to be an environment user. If you're able to run rsync from a command line with given user, you should be able to pass that user name string to run_sync() as described in https://developer.delphix.com/References/Platform_Libraries/
    2. You can resynchronize a directly linked dSource by executing a snapshot operation (the camera icon in the UI).


    Is reducing the number of network calls the primary reason for implementing a direct plugin?

    - Filip



    ------------------------------
    Filip Drozdowski
    Senior Member of Technical Staff I
    Delphix
    ------------------------------



  • 3.  RE: Questions on using direct linking

    Posted 01-23-2020 08:19:00 PM
    Hi Paul,

    I wanted to make a correction regarding 1). You should be able to handle errors thrown by run_sync() operation by catching a LibraryError. Here's an example:

    from dlpx.virtualization import libs
    from
    dlpx.virtualization.libs.exceptions import LibraryError

    ...

    source_directory = "sourceDirectory"
    rsync_user = "rsyncUser"
    exclude_paths = ["/path1", "/path2"]
    sym_links_to_follow = ["/path3", "/path4"]

    try:
    libs.run_sync(connection, source_directory, rsync_user, exclude_paths, sym_links_to_follow)
    except LibraryError:
    ## Handle the error here.
    print("run_sync() failed")
    ​I don't believe the LibraryError is currently documented. We will make sure to fix the documentation.

    -Filip

    ------------------------------
    Filip Drozdowski
    Senior Member of Technical Staff I
    Delphix
    ------------------------------



  • 4.  RE: Questions on using direct linking

    Posted 01-27-2020 12:48:00 PM
    Hi Paul, 

    Thank you for all the detail. This was diagnosed as a bug in the Delphix engine. We have put this bug on our backlog and will address in a future Delphix release (tentatively one of our 6.0.x releases this year). 

    Let us know if you have any questions or concerns. 

    Best, 

    Alexandros

    ------------------------------
    Alexandros Mathopoulos
    Product Manager 2
    Delphix
    ------------------------------