Delphix Products

Expand all | Collapse all

DB_SYNC job failed due to an error during execution

Adam Bowen04-20-2015 12:26:00 PM

  • 1.  DB_SYNC job failed due to an error during execution

    Posted 04-16-2015 12:13:00 PM
    Hello,

    After launching the first snapshot from a sqlserver source (VM), on Landshark 2.0, we get this error:
    "DB_SYNC job for ... failed due to an error during execution:
    Failed to discover and mount LUNs from the Delphix Engine through the iSCSI initiator on target host ... .
    Make sure that the iSCSI initiator is running and that the user ... has privileges to perform disk management on the host."

    We double-checked that:
    delphix user is a domain user with local administrator privileges on both source and target,
    it has sysadmin role on both SQL Server instances (source and target),
    and has "Logon as Batch job" rights on target.
    Delphix Connector and iSCSI initiator services are up and running on target machine.
    (target machine is used as staging too)

    Could you help us to solve this issue?
    Thanks. Note: This conversation was created from a reply on: Tip of the Day: SQL Server Set Up and a Windows User.


  • 2.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-16-2015 05:34:00 PM
    Hello,

    Can you execute the "iscsicli.exe" command from the command line without getting an error? If you get an error while trying to execute this command, you will get the same error you reported from Delphix:

    C:\Windows\system32>iscsicli
    Microsoft iSCSI Initiator Version 6.1 Build 7601  Error getting node name: Access is denied.
    If you are able to execute it without error, you should see the IQN returned which is what Delphix is looking for: C:\Users\delphixaduser>iscsicli Microsoft iSCSI Initiator Version 6.1 Build 7601  [iqn.1991-05.com.microsoft:10-43-3-155.ad.delphix.com]  Thanks,   Neal


  • 3.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-17-2015 06:47:00 AM
    Hello,

    thanks for your reply.

    Running "iscsicli.exe" as delphix user on target machine will result in something like your example:

    [iqn.1991-05.com.microsoft:delphix]

    so, it seems ok.

    What do you suggest now?

    Thanks
    Giorgio



  • 4.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-17-2015 12:50:00 PM
    Hello,

    Did you execute the "iscsicli.exe" as the same user defined as the "Environment User"? In other words, you did not try executing it from a different user account such as "Administrator"?

    Could there be a firewall between Landshark and SQL Server blocking port 3260? There are certain ports that need to be accessible:
       http://docs.delphix.com/display/DOCS42/Network+and+Connectivity+Requirements+for+SQL+Server+Environm...

    Thanks,
      Neal




  • 5.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-17-2015 01:24:00 PM
    Hello,

    I executed the "iscsicli.exe" as the same user defined as the "Environment User" and also with "Administrator" getting same result. ([iqn.1991-05.com.microsoft:delphix])

    Source, Target and Landshark machines are on same network without any firewall / network limitations.

    Just to be sure I turned off (temporarly) windows firewall on source and target machines.

    Unfortunately still not works!

    May I do some testing activities on landshark?

    Thanks,
    Giorgio







  • 6.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-17-2015 01:51:00 PM
    Hello,

    I work in Delphix technical support and would normally request that a technical support case be opened and a support bundle uploaded to it. It is my understanding that the Landshark environment is supported by a different group here at Delphix so I would expect one of them to follow on soon with a request or suggestion.

    Thanks,
      Neal



  • 7.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 11:08:00 AM
    Giorgio, what happens if you change the environment user to the Administrator account? Does the error persist? This is just for troubleshooting.


  • 8.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 12:10:00 PM
    Hello Adam,

    the error persists even using the Administrator account instead of delphix as the environment user (for both source and target/staging Environment).

    Thanks,
    Giorgio







  • 9.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 12:26:00 PM
    What version of Windows is this?


  • 10.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 12:31:00 PM
    Can you post the iqn's from all the Windows environments attached to this Delphix engine?


  • 11.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 12:33:00 PM
    Also, can you post a screenshot of this from your target?  "iSCSI Initiator" GUI ->Targets -> Discovered Targets -> Status field


  • 12.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 01:14:00 PM
    Hello Adam,

    source and target/staging are VM both equipped with:
    Windows Server 2008 R2 SP1 64bit
    SQLServer Enterprise 11 (2012) 64bit

    source iqn (iSCSI initiator GUI > Configuration): iqn.1991-05.com.microsoft:delphix1.delphix.delphix
    target iqn (iSCSI initiator GUI > Configuration): iqn.1991-05.com.microsoft:delphix2.delphix.delphix

    here is the screenshot taken from target machine:


    Hope this helps.

    Thanks,
    Giorgio





  • 13.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 01:17:00 PM
    Do you get any errors when refreshing your source or target environment in the Delphix Console?


  • 14.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 01:24:00 PM
    I did source and target environment refresh many times.
    Never gotta an error.


  • 15.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 01:34:00 PM
    Then I am even more perplexed as the refresh actually detects and builds these iSCSI LUNs as part of the refresh process, I believe. That is why you have the discovered Target in the iSCSI initiator. 

    When you first created the dSource, did it have a time card entry? Or was it blank, and then you tried to take a snapshot?


  • 16.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 01:38:00 PM
    It was blank.
    This issue occurs trying to get the very first snapshot


  • 17.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 01:41:00 PM
    Ok. A snapshot should be there at the end of successfully adding the dSource (look in the dashboard). Can you post the entire job log from adding the dSource? If not, can you please delete and re-add it, then post the entire log? Then create and download a support bundle and email it to hsic@delphix.com.


  • 18.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 02:08:00 PM
    Hello,

    I delete and re-created dSource.

    Here is job logs:

    DB_LINK job started for "sqlserver_sources/test_source".
    DB_LINK job for "sqlserver_sources/test_source" completed successfully.
    DB_SYNC job started for "sqlserver_sources/test_source".
    Starting sync from the source database "test_source"
    Creating new TimeFlow for dSource "test_source"
    Exporting storage containers.
    Generating scripts needed for validated sync
    Validating user environment settings on the staging host "10.80.3.221"
    Mounting datasets on the staging host "10.80.3.221"
    Failed to discover and mount LUNs from the Delphix Engine through the iSCSI initiator on target host "10.80.3.221".

    Thank you so much,
    I appreciate your helps.

    Giorgio




  • 19.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 02:13:00 PM
    Can you go back to that target in the iSCSI initiator and disconnect it? Then refresh the target in the Delphix engine, then delete and re-add the dSource. Post the logs


  • 20.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 02:30:00 PM
    iSCSI initiator  disconnected
    target environment refreshed
    dSource deleted and re-created (with same settings)

    job logs are IDENTICAL to the previous try.

    DB_LINK job started for "sqlserver_sources/test_source".
    DB_LINK job for "sqlserver_sources/test_source" completed successfully.
    DB_SYNC job started for "sqlserver_sources/test_source".
    Starting sync from the source database "test_source"
    Creating new TimeFlow for dSource "test_source"
    Exporting storage containers.
    Generating scripts needed for validated sync
    Validating user environment settings on the staging host "10.80.3.221"
    Mounting datasets on the staging host "10.80.3.221"
    Failed to discover and mount LUNs from the Delphix Engine through the iSCSI initiator on target host "10.80.3.221".




  • 21.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-20-2015 02:50:00 PM
    Can you download and email the support bundle to me?


  • 22.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-21-2015 07:37:00 AM
    I sent you an email with a Dropbox link to download the support bundle.

    Thanks,
    Giorgio


  • 23.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-21-2015 09:50:00 AM
    Got it Giorgio, I have found the root error message, but I am not sure what is causing it yet. I will update you when I know a little more.


  • 24.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-21-2015 09:56:00 AM
    Great. I look forward to get some news from you soon (hope).
    Thanks again.


  • 25.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-21-2015 03:43:00 PM
    Can you send me the screen shots from here?
    1. In Control Panel, open Regional and Language Options.

    2. The Languages tab, 

    3. The Regional Options tab

    4. The Advanced tab




  • 26.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-21-2015 08:55:00 PM
    Giorgio, we believe it is a localization issue since you are on an Italian locale in Windows. It is something we are looking into and will address some time in the future. For now, there are a few things we can try.

    1) Set the user locale for the environment user for the Target to us_EN

    2) Set the system locale for the Target to us_EN

    3) we have better support for languages in 4.2, I can send you a link for the download and you can try with that




  • 27.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-30-2015 07:14:00 AM
    Hi,

    I am also having the same issue. Did you able to find any solution?

    Thanks,
    Vikas


  • 28.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-30-2015 11:50:00 AM
    Hello Adam,

    the following screenshots show the actual configuration on both source and target environments.











    Unfortunatley, even though I switched the user and system locale to us_En, DB_SYNC still not works (always the same error).

    Have you other advices?

    Could you provide me a link to download the 4.2 version, I will try with that.

    Thanks,
    Giorgio



  • 29.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-30-2015 12:42:00 PM
    Can you refresh your target environment, try to snapshot again, and then send me the support bundle?


  • 30.  RE: DB_SYNC job failed due to an error during execution

    Posted 04-30-2015 01:18:00 PM
    I sent you an email with a Dropbox link to download the support bundle.

    Thanks,
    Giorgio


  • 31.  RE: DB_SYNC job failed due to an error during execution

    Posted 05-06-2015 02:42:00 PM
    Hello,

    we are getting the same issues with 4.2 version.

    Digging into the error messages we found "No volume specified" error from PS script output:
    ...
    Microsoft DiskPart versione 6.1.7601
    Copyright (C) 1999-2008 Microsoft Corporation.
    Nel computer DELPHIX2
    Il disco attualmente selezionato � il disco 4.
    La partizione attualmente selezionata � la partizione 1.
    Nessun volume specificato.
    Selezionare un volume e riprovare.
    ERROR_ASSIGN_MOUNTPATH: failed to assign mount path for disk 4 at C:\Delphix\De
    lphixConnector42\564ddbcc-9f64-0680-a2dc-1e5959a3cdc9-staging-3\DATA, error -21
    47024809
    Looking into  our stanging env, which I remind you is a 2008 r2, we found the file C:\Delphix\De
    lphixConnector42\564ddbcc-9f64-0680-a2dc-1e5959a3cdc9-staging-3\SCRIPT\MountLunData.ps1
    which we think it is raising the error (see lines 350 - 358).
    # Assign a mount path to the volume
    $diskScript = "select disk $diskDeviceNumber'
    select partition $partition'
    assign mount='"$mountpath'"'
    exit"
    $err = diskpartRetry $diskScript 10 6
    if ($err -ne 0) {
       "ERROR_ASSIGN_MOUNTPATH: failed to assign mount path for disk $diskDeviceNumber at $mountpath, error $err"
       exit $err
    }
    As you can see the Remarks on Microsoft Technet for "ASSIGN" syntax on Win 2008 r2 seems to be mandatory  setting a volume:
    A volume must be selected for this operation to succeed. Use the select volume command to select a volume and shift the focus to it.
    That is missing in ps script. So this could be the point.
    What's your opinion?

    Regards,
    Giorgio


  • 32.  RE: DB_SYNC job failed due to an error during execution

    Posted 05-06-2015 04:09:00 PM
    It's an issue with the locale and the mount script. The diskpart commands are coming back in non-English and that leads to the failure to mount the LUN,  We will work on fixing this in a future release.


  • 33.  RE: DB_SYNC job failed due to an error during execution

    Posted 05-07-2015 06:55:00 AM
    Hello,

    do you think that this issue could be bypassed by installing the MUI package for Windows Server 2008 (and PowerShell) and setting the English language for Delphix user?

    Thanks,
    Giorgio




  • 34.  RE: DB_SYNC job failed due to an error during execution

    Posted 05-22-2015 10:07:00 AM
    Hello again,

    we found a Windows 2008 server in English.

    Now, obviously only for testing purposes, could we use only one server for source, target and staging?

    Should we pay attention on something in particular in that case?

    Thanks



  • 35.  RE: DB_SYNC job failed due to an error during execution

    Posted 05-22-2015 10:09:00 AM
    Yes, I have done it. Purely for exploratory purposes. As you can imagine, the performance won't be great.