Delphix Toolkits (dxToolkit and dxmToolkit)

Hi,

I have a dSource that was setup using Staging Push (SQL Server) as we use Rubrik as a backup provider for our db's. I was able to restore the staging db using an automated script and also tried to run the snapsync using dxtoolkit command after the staging db restore is complete.

But I get the below error when trying to do a snapync using dxtoolkit command:

c:\dxtoolkit2>dx_snapshot_db -d delphixservername -name DBName
No data returned for resources/json/delphix/sourceconfig. Try to increase timeout
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Starting job JOB-300 for database DBName.
0
Job JOB-300 finished with state: FAILED
Last message is: Sync parameter type "MSSQL_NEW_COPY_ONLY_FULL_BACKUP_SYNC_PARAMETERS" is not supported for sync strategy type "MSSQL_STAGING_PUSH_SYNC_STRATEGY"

But when I do it via GUI, it finished successfully.

I am trying to automate this whole process with staging push (staging db restore + snapsync) as we need to maintain the staging db restores.

Can some one help resolve this error while trying to use dxtoolkit command?

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc
------------------------------

Hello,

I see two errors.

1. There is a mention of increasing the timeout for the command. Can you edit your "dxtools.conf" configuration file and increase the timeout?
% grep -i timeout dxtools.conf
            "timeout" : "60"

2. It mentions that a new, copy only backup is invalid for a staging push dSource. If you use the "--help" parameter, it gives you some details on the other backup types:
    -usebackup yes|no For MS SQL / Sybase dSource only - if this flag is set
    to yes - snapshot will to loaded from last known backup, if flag is set
    to no Delphix Engine will run full backup Default value is no
    -backupfileslist backupfile1,backupfile2,... For Sybase dSource only -
    specify a list of backup files as a list of comma separated backup file
    names
    -backupfilesfile /path/to/file_with_backup For Sybase dSource only -
    specify a file contains a list of backup files. One file per line
    -backupuuid uuid For MS SQL only - UUID of backup to ingest

So you probably want to use the "-usebackup yes" parameter so that it does not try to initiate a copy only backup. Then you have to decide if you will provide a backupset UUID (-backupuuid) or the names of the backup files you want to restore (-backupfileslist).

I hope that helps,
Neal

------------------------------
Neal Stack
Senior Principal Technical Support Engineer
Delphix

Original Message:
Sent: 05-01-2023 10:46:09 AM
From: Pratyusha B
Subject: Staging Push snapsync dxtool command error

Hi,

I have a dSource that was setup using Staging Push (SQL Server) as we use Rubrik as a backup provider for our db's. I was able to restore the staging db using an automated script and also tried to run the snapsync using dxtoolkit command after the staging db restore is complete.

But I get the below error when trying to do a snapync using dxtoolkit command:

c:\dxtoolkit2>dx_snapshot_db -d delphixservername -name DBName
No data returned for resources/json/delphix/sourceconfig. Try to increase timeout
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Starting job JOB-300 for database DBName.
0
Job JOB-300 finished with state: FAILED
Last message is: Sync parameter type "MSSQL_NEW_COPY_ONLY_FULL_BACKUP_SYNC_PARAMETERS" is not supported for sync strategy type "MSSQL_STAGING_PUSH_SYNC_STRATEGY"

But when I do it via GUI, it finished successfully.

I am trying to automate this whole process with staging push (staging db restore + snapsync) as we need to maintain the staging db restores.

Can some one help resolve this error while trying to use dxtoolkit command?

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc
------------------------------

Hi Neal,

Thank you for your response. So let me be clear on what I am trying to do.

Ours is a SQL Server platform and we use Rubrik as our backup provider. As delphix does not support Rubrik backup provider, I am using staging push method to configure the dSource:

https://docs.delphix.com/docs/datasets/sql-server-environments-and-data-sources/linking-data-sources-and-syncing-data-with-sql-server/staging-push-implementation-for-sql-server

Staging Push gives end-users control over some Staging DB processes so that the nuanced, staging-based ingestion can be orchestrated externally. It will give control of the staging database to the end-user to pull from any backup provider (as a part of this, you'll be responsible for keeping the Staging DB in sync with the production database).

So, I am responsible for restoring the staging db with the latest backup that's available and keep it in sync with the source database.
I am trying to automate this whole process and came up with a script to do the staging db restore with the latest backup and the next step would be to run a snap-sync of the dSource after the staging db has been restored with the latest backup.

I am using dxtool kit dx-snapshot-db command to achieve the 2nd part (to run a snap-sync of the dSource after the staging db has been restored with the latest backup).

Is that the right command to do the snap-sync of the dSource? if not, what's the right command to use?

Please let me know if my explanation is clear or not.

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc

Original Message:
Sent: 05-01-2023 10:55:32 AM
From: Neal Stack
Subject: Staging Push snapsync dxtool command error

Hello,

I see two errors.

1. There is a mention of increasing the timeout for the command. Can you edit your "dxtools.conf" configuration file and increase the timeout?
% grep -i timeout dxtools.conf
            "timeout" : "60"

2. It mentions that a new, copy only backup is invalid for a staging push dSource. If you use the "--help" parameter, it gives you some details on the other backup types:
    -usebackup yes|no For MS SQL / Sybase dSource only - if this flag is set
    to yes - snapshot will to loaded from last known backup, if flag is set
    to no Delphix Engine will run full backup Default value is no
    -backupfileslist backupfile1,backupfile2,... For Sybase dSource only -
    specify a list of backup files as a list of comma separated backup file
    names
    -backupfilesfile /path/to/file_with_backup For Sybase dSource only -
    specify a file contains a list of backup files. One file per line
    -backupuuid uuid For MS SQL only - UUID of backup to ingest

So you probably want to use the "-usebackup yes" parameter so that it does not try to initiate a copy only backup. Then you have to decide if you will provide a backupset UUID (-backupuuid) or the names of the backup files you want to restore (-backupfileslist).

I hope that helps,
Neal

------------------------------
Neal Stack
Senior Principal Technical Support Engineer
Delphix

Original Message:
Sent: 05-01-2023 10:46:09 AM
From: Pratyusha B
Subject: Staging Push snapsync dxtool command error

Hi,

I have a dSource that was setup using Staging Push (SQL Server) as we use Rubrik as a backup provider for our db's. I was able to restore the staging db using an automated script and also tried to run the snapsync using dxtoolkit command after the staging db restore is complete.

But I get the below error when trying to do a snapync using dxtoolkit command:

c:\dxtoolkit2>dx_snapshot_db -d delphixservername -name DBName
No data returned for resources/json/delphix/sourceconfig. Try to increase timeout
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Starting job JOB-300 for database DBName.
0
Job JOB-300 finished with state: FAILED
Last message is: Sync parameter type "MSSQL_NEW_COPY_ONLY_FULL_BACKUP_SYNC_PARAMETERS" is not supported for sync strategy type "MSSQL_STAGING_PUSH_SYNC_STRATEGY"

But when I do it via GUI, it finished successfully.

I am trying to automate this whole process with staging push (staging db restore + snapsync) as we need to maintain the staging db restores.

Can some one help resolve this error while trying to use dxtoolkit command?

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc
------------------------------

Hello Pratyusha,

Thanks for the explanation, I am familiar with Staging Push.

I've reproduced your issue internally and I am checking with the developers ( https://github.com/delphix/dxtoolkit/issues ). We may have to raise a new feature request on GitHub to add support for snapshotting Staging Push dSources.

I'll reply back when I hear from them.

Thanks,
Neal

Hi Neal,

Thank you for your response. So let me be clear on what I am trying to do.

Ours is a SQL Server platform and we use Rubrik as our backup provider. As delphix does not support Rubrik backup provider, I am using staging push method to configure the dSource:

https://docs.delphix.com/docs/datasets/sql-server-environments-and-data-sources/linking-data-sources-and-syncing-data-with-sql-server/staging-push-implementation-for-sql-server

Staging Push gives end-users control over some Staging DB processes so that the nuanced, staging-based ingestion can be orchestrated externally. It will give control of the staging database to the end-user to pull from any backup provider (as a part of this, you'll be responsible for keeping the Staging DB in sync with the production database).

So, I am responsible for restoring the staging db with the latest backup that's available and keep it in sync with the source database.
I am trying to automate this whole process and came up with a script to do the staging db restore with the latest backup and the next step would be to run a snap-sync of the dSource after the staging db has been restored with the latest backup.

I am using dxtool kit dx-snapshot-db command to achieve the 2nd part (to run a snap-sync of the dSource after the staging db has been restored with the latest backup).

Is that the right command to do the snap-sync of the dSource? if not, what's the right command to use?

Please let me know if my explanation is clear or not.

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc

Original Message:
Sent: 05-01-2023 10:55:32 AM
From: Neal Stack
Subject: Staging Push snapsync dxtool command error

Hello,

I see two errors.

1. There is a mention of increasing the timeout for the command. Can you edit your "dxtools.conf" configuration file and increase the timeout?
% grep -i timeout dxtools.conf
            "timeout" : "60"

2. It mentions that a new, copy only backup is invalid for a staging push dSource. If you use the "--help" parameter, it gives you some details on the other backup types:
    -usebackup yes|no For MS SQL / Sybase dSource only - if this flag is set
    to yes - snapshot will to loaded from last known backup, if flag is set
    to no Delphix Engine will run full backup Default value is no
    -backupfileslist backupfile1,backupfile2,... For Sybase dSource only -
    specify a list of backup files as a list of comma separated backup file
    names
    -backupfilesfile /path/to/file_with_backup For Sybase dSource only -
    specify a file contains a list of backup files. One file per line
    -backupuuid uuid For MS SQL only - UUID of backup to ingest

So you probably want to use the "-usebackup yes" parameter so that it does not try to initiate a copy only backup. Then you have to decide if you will provide a backupset UUID (-backupuuid) or the names of the backup files you want to restore (-backupfileslist).

I hope that helps,
Neal

------------------------------
Neal Stack
Senior Principal Technical Support Engineer
Delphix

Original Message:
Sent: 05-01-2023 10:46:09 AM
From: Pratyusha B
Subject: Staging Push snapsync dxtool command error

Hi,

I have a dSource that was setup using Staging Push (SQL Server) as we use Rubrik as a backup provider for our db's. I was able to restore the staging db using an automated script and also tried to run the snapsync using dxtoolkit command after the staging db restore is complete.

But I get the below error when trying to do a snapync using dxtoolkit command:

c:\dxtoolkit2>dx_snapshot_db -d delphixservername -name DBName
No data returned for resources/json/delphix/sourceconfig. Try to increase timeout
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Starting job JOB-300 for database DBName.
0
Job JOB-300 finished with state: FAILED
Last message is: Sync parameter type "MSSQL_NEW_COPY_ONLY_FULL_BACKUP_SYNC_PARAMETERS" is not supported for sync strategy type "MSSQL_STAGING_PUSH_SYNC_STRATEGY"

But when I do it via GUI, it finished successfully.

I am trying to automate this whole process with staging push (staging db restore + snapsync) as we need to maintain the staging db restores.

Can some one help resolve this error while trying to use dxtoolkit command?

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc
------------------------------

Hi Neal,

Thanks for looking into this. We really need this working as we are in the process of automating the entire process (Staging db refresh + SnapSync + VDB Refresh). We are currently stuck at 2nd step (SnapSync using DxToolKit). Any help would be appreciated.

Thanks.

Hello Pratyusha,

Thanks for the explanation, I am familiar with Staging Push.

I've reproduced your issue internally and I am checking with the developers ( https://github.com/delphix/dxtoolkit/issues ). We may have to raise a new feature request on GitHub to add support for snapshotting Staging Push dSources.

I'll reply back when I hear from them.

Thanks,
Neal

Hi Neal,

Thank you for your response. So let me be clear on what I am trying to do.

Ours is a SQL Server platform and we use Rubrik as our backup provider. As delphix does not support Rubrik backup provider, I am using staging push method to configure the dSource:

https://docs.delphix.com/docs/datasets/sql-server-environments-and-data-sources/linking-data-sources-and-syncing-data-with-sql-server/staging-push-implementation-for-sql-server

Staging Push gives end-users control over some Staging DB processes so that the nuanced, staging-based ingestion can be orchestrated externally. It will give control of the staging database to the end-user to pull from any backup provider (as a part of this, you'll be responsible for keeping the Staging DB in sync with the production database).

So, I am responsible for restoring the staging db with the latest backup that's available and keep it in sync with the source database.
I am trying to automate this whole process and came up with a script to do the staging db restore with the latest backup and the next step would be to run a snap-sync of the dSource after the staging db has been restored with the latest backup.

I am using dxtool kit dx-snapshot-db command to achieve the 2nd part (to run a snap-sync of the dSource after the staging db has been restored with the latest backup).

Is that the right command to do the snap-sync of the dSource? if not, what's the right command to use?

Please let me know if my explanation is clear or not.

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc

Original Message:
Sent: 05-01-2023 10:55:32 AM
From: Neal Stack
Subject: Staging Push snapsync dxtool command error

Hello,

I see two errors.

1. There is a mention of increasing the timeout for the command. Can you edit your "dxtools.conf" configuration file and increase the timeout?
% grep -i timeout dxtools.conf
            "timeout" : "60"

2. It mentions that a new, copy only backup is invalid for a staging push dSource. If you use the "--help" parameter, it gives you some details on the other backup types:
    -usebackup yes|no For MS SQL / Sybase dSource only - if this flag is set
    to yes - snapshot will to loaded from last known backup, if flag is set
    to no Delphix Engine will run full backup Default value is no
    -backupfileslist backupfile1,backupfile2,... For Sybase dSource only -
    specify a list of backup files as a list of comma separated backup file
    names
    -backupfilesfile /path/to/file_with_backup For Sybase dSource only -
    specify a file contains a list of backup files. One file per line
    -backupuuid uuid For MS SQL only - UUID of backup to ingest

So you probably want to use the "-usebackup yes" parameter so that it does not try to initiate a copy only backup. Then you have to decide if you will provide a backupset UUID (-backupuuid) or the names of the backup files you want to restore (-backupfileslist).

I hope that helps,
Neal

------------------------------
Neal Stack
Senior Principal Technical Support Engineer
Delphix

Original Message:
Sent: 05-01-2023 10:46:09 AM
From: Pratyusha B
Subject: Staging Push snapsync dxtool command error

Hi,

I have a dSource that was setup using Staging Push (SQL Server) as we use Rubrik as a backup provider for our db's. I was able to restore the staging db using an automated script and also tried to run the snapsync using dxtoolkit command after the staging db restore is complete.

But I get the below error when trying to do a snapync using dxtoolkit command:

c:\dxtoolkit2>dx_snapshot_db -d delphixservername -name DBName
No data returned for resources/json/delphix/sourceconfig. Try to increase timeout
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Starting job JOB-300 for database DBName.
0
Job JOB-300 finished with state: FAILED
Last message is: Sync parameter type "MSSQL_NEW_COPY_ONLY_FULL_BACKUP_SYNC_PARAMETERS" is not supported for sync strategy type "MSSQL_STAGING_PUSH_SYNC_STRATEGY"

But when I do it via GUI, it finished successfully.

I am trying to automate this whole process with staging push (staging db restore + snapsync) as we need to maintain the staging db restores.

Can some one help resolve this error while trying to use dxtoolkit command?

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc
------------------------------

Hello Pratyusha,

I heard back from the dxtoolkit developers and dxtoolkit has not been tested with Staging Push at all so you would need to raise a new feature request here:

https://github.com/delphix/dxtoolkit/issues

Thanks,
Neal

Hi Neal,

Thanks for looking into this. We really need this working as we are in the process of automating the entire process (Staging db refresh + SnapSync + VDB Refresh). We are currently stuck at 2nd step (SnapSync using DxToolKit). Any help would be appreciated.

Thanks.

Hello Pratyusha,

Thanks for the explanation, I am familiar with Staging Push.

I've reproduced your issue internally and I am checking with the developers ( https://github.com/delphix/dxtoolkit/issues ). We may have to raise a new feature request on GitHub to add support for snapshotting Staging Push dSources.

I'll reply back when I hear from them.

Thanks,
Neal

Hi Neal,

Thank you for your response. So let me be clear on what I am trying to do.

Ours is a SQL Server platform and we use Rubrik as our backup provider. As delphix does not support Rubrik backup provider, I am using staging push method to configure the dSource:

https://docs.delphix.com/docs/datasets/sql-server-environments-and-data-sources/linking-data-sources-and-syncing-data-with-sql-server/staging-push-implementation-for-sql-server

Staging Push gives end-users control over some Staging DB processes so that the nuanced, staging-based ingestion can be orchestrated externally. It will give control of the staging database to the end-user to pull from any backup provider (as a part of this, you'll be responsible for keeping the Staging DB in sync with the production database).

So, I am responsible for restoring the staging db with the latest backup that's available and keep it in sync with the source database.
I am trying to automate this whole process and came up with a script to do the staging db restore with the latest backup and the next step would be to run a snap-sync of the dSource after the staging db has been restored with the latest backup.

I am using dxtool kit dx-snapshot-db command to achieve the 2nd part (to run a snap-sync of the dSource after the staging db has been restored with the latest backup).

Is that the right command to do the snap-sync of the dSource? if not, what's the right command to use?

Please let me know if my explanation is clear or not.

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc

Original Message:
Sent: 05-01-2023 10:55:32 AM
From: Neal Stack
Subject: Staging Push snapsync dxtool command error

Hello,

I see two errors.

1. There is a mention of increasing the timeout for the command. Can you edit your "dxtools.conf" configuration file and increase the timeout?
% grep -i timeout dxtools.conf
            "timeout" : "60"

2. It mentions that a new, copy only backup is invalid for a staging push dSource. If you use the "--help" parameter, it gives you some details on the other backup types:
    -usebackup yes|no For MS SQL / Sybase dSource only - if this flag is set
    to yes - snapshot will to loaded from last known backup, if flag is set
    to no Delphix Engine will run full backup Default value is no
    -backupfileslist backupfile1,backupfile2,... For Sybase dSource only -
    specify a list of backup files as a list of comma separated backup file
    names
    -backupfilesfile /path/to/file_with_backup For Sybase dSource only -
    specify a file contains a list of backup files. One file per line
    -backupuuid uuid For MS SQL only - UUID of backup to ingest

So you probably want to use the "-usebackup yes" parameter so that it does not try to initiate a copy only backup. Then you have to decide if you will provide a backupset UUID (-backupuuid) or the names of the backup files you want to restore (-backupfileslist).

I hope that helps,
Neal

------------------------------
Neal Stack
Senior Principal Technical Support Engineer
Delphix

Original Message:
Sent: 05-01-2023 10:46:09 AM
From: Pratyusha B
Subject: Staging Push snapsync dxtool command error

Hi,

I have a dSource that was setup using Staging Push (SQL Server) as we use Rubrik as a backup provider for our db's. I was able to restore the staging db using an automated script and also tried to run the snapsync using dxtoolkit command after the staging db restore is complete.

But I get the below error when trying to do a snapync using dxtoolkit command:

c:\dxtoolkit2>dx_snapshot_db -d delphixservername -name DBName
No data returned for resources/json/delphix/sourceconfig. Try to increase timeout
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Starting job JOB-300 for database DBName.
0
Job JOB-300 finished with state: FAILED
Last message is: Sync parameter type "MSSQL_NEW_COPY_ONLY_FULL_BACKUP_SYNC_PARAMETERS" is not supported for sync strategy type "MSSQL_STAGING_PUSH_SYNC_STRATEGY"

But when I do it via GUI, it finished successfully.

I am trying to automate this whole process with staging push (staging db restore + snapsync) as we need to maintain the staging db restores.

Can some one help resolve this error while trying to use dxtoolkit command?

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc
------------------------------

Hi Neal,

Thank you for your response. Meanwhile, is there any workaround for this? We have around 50 db's that we are going to implement staging push for and taking the snapshot manually via GUI after every staging db restore is going to be a tedious process. Just curious, will delphix automatically take the snapshot after the staging db restore?

The following link has this:

Staging Push Implementation for SQL Server (delphix.com)

SnapSync Criteria

• Delphix tries to fetch the last restored backup. If no backup is found, the snapshot is skipped with a warning.

• If a snapshot already exists for the backup in the current timeflow, the snapshot is skipped with a warning.
  
  The first point says delphix tries to fetch the last restored backup. Does that mean it will take the snapshot once the staging db has been restored with a new backup?
  
  I am trying to see if it will, but don't see it doing in the UI.
  
  Please let me know.
  
  Thanks.

Hello Pratyusha,

I heard back from the dxtoolkit developers and dxtoolkit has not been tested with Staging Push at all so you would need to raise a new feature request here:

https://github.com/delphix/dxtoolkit/issues

Thanks,
Neal

Hi Neal,

Thanks for looking into this. We really need this working as we are in the process of automating the entire process (Staging db refresh + SnapSync + VDB Refresh). We are currently stuck at 2nd step (SnapSync using DxToolKit). Any help would be appreciated.

Thanks.

Hello Pratyusha,

Thanks for the explanation, I am familiar with Staging Push.

I've reproduced your issue internally and I am checking with the developers ( https://github.com/delphix/dxtoolkit/issues ). We may have to raise a new feature request on GitHub to add support for snapshotting Staging Push dSources.

I'll reply back when I hear from them.

Thanks,
Neal

Hi Neal,

Thank you for your response. So let me be clear on what I am trying to do.

Ours is a SQL Server platform and we use Rubrik as our backup provider. As delphix does not support Rubrik backup provider, I am using staging push method to configure the dSource:

https://docs.delphix.com/docs/datasets/sql-server-environments-and-data-sources/linking-data-sources-and-syncing-data-with-sql-server/staging-push-implementation-for-sql-server

Staging Push gives end-users control over some Staging DB processes so that the nuanced, staging-based ingestion can be orchestrated externally. It will give control of the staging database to the end-user to pull from any backup provider (as a part of this, you'll be responsible for keeping the Staging DB in sync with the production database).

So, I am responsible for restoring the staging db with the latest backup that's available and keep it in sync with the source database.
I am trying to automate this whole process and came up with a script to do the staging db restore with the latest backup and the next step would be to run a snap-sync of the dSource after the staging db has been restored with the latest backup.

I am using dxtool kit dx-snapshot-db command to achieve the 2nd part (to run a snap-sync of the dSource after the staging db has been restored with the latest backup).

Is that the right command to do the snap-sync of the dSource? if not, what's the right command to use?

Please let me know if my explanation is clear or not.

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc

Original Message:
Sent: 05-01-2023 10:55:32 AM
From: Neal Stack
Subject: Staging Push snapsync dxtool command error

Hello,

I see two errors.

1. There is a mention of increasing the timeout for the command. Can you edit your "dxtools.conf" configuration file and increase the timeout?
% grep -i timeout dxtools.conf
            "timeout" : "60"

2. It mentions that a new, copy only backup is invalid for a staging push dSource. If you use the "--help" parameter, it gives you some details on the other backup types:
    -usebackup yes|no For MS SQL / Sybase dSource only - if this flag is set
    to yes - snapshot will to loaded from last known backup, if flag is set
    to no Delphix Engine will run full backup Default value is no
    -backupfileslist backupfile1,backupfile2,... For Sybase dSource only -
    specify a list of backup files as a list of comma separated backup file
    names
    -backupfilesfile /path/to/file_with_backup For Sybase dSource only -
    specify a file contains a list of backup files. One file per line
    -backupuuid uuid For MS SQL only - UUID of backup to ingest

So you probably want to use the "-usebackup yes" parameter so that it does not try to initiate a copy only backup. Then you have to decide if you will provide a backupset UUID (-backupuuid) or the names of the backup files you want to restore (-backupfileslist).

I hope that helps,
Neal

------------------------------
Neal Stack
Senior Principal Technical Support Engineer
Delphix

Original Message:
Sent: 05-01-2023 10:46:09 AM
From: Pratyusha B
Subject: Staging Push snapsync dxtool command error

Hi,

I have a dSource that was setup using Staging Push (SQL Server) as we use Rubrik as a backup provider for our db's. I was able to restore the staging db using an automated script and also tried to run the snapsync using dxtoolkit command after the staging db restore is complete.

But I get the below error when trying to do a snapync using dxtoolkit command:

c:\dxtoolkit2>dx_snapshot_db -d delphixservername -name DBName
No data returned for resources/json/delphix/sourceconfig. Try to increase timeout
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Starting job JOB-300 for database DBName.
0
Job JOB-300 finished with state: FAILED
Last message is: Sync parameter type "MSSQL_NEW_COPY_ONLY_FULL_BACKUP_SYNC_PARAMETERS" is not supported for sync strategy type "MSSQL_STAGING_PUSH_SYNC_STRATEGY"

But when I do it via GUI, it finished successfully.

I am trying to automate this whole process with staging push (staging db restore + snapsync) as we need to maintain the staging db restores.

Can some one help resolve this error while trying to use dxtoolkit command?

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc
------------------------------

Hello,

I will file a documentation bug to clarify that. Where it says "Delphix tries to fetch the last restored backup. If no backup is found, the snapshot is skipped with a warning.", is confusing. I think they may have taken that from regular dSources where Delphix is querying the SQL Server backup sets from the source database and ingesting them.

For staging push, Delphix does not ingest the backups automatically. It leaves that to end-user control:
> Staging Push gives end-users control over some Staging DB processes so that the nuanced, staging-based ingestion can be orchestrated externally. It will give control of the staging database to the end-user to pull from any backup provider (as a part of this, you'll be responsible for keeping the Staging DB in sync with the production database).

You can script this using our command line interface (CLI):
% echo "/database; select McLaren_Staging; sync; set type=MSSqlNoBackupSyncParameters; commit" | ssh admin@neal6017.dcol2.delphix.com
Pseudo-terminal will not be allocated because stdin is not a terminal.
(admin@neal6017.dcol2.delphix.com) Password: 
    Dispatched job JOB-2076
    DB_SYNC job started for "MSSQL/McLaren_Staging".
    Starting SnapSync for dSource "McLaren_Staging".
    Validating the staging database "McLaren_Staging" for dSource "McLaren_Staging" for sync.
    Taking a temporary snapshot for the dSource "McLaren_Staging".
    Validating and finalizing temporary snapshot for dSource "McLaren_Staging".
    Finalized and successfully created snapshot for dSource "McLaren_Staging".
    SnapSync for dSource "McLaren_Staging" completed successfully.
    DB_SYNC job for "MSSQL/McLaren_Staging" completed successfully.

When you use the CLI, you can enable tracing (setopt trace=true) to see what the underlying REST API calls are. For example:
neal6017 database 'McLaren_Staging' sync *> setopt trace=true 
neal6017 database 'McLaren_Staging' sync *> commit
=== POST /resources/json/delphix/database/MSSQL_DB_CONTAINER-136/sync ===
{
    "type": "MSSqlNoBackupSyncParameters"
}
=== RESPONSE ===
{
    "type": "OKResult",
    "status": "OK",
    "result": "",
    "job": "JOB-2077",
    "action": "ACTION-3902"
}
=== END ===
    Dispatched job JOB-2077

If you are comfortable with REST API programming, you can use Curl and we have many examples in our documentation:
* https://documentation.delphix.com/continuous-data-10-0-0-0/docs/web-service-protocol
* https://documentation.delphix.com/continuous-data-10-0-0-0/docs/api-cookbook-common-tasks-workflows-and-examples

Thanks,
Neal

Hi Neal,

Thank you for your response. Meanwhile, is there any workaround for this? We have around 50 db's that we are going to implement staging push for and taking the snapshot manually via GUI after every staging db restore is going to be a tedious process. Just curious, will delphix automatically take the snapshot after the staging db restore?

The following link has this:

Staging Push Implementation for SQL Server (delphix.com)

SnapSync Criteria

• Delphix tries to fetch the last restored backup. If no backup is found, the snapshot is skipped with a warning.

• If a snapshot already exists for the backup in the current timeflow, the snapshot is skipped with a warning.
  
  The first point says delphix tries to fetch the last restored backup. Does that mean it will take the snapshot once the staging db has been restored with a new backup?
  
  I am trying to see if it will, but don't see it doing in the UI.
  
  Please let me know.
  
  Thanks.

Hello Pratyusha,

I heard back from the dxtoolkit developers and dxtoolkit has not been tested with Staging Push at all so you would need to raise a new feature request here:

https://github.com/delphix/dxtoolkit/issues

Thanks,
Neal

Hi Neal,

Thanks for looking into this. We really need this working as we are in the process of automating the entire process (Staging db refresh + SnapSync + VDB Refresh). We are currently stuck at 2nd step (SnapSync using DxToolKit). Any help would be appreciated.

Thanks.

Hello Pratyusha,

Thanks for the explanation, I am familiar with Staging Push.

I've reproduced your issue internally and I am checking with the developers ( https://github.com/delphix/dxtoolkit/issues ). We may have to raise a new feature request on GitHub to add support for snapshotting Staging Push dSources.

I'll reply back when I hear from them.

Thanks,
Neal

Hi Neal,

Thank you for your response. So let me be clear on what I am trying to do.

Ours is a SQL Server platform and we use Rubrik as our backup provider. As delphix does not support Rubrik backup provider, I am using staging push method to configure the dSource:

https://docs.delphix.com/docs/datasets/sql-server-environments-and-data-sources/linking-data-sources-and-syncing-data-with-sql-server/staging-push-implementation-for-sql-server

Staging Push gives end-users control over some Staging DB processes so that the nuanced, staging-based ingestion can be orchestrated externally. It will give control of the staging database to the end-user to pull from any backup provider (as a part of this, you'll be responsible for keeping the Staging DB in sync with the production database).

So, I am responsible for restoring the staging db with the latest backup that's available and keep it in sync with the source database.
I am trying to automate this whole process and came up with a script to do the staging db restore with the latest backup and the next step would be to run a snap-sync of the dSource after the staging db has been restored with the latest backup.

I am using dxtool kit dx-snapshot-db command to achieve the 2nd part (to run a snap-sync of the dSource after the staging db has been restored with the latest backup).

Is that the right command to do the snap-sync of the dSource? if not, what's the right command to use?

Please let me know if my explanation is clear or not.

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc

Original Message:
Sent: 05-01-2023 10:55:32 AM
From: Neal Stack
Subject: Staging Push snapsync dxtool command error

Hello,

I see two errors.

1. There is a mention of increasing the timeout for the command. Can you edit your "dxtools.conf" configuration file and increase the timeout?
% grep -i timeout dxtools.conf
            "timeout" : "60"

2. It mentions that a new, copy only backup is invalid for a staging push dSource. If you use the "--help" parameter, it gives you some details on the other backup types:
    -usebackup yes|no For MS SQL / Sybase dSource only - if this flag is set
    to yes - snapshot will to loaded from last known backup, if flag is set
    to no Delphix Engine will run full backup Default value is no
    -backupfileslist backupfile1,backupfile2,... For Sybase dSource only -
    specify a list of backup files as a list of comma separated backup file
    names
    -backupfilesfile /path/to/file_with_backup For Sybase dSource only -
    specify a file contains a list of backup files. One file per line
    -backupuuid uuid For MS SQL only - UUID of backup to ingest

So you probably want to use the "-usebackup yes" parameter so that it does not try to initiate a copy only backup. Then you have to decide if you will provide a backupset UUID (-backupuuid) or the names of the backup files you want to restore (-backupfileslist).

I hope that helps,
Neal

------------------------------
Neal Stack
Senior Principal Technical Support Engineer
Delphix

Original Message:
Sent: 05-01-2023 10:46:09 AM
From: Pratyusha B
Subject: Staging Push snapsync dxtool command error

Hi,

I have a dSource that was setup using Staging Push (SQL Server) as we use Rubrik as a backup provider for our db's. I was able to restore the staging db using an automated script and also tried to run the snapsync using dxtoolkit command after the staging db restore is complete.

But I get the below error when trying to do a snapync using dxtoolkit command:

c:\dxtoolkit2>dx_snapshot_db -d delphixservername -name DBName
No data returned for resources/json/delphix/sourceconfig. Try to increase timeout
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Starting job JOB-300 for database DBName.
0
Job JOB-300 finished with state: FAILED
Last message is: Sync parameter type "MSSQL_NEW_COPY_ONLY_FULL_BACKUP_SYNC_PARAMETERS" is not supported for sync strategy type "MSSQL_STAGING_PUSH_SYNC_STRATEGY"

But when I do it via GUI, it finished successfully.

I am trying to automate this whole process with staging push (staging db restore + snapsync) as we need to maintain the staging db restores.

Can some one help resolve this error while trying to use dxtoolkit command?

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc
------------------------------

Hi Neal,

Yes, I am aware that with Staging Push method, we are responsible for staging db restores. Per the documentation, I thought it will take the snapshot too once the staging db has been refreshed with the latest backup. But looks like we have to explicitly initiate the snapshot of the dSource once the staging db has been refreshed. Please confirm this.

Regarding CLI, are you saying instead of using dxtoolkit, we can also use CLI to initiate the snapshot of the dSource?

Hello,

I will file a documentation bug to clarify that. Where it says "Delphix tries to fetch the last restored backup. If no backup is found, the snapshot is skipped with a warning.", is confusing. I think they may have taken that from regular dSources where Delphix is querying the SQL Server backup sets from the source database and ingesting them.

For staging push, Delphix does not ingest the backups automatically. It leaves that to end-user control:
> Staging Push gives end-users control over some Staging DB processes so that the nuanced, staging-based ingestion can be orchestrated externally. It will give control of the staging database to the end-user to pull from any backup provider (as a part of this, you'll be responsible for keeping the Staging DB in sync with the production database).

You can script this using our command line interface (CLI):
% echo "/database; select McLaren_Staging; sync; set type=MSSqlNoBackupSyncParameters; commit" | ssh admin@neal6017.dcol2.delphix.com
Pseudo-terminal will not be allocated because stdin is not a terminal.
(admin@neal6017.dcol2.delphix.com) Password: 
    Dispatched job JOB-2076
    DB_SYNC job started for "MSSQL/McLaren_Staging".
    Starting SnapSync for dSource "McLaren_Staging".
    Validating the staging database "McLaren_Staging" for dSource "McLaren_Staging" for sync.
    Taking a temporary snapshot for the dSource "McLaren_Staging".
    Validating and finalizing temporary snapshot for dSource "McLaren_Staging".
    Finalized and successfully created snapshot for dSource "McLaren_Staging".
    SnapSync for dSource "McLaren_Staging" completed successfully.
    DB_SYNC job for "MSSQL/McLaren_Staging" completed successfully.

When you use the CLI, you can enable tracing (setopt trace=true) to see what the underlying REST API calls are. For example:
neal6017 database 'McLaren_Staging' sync *> setopt trace=true 
neal6017 database 'McLaren_Staging' sync *> commit
=== POST /resources/json/delphix/database/MSSQL_DB_CONTAINER-136/sync ===
{
    "type": "MSSqlNoBackupSyncParameters"
}
=== RESPONSE ===
{
    "type": "OKResult",
    "status": "OK",
    "result": "",
    "job": "JOB-2077",
    "action": "ACTION-3902"
}
=== END ===
    Dispatched job JOB-2077

If you are comfortable with REST API programming, you can use Curl and we have many examples in our documentation:
* https://documentation.delphix.com/continuous-data-10-0-0-0/docs/web-service-protocol
* https://documentation.delphix.com/continuous-data-10-0-0-0/docs/api-cookbook-common-tasks-workflows-and-examples

Thanks,
Neal

Hi Neal,

Thank you for your response. Meanwhile, is there any workaround for this? We have around 50 db's that we are going to implement staging push for and taking the snapshot manually via GUI after every staging db restore is going to be a tedious process. Just curious, will delphix automatically take the snapshot after the staging db restore?

The following link has this:

Staging Push Implementation for SQL Server (delphix.com)

SnapSync Criteria

• Delphix tries to fetch the last restored backup. If no backup is found, the snapshot is skipped with a warning.

• If a snapshot already exists for the backup in the current timeflow, the snapshot is skipped with a warning.
  
  The first point says delphix tries to fetch the last restored backup. Does that mean it will take the snapshot once the staging db has been restored with a new backup?
  
  I am trying to see if it will, but don't see it doing in the UI.
  
  Please let me know.
  
  Thanks.

Hello Pratyusha,

I heard back from the dxtoolkit developers and dxtoolkit has not been tested with Staging Push at all so you would need to raise a new feature request here:

https://github.com/delphix/dxtoolkit/issues

Thanks,
Neal

Hi Neal,

Thanks for looking into this. We really need this working as we are in the process of automating the entire process (Staging db refresh + SnapSync + VDB Refresh). We are currently stuck at 2nd step (SnapSync using DxToolKit). Any help would be appreciated.

Thanks.

Hello Pratyusha,

Thanks for the explanation, I am familiar with Staging Push.

I've reproduced your issue internally and I am checking with the developers ( https://github.com/delphix/dxtoolkit/issues ). We may have to raise a new feature request on GitHub to add support for snapshotting Staging Push dSources.

I'll reply back when I hear from them.

Thanks,
Neal

Hi Neal,

Thank you for your response. So let me be clear on what I am trying to do.

Ours is a SQL Server platform and we use Rubrik as our backup provider. As delphix does not support Rubrik backup provider, I am using staging push method to configure the dSource:

https://docs.delphix.com/docs/datasets/sql-server-environments-and-data-sources/linking-data-sources-and-syncing-data-with-sql-server/staging-push-implementation-for-sql-server

Staging Push gives end-users control over some Staging DB processes so that the nuanced, staging-based ingestion can be orchestrated externally. It will give control of the staging database to the end-user to pull from any backup provider (as a part of this, you'll be responsible for keeping the Staging DB in sync with the production database).

So, I am responsible for restoring the staging db with the latest backup that's available and keep it in sync with the source database.
I am trying to automate this whole process and came up with a script to do the staging db restore with the latest backup and the next step would be to run a snap-sync of the dSource after the staging db has been restored with the latest backup.

I am using dxtool kit dx-snapshot-db command to achieve the 2nd part (to run a snap-sync of the dSource after the staging db has been restored with the latest backup).

Is that the right command to do the snap-sync of the dSource? if not, what's the right command to use?

Please let me know if my explanation is clear or not.

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc

Original Message:
Sent: 05-01-2023 10:55:32 AM
From: Neal Stack
Subject: Staging Push snapsync dxtool command error

Hello,

I see two errors.

1. There is a mention of increasing the timeout for the command. Can you edit your "dxtools.conf" configuration file and increase the timeout?
% grep -i timeout dxtools.conf
            "timeout" : "60"

2. It mentions that a new, copy only backup is invalid for a staging push dSource. If you use the "--help" parameter, it gives you some details on the other backup types:
    -usebackup yes|no For MS SQL / Sybase dSource only - if this flag is set
    to yes - snapshot will to loaded from last known backup, if flag is set
    to no Delphix Engine will run full backup Default value is no
    -backupfileslist backupfile1,backupfile2,... For Sybase dSource only -
    specify a list of backup files as a list of comma separated backup file
    names
    -backupfilesfile /path/to/file_with_backup For Sybase dSource only -
    specify a file contains a list of backup files. One file per line
    -backupuuid uuid For MS SQL only - UUID of backup to ingest

So you probably want to use the "-usebackup yes" parameter so that it does not try to initiate a copy only backup. Then you have to decide if you will provide a backupset UUID (-backupuuid) or the names of the backup files you want to restore (-backupfileslist).

I hope that helps,
Neal

------------------------------
Neal Stack
Senior Principal Technical Support Engineer
Delphix

Original Message:
Sent: 05-01-2023 10:46:09 AM
From: Pratyusha B
Subject: Staging Push snapsync dxtool command error

Hi,

I have a dSource that was setup using Staging Push (SQL Server) as we use Rubrik as a backup provider for our db's. I was able to restore the staging db using an automated script and also tried to run the snapsync using dxtoolkit command after the staging db restore is complete.

But I get the below error when trying to do a snapync using dxtoolkit command:

c:\dxtoolkit2>dx_snapshot_db -d delphixservername -name DBName
No data returned for resources/json/delphix/sourceconfig. Try to increase timeout
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Starting job JOB-300 for database DBName.
0
Job JOB-300 finished with state: FAILED
Last message is: Sync parameter type "MSSQL_NEW_COPY_ONLY_FULL_BACKUP_SYNC_PARAMETERS" is not supported for sync strategy type "MSSQL_STAGING_PUSH_SYNC_STRATEGY"

But when I do it via GUI, it finished successfully.

I am trying to automate this whole process with staging push (staging db restore + snapsync) as we need to maintain the staging db restores.

Can some one help resolve this error while trying to use dxtoolkit command?

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc
------------------------------

Hello,

With the Staging Push model, the end user is responsible for:

• Restoring the backup into the staging database.
• Restoring the backup into the staging database.
• Deciding whether to manually initiate a SnapSync or utilize a SnapSync policy. After a backup has been restored on the staging database, a SnapSync must be initiated to create a snapshot on the dSource's timeflow.
• Defining a pre-sync hook to restore the backup if you utilize a SnapSync policy.
• Defining a SnapSync policy to run after the source database is backed up (I know my source database's nightly backup completes by 1AM every night, I want Delphix to run a SnapSync job at 6AM every morning to restore this backup into the staging database).
  • Creating policies: https://documentation.delphix.com/docs/getting-started-managing-policies
  • Example hook which can be run in a pre-sync hook: https://documentation.delphix.com/continuous-data-10-0-0-0/docs/staging-push-implementation-for-sql-server?highlight=staging%20push#procedure1

If a SnapSync policy is not utilized, the end user is additionally responsible for initiating the SnapSync. This can be done by clicking the camera, using the CLI or rest APIs.

Hi Neal,

Yes, I am aware that with Staging Push method, we are responsible for staging db restores. Per the documentation, I thought it will take the snapshot too once the staging db has been refreshed with the latest backup. But looks like we have to explicitly initiate the snapshot of the dSource once the staging db has been refreshed. Please confirm this.

Regarding CLI, are you saying instead of using dxtoolkit, we can also use CLI to initiate the snapshot of the dSource?

Hello,

I will file a documentation bug to clarify that. Where it says "Delphix tries to fetch the last restored backup. If no backup is found, the snapshot is skipped with a warning.", is confusing. I think they may have taken that from regular dSources where Delphix is querying the SQL Server backup sets from the source database and ingesting them.

For staging push, Delphix does not ingest the backups automatically. It leaves that to end-user control:
> Staging Push gives end-users control over some Staging DB processes so that the nuanced, staging-based ingestion can be orchestrated externally. It will give control of the staging database to the end-user to pull from any backup provider (as a part of this, you'll be responsible for keeping the Staging DB in sync with the production database).

You can script this using our command line interface (CLI):
% echo "/database; select McLaren_Staging; sync; set type=MSSqlNoBackupSyncParameters; commit" | ssh admin@neal6017.dcol2.delphix.com
Pseudo-terminal will not be allocated because stdin is not a terminal.
(admin@neal6017.dcol2.delphix.com) Password: 
    Dispatched job JOB-2076
    DB_SYNC job started for "MSSQL/McLaren_Staging".
    Starting SnapSync for dSource "McLaren_Staging".
    Validating the staging database "McLaren_Staging" for dSource "McLaren_Staging" for sync.
    Taking a temporary snapshot for the dSource "McLaren_Staging".
    Validating and finalizing temporary snapshot for dSource "McLaren_Staging".
    Finalized and successfully created snapshot for dSource "McLaren_Staging".
    SnapSync for dSource "McLaren_Staging" completed successfully.
    DB_SYNC job for "MSSQL/McLaren_Staging" completed successfully.

When you use the CLI, you can enable tracing (setopt trace=true) to see what the underlying REST API calls are. For example:
neal6017 database 'McLaren_Staging' sync *> setopt trace=true 
neal6017 database 'McLaren_Staging' sync *> commit
=== POST /resources/json/delphix/database/MSSQL_DB_CONTAINER-136/sync ===
{
    "type": "MSSqlNoBackupSyncParameters"
}
=== RESPONSE ===
{
    "type": "OKResult",
    "status": "OK",
    "result": "",
    "job": "JOB-2077",
    "action": "ACTION-3902"
}
=== END ===
    Dispatched job JOB-2077

If you are comfortable with REST API programming, you can use Curl and we have many examples in our documentation:
* https://documentation.delphix.com/continuous-data-10-0-0-0/docs/web-service-protocol
* https://documentation.delphix.com/continuous-data-10-0-0-0/docs/api-cookbook-common-tasks-workflows-and-examples

Thanks,
Neal

Hi Neal,

Thank you for your response. Meanwhile, is there any workaround for this? We have around 50 db's that we are going to implement staging push for and taking the snapshot manually via GUI after every staging db restore is going to be a tedious process. Just curious, will delphix automatically take the snapshot after the staging db restore?

The following link has this:

Staging Push Implementation for SQL Server (delphix.com)

SnapSync Criteria

• Delphix tries to fetch the last restored backup. If no backup is found, the snapshot is skipped with a warning.

• If a snapshot already exists for the backup in the current timeflow, the snapshot is skipped with a warning.
  
  The first point says delphix tries to fetch the last restored backup. Does that mean it will take the snapshot once the staging db has been restored with a new backup?
  
  I am trying to see if it will, but don't see it doing in the UI.
  
  Please let me know.
  
  Thanks.

Hello Pratyusha,

I heard back from the dxtoolkit developers and dxtoolkit has not been tested with Staging Push at all so you would need to raise a new feature request here:

https://github.com/delphix/dxtoolkit/issues

Thanks,
Neal

Hi Neal,

Thanks for looking into this. We really need this working as we are in the process of automating the entire process (Staging db refresh + SnapSync + VDB Refresh). We are currently stuck at 2nd step (SnapSync using DxToolKit). Any help would be appreciated.

Thanks.

Hello Pratyusha,

Thanks for the explanation, I am familiar with Staging Push.

I've reproduced your issue internally and I am checking with the developers ( https://github.com/delphix/dxtoolkit/issues ). We may have to raise a new feature request on GitHub to add support for snapshotting Staging Push dSources.

I'll reply back when I hear from them.

Thanks,
Neal

Hi Neal,

Thank you for your response. So let me be clear on what I am trying to do.

Ours is a SQL Server platform and we use Rubrik as our backup provider. As delphix does not support Rubrik backup provider, I am using staging push method to configure the dSource:

https://docs.delphix.com/docs/datasets/sql-server-environments-and-data-sources/linking-data-sources-and-syncing-data-with-sql-server/staging-push-implementation-for-sql-server

Staging Push gives end-users control over some Staging DB processes so that the nuanced, staging-based ingestion can be orchestrated externally. It will give control of the staging database to the end-user to pull from any backup provider (as a part of this, you'll be responsible for keeping the Staging DB in sync with the production database).

So, I am responsible for restoring the staging db with the latest backup that's available and keep it in sync with the source database.
I am trying to automate this whole process and came up with a script to do the staging db restore with the latest backup and the next step would be to run a snap-sync of the dSource after the staging db has been restored with the latest backup.

I am using dxtool kit dx-snapshot-db command to achieve the 2nd part (to run a snap-sync of the dSource after the staging db has been restored with the latest backup).

Is that the right command to do the snap-sync of the dSource? if not, what's the right command to use?

Please let me know if my explanation is clear or not.

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc

Original Message:
Sent: 05-01-2023 10:55:32 AM
From: Neal Stack
Subject: Staging Push snapsync dxtool command error

Hello,

I see two errors.

1. There is a mention of increasing the timeout for the command. Can you edit your "dxtools.conf" configuration file and increase the timeout?
% grep -i timeout dxtools.conf
            "timeout" : "60"

2. It mentions that a new, copy only backup is invalid for a staging push dSource. If you use the "--help" parameter, it gives you some details on the other backup types:
    -usebackup yes|no For MS SQL / Sybase dSource only - if this flag is set
    to yes - snapshot will to loaded from last known backup, if flag is set
    to no Delphix Engine will run full backup Default value is no
    -backupfileslist backupfile1,backupfile2,... For Sybase dSource only -
    specify a list of backup files as a list of comma separated backup file
    names
    -backupfilesfile /path/to/file_with_backup For Sybase dSource only -
    specify a file contains a list of backup files. One file per line
    -backupuuid uuid For MS SQL only - UUID of backup to ingest

So you probably want to use the "-usebackup yes" parameter so that it does not try to initiate a copy only backup. Then you have to decide if you will provide a backupset UUID (-backupuuid) or the names of the backup files you want to restore (-backupfileslist).

I hope that helps,
Neal

------------------------------
Neal Stack
Senior Principal Technical Support Engineer
Delphix

Original Message:
Sent: 05-01-2023 10:46:09 AM
From: Pratyusha B
Subject: Staging Push snapsync dxtool command error

Hi,

I have a dSource that was setup using Staging Push (SQL Server) as we use Rubrik as a backup provider for our db's. I was able to restore the staging db using an automated script and also tried to run the snapsync using dxtoolkit command after the staging db restore is complete.

But I get the below error when trying to do a snapync using dxtoolkit command:

c:\dxtoolkit2>dx_snapshot_db -d delphixservername -name DBName
No data returned for resources/json/delphix/sourceconfig. Try to increase timeout
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Use of uninitialized value $repository in concatenation (.) or string at C:\Users\PBALUS~1.ADM\AppData\Local\Temp\par-7062616c7573752e61646d696e\cache-c7100440b54fc93e1dcae7bfcbfa53dea088df94\inc\lib/Databases.pm line 290.
Starting job JOB-300 for database DBName.
0
Job JOB-300 finished with state: FAILED
Last message is: Sync parameter type "MSSQL_NEW_COPY_ONLY_FULL_BACKUP_SYNC_PARAMETERS" is not supported for sync strategy type "MSSQL_STAGING_PUSH_SYNC_STRATEGY"

But when I do it via GUI, it finished successfully.

I am trying to automate this whole process with staging push (staging db restore + snapsync) as we need to maintain the staging db restores.

Can some one help resolve this error while trying to use dxtoolkit command?

------------------------------
Pratyusha B
Data Engineer SW DEV
Cotiviti Inc
------------------------------