Delphix Products

Automated Data Pipeline Pooling 

9 days ago

In this demo, @Diego Loureda shows how simple it is to integrate data in a pipeline. Historically, data has been heavy and difficult to move. This has caused friction and one place where we rarely see quality data is in CI/CD pipelines. We will review a simplified version of the day of a developer and a QA engineer using Delphix to add data to their pipelines, and not just old data or shared data. Using Self Service, we will assign one container to the pipeline and create others for final users, developers, and QA. This way, the pipeline can provide the data needed for any test or development phase, and also create bookmarks and share them with the other container users.

We will also review some advanced techniques because, in the real world, sometimes we find that pipelines may need to be executed hundreds of times a day or in parallel by many different users. For these advanced integrations, we created what we called the pooling approach. With this approach, Delphix will create an abstraction layer for the data, controlled by an external control file. This way, we will be able to provide to the pipelines the exact data they need (full dataset from production, masked copy of production, subset of production data, synthetic data, et cetera) at any point in time, and not adding any time to the pipeline execution time.
#video
#pooling
#CI/CD
#pipelines

Statistics
0 Favorited
5 Views
1 Files
0 Shares
1 Downloads
Attachment(s)
mp4 file
Automation Demo (DCC) - Pooling - 1080p.mp4   173.82MB   1 version
Uploaded - 10-12-2020