API/Python/Open-Source Solutions

 View Only

Delphix DevOps demo with Vagrant and Chef

By michael.kuehl@delphix.com - posted 02-03-2016 03:28:08 PM


In preparation for a DevOps conference in Berlin last year, I started to work on a simple demo. The demo uses "Fat Free CRM", an open-source CRM application, to show how Delphix supports devops workflows like server provisioning, database reset/refresh and secure test data provisioning (using the Delphix Masking Engine). The simplest scenario is the provisioning of a ‘development environment’, adding a full virtual copy of the ‘production database' to the new server for development or testing with real data instead of sample data or a subset from production.

Since the initial version, I refactored the demo into a number of individual projects that are maintained separately but build upon each other. The original project is still around and available on Github .

Currently, two projects provide the foundation for the current demo:

  • delphix_devops_vagrant’, a Vagrantfile and a set of provisioning scripts to build demo environments with two databases (MySQL, Postgres), the sample application and all Delphix pre-requisites
  • delphix_rb’, a Ruby gem that interacts with the Delphix REST API

In addition to the above, there is also a Vagrant plug-in that registers a new virtual machine as an environment in Delphix and removes it from Delphix once the virtual machine is destroyed.

In this first article I will just cover the basics of the Vagrantfile and how to build the virtual machines used in the demo. Details can be found in the project’s README.md on Github.

To get started, a working installation of Vagrant and VirtualBox is required. VirtualBox is Vagrant’s default virtualization ‘provider’ and I use it to create and run the virtual machines that simulate the production- and development environment. The Delphix Engine however, is typically a virtual machine run by VMware. It is important that all three virtual machines share the same network on the host (my laptop ...) i.e. they have to be in the same IP range/subnet and they must share the same network interface. Since VMware has very view options how to configure networking (on the Mac at least), the Vagrantfile has options to adapt to this.

Once the Delphix Engine’s VM is up-and-running, Vagrant provisions the ‘production environment’ first. The first time the Vagrantfile is executed, Vagrant will search for and downloads the vitual machine image (CentOS 6.7) used to build the environments. When the virtual machine starts up, Vagrant executes ‘provisioners’ defined in the Vagrantfile in order to build the new environment.

The provisioning tasks

  • make sure all Delphix pre-reqisites are met
  • install and configure MySQL and Postgres
  • install and configure the CRM application
  • - register the environment in Delphix

Once the ‘production environment' is ready, we can log into the CRM application and view its data. The CRM application has scripts to create sample data to begin with and the demo uses this so we don’t have to enter records manually. The next step is to create a dSource from the CRM application’s database before starting the provisioning of the ‘development environment'. The dSource is used to create a VDB the CRM application on the ‘development environment' points to.

By now we have two environments running (source/target or production/development) that can be used to demonstrate the various Delphix features and capabilties.

What is next? The Vagrant/Delphix example shows how to build the environments used in other demos but does not automate any Delphix tasks besides registering the virtual machines in the Delphix Engine. A dedicated Devops/Chef demo builds upon this and shows how to automate more complex Delphix tasks like e.g. the creation of dSources or VDBs. In addition to this, I will follow-up with posts on how to use the Delphix Rubygem and the Vagrant plug-in. 

A new demo showing how to combine Docker and Delphix vFiles is also in the works!