Improving Performance Testing with Agile Data
Application performance testing depends on a number of factors and can be dramatically improved with Agile Data:
Application software
Application server hardware and configurations
Network
Database or data warehouse software and configurations
Database or data warehouse server hardware and configurations
Dataset size and integrity.
Delphix customers use the Agile Data Platform to improve performance testing by quickly being able to run comparative test (A/B testing) of any changes in the entire application stack. Delphix ensures higher fidelity of testing by providing fresh, full-sized datasets--one of the biggest sources of false negatives in performance testing. In addition, for those customers that invest in a physical, “like-production” final test environment, Delphix provides a virtual to physical feature (V2P) to restore data into the physical environment to automate and accelerate final performance tests.
A metrics based approach
We often hear concerns from DBAs that testing performance in an environment that doesn’t match the production configuration exactly is meaningless. Delphix customers, however, are improving the performance of their application releases by using virtual databases (VDBs) to test a wide array of data modeling and configuration changes that affect query run time. They do so by capturing a baseline of performance metrics in their virtual environment, then use the many benefits of VDBs to try all manner of configuration and code changes. By comparing performance before and after the change they see the relative performance delta and decide whether or not to keep it. The ease of A/B testing means all changes can be measured for impact to application performance.
Delphix plugs in easily to performance testing suites like LoadRunner and WinRunner. Leveraging Delphix APIs you can automate workflows that run testing iterations and quickly reset test environments to rerun scripts when errors have been found.
Performance testing requires full, fresh data
Many organizations don’t even attempt to test performance until very late in their development cycle because it is only in the UAT (or equivalent) environment that they have a full copy of their production data set. Errors and inefficiencies found at this stage are expensive to fix and are often promoted to production due to pressures from the business to meet release schedules.
Delphix customers give each developer, or team of developers, a full, fresh copy of the database where they can validate the performance of their code in the UNIT TESTING phase of their projects. Poorly written queries are identified by the developer and can be fixed immediately, before their code is submitted for integration with the rest of the application. The test/fix iteration is much tighter and results in higher quality, better performing application releases.
How does Delphix enable this?
VDBs created by Delphix have many advantages over a physical database, and therefore can be used in unique ways. Consider the following:
Self service. Delphix automates all the complexity required to make changes to a database, allowing developers and testers to get what they need without waiting on associated support organizations.
Fast provisioning. VDBs require no data movement at the time they are created, so even large databases can be created in a few minutes.
Easy data refresh. Refreshing a VDB with the latest data from production can be done with 1 click. Never test against synthetic data.
Data rewind/reset. Delphix tracks all changes made to a VDB and can rewind the state of the database to any point in time at the request of the user. Run a test, rewind, change parameters, run the test again.
Efficient use of infrastructure. VDBs run in a tiny storage footprint, allowing teams to run many more database environments in parallel.
Efficient use of licenses. Turning VDBs on and off is trivial. Test environments can be spun up as needed and suspended when testing is finished. Suspended VDBs use no resources of the DBMS.
Database relocation. VDBs are easily moved between database hosts, even across datacenters.
Following are examples of performance changes that can easily be tested in VDBs:
Database Configuration
changes to initialization parameters (eg. optimizer_index_cost_adj, optimizer_index_caching, etc)
changes to redo size, parameters
DBMS version and patch set
SGA size
CPU type, speed (move VDB between database hosts)
Different DB statistics, statistics gathering methods
Data Modeling
Index changes
SQL Profiles - Like the old stored outlines, you can set up a complex system of profiles on a VDB and test different explain plans
Run complex and potentially debilitating queries on a VDB to minimize impact, use TKPROF and heavy tracing you can’t do elsewhere
Application Configuration
Testing application server connection pool sizes/limits
network bandwidth testing for multi-hop/firewall configuration
theoretical maximums for concurrent batch jobs (not just at the DB, but the app tier as well)
testing database monitoring solutions/thresholds/configuration impact
Oracle trace event impact when turned on (deviation from a baseline)
Enabling a physical UAT environment with Delphix
As mentioned above, many Delphix customers will still maintain a final testing environment that matches the production setup exactly. They will have fibre channel (or equivalent) connections to the SAN directly from their DBMS host. Even in this environment, which bypasses the Delphix Engine, our software can provide great benefit to the testing process.
The V2P feature can be used to migrate any data set contained within Delphix to a physical storage environment. That means any data set captured from production, or any data set modified by a VDB can be pushed to UAT in an automated fashion by Delphix. Running a V2P operation is not as fast as creating/refreshing a VDB because it requires data movement, but it is faster than restoring a traditional database backup and automates all the instance creation and configuration tasks.
Bringing it all together
The high level life cycle of performance testing on the Delphix Agile Data Platform looks something like the following:
1. Create and/or refresh development environments with the latest full data set from production.
2. Use VDBs to iterate quickly on unit tests of new code, data modeling changes, DBMS configuration changes.
3. Integrate and test all changes in a highly parallelized QA environment, using VDBs to minimize the setup time between test runs.
4. Run V2P to migrate release candidates to UAT for final performance verification.
5. Promote changes to production.
#Tip