Storing, optimizing, protecting, and serving up data of any type is often quite expensive. Unfortunately, the process many businesses use to provide data to their developers and testers is more expensive than it should be, and goes well beyond the cost of data storage. 

In this article we’ll look at the cost drivers behind test data management (TDM) and how those costs can impact the budgets of application development teams.  In addition, you’ll read about analyst research and case studies that cite significant cost savings, development productivity enhancements, and faster time-to-market that come from improved approaches to TDM.

Your Application Development Budget

The way businesses assign budgets to application development teams varies significantly from one business to another, often due to the development methodology they employ. Many businesses still use project plans to set their capital budgets and then separately estimate the non-capital (operating) expenditures for supporting and maintaining their enterprise apps. Many others have moved on to funding by products or value streams, each of which are inclusive of both CapEx and OpEx. 

There are costs that are borne directly by the application development teams (or DevOps teams) and costs that are driven by those teams but included in other budgets. Common examples of the former (direct costs) include the salaries and benefits of the application developers and the purchases they make. Examples of the latter (indirect costs) include things like the servers and storage in your data center that you use for development. Those costs probably hit the budget of your Infrastructure & Operations (I&O) team. (Sometimes I&O teams charge their costs back to the development teams.)

Knowing a bit about how costs are assigned or allocated to your team will help you understand the impact of TDM on your budget. Let’s explore each type of cost, the impact of TDM on them, and the DevOps TDM best practices for mitigating increased costs.

Direct Costs to Application Development

Test data typically drives up the cost of your application development, cloud storage and infrastructure, while sometimes engineering and admin roles are also included in your app dev budget.

App Developer Time and Productivity

Your developers spend time obtaining and handling the test data they need to code software, test it, and resolve issues such as software failures in production. For many developers, requesting data consumes precious time when they would rather be programming. Requesting and provisioning data often occurs many times during a single release due to different requirements such as unit testing, user acceptance testing, systems integration testing, and more. 

But it’s the quality of test data that impacts your teams the most. Per a study by IDC, the time spent by developers on software defects was reduced by 53 percent after implementing Delphix software. The primary cause cited was the improvement in test data quality. In other words, better quality test data results in better quality software and a shifting left of quality, where defects are fixed faster and earlier in the software development lifecycle. 

The overall reduction of time cited by IDC, from setting up development environments through coding, testing and deployment, was 40 percent. This was due to a combination of “faster data” and “higher-quality data.” This translated to a 30 percent reduction in the development time per new application, a 33 percent increase in the average number of releases per app per year, and a 26 percent increase in the number of new updates/features delivered.

Cloud Storage and Infrastructure

As businesses have shifted to using public cloud infrastructure for development, the cost of infrastructure has often shifted from indirect (borne by I&O) to direct (borne by app dev) and from mostly fixed (unchanging over a period of time) to variable (changes with the amount and time of consumption). This means that app dev teams have had to include cloud costs in their own budgets. 

Many have since adopted a practice known as Cloud FinOps, which is designed to optimize the value and cost of cloud consumption. But waste remains a significant problem. According to Flexera’s 2023 State of the Cloud report, which analyzed survey responses from 750 cloud decision-makers and users globally, approximately 28 percent of all cloud spend is wasted.

When test data is stored and processed in the cloud, those costs tend to hit the app dev budget directly. Therefore, it’s imperative that the amount of test data is kept to a minimum. This is where data virtualization, a core component of effective DevOps TDM approaches, has such an impact. Test data management is ripe for data virtualization, whereby virtual databases or sources take the place of physical databases. 

This approach can drive enormous savings. Per the IDC study, Delphix’s customers saw an average reduction of 82 percent in test data storage and 32 percent in server footprints. These benefits are commonly cited by Delphix’s customers, such as Worldpay (75 percent to 80 percent storage reduction), Molina Healthcare (-4PB of data), Michelin (10X reduced data footprint), and Cal State University ($7.5 million reduction).

For those employing public cloud services for test data management, these savings will often be realized in your app dev budget. And since many teams centralize their DevOps TDM practices, they often centralize the budget as well, meaning many app dev teams no longer bear the direct costs of their test data infrastructure at all.

DevOps Engineering and Database Administration

Some development teams, especially DevOps ones, have roles that include infrastructure administration. These might include DevOps engineers or DBAs that spend time setting up development and testing environments, supporting the build processes, and other tasks needed to get complex software applications from development into production. These teams benefit significantly from automation. 

Since the cost of these roles is typically borne by I&O or other teams, we will address them below. However, if those costs are borne directly by app dev (or app dev is charged back for them), then the efficiencies gained through automation will be realized in the app dev budget.

Indirect Costs to Application Development

Poor test data management often drives higher costs of I&O and IT security and compliance. While these often hit other budgets, many businesses charge some or all of them back to the app dev teams that consume them.

Internal Infrastructure & Operations

As described above, test data is often stored and processed on public cloud services. DevOps TDM practices, in particular data virtualization, reduces the amount of storage and servers needed. This reduction applies equally to any on-premises storage and servers consumed for test data. 

Test data management also drives the cost of the labor used to manage it. By automating data provisioning (with data virtualization, API integration, and other capabilities) and data masking, you can reduce the labor for TDM while accelerating the provisioning and deprovisioning of test data at the same time. 

In the IDC study, staff productivity improvements were found for managing and provisioning databases, running and managing workloads, security management, compliance, and managing infrastructure. Adding these up, the combined average productivity increase was 24 percent with an average value of $5.3 million.

IT Security and Compliance

IT security costs encompass things like vulnerability scanning, intrusion prevention and detection, and data leakage prevention (DLP). When sensitive data is present in development or testing environments, these tools and processes must be applied to those environments. Trouble is, they are often quite expensive.

One way to reduce their costs is to simply reduce the server footprint of your testing environment. As described above, the data virtualization of DevOps TDM helps you do just that. Another way to reduce those costs is to simply avoid putting sensitive data into those non-production (lower-tier) environments. To support this, DevOps TDM often provides automated sensitive data discovery and masking. In other words, sensitive data such as PII, account numbers, and authorization data are automatically obfuscated (masked) before being provisioned for developers and testers to use. When done well, this process preserves the quality of test data while mitigating the risk of sensitive data leakage.

Using data masking to eliminate sensitive data from non-production environments is a standard practice at Worldpay, as shared by Arvind Anandam, Director of Platform Engineering at Worldpay: 

“We've got one production environment, but we've got about anywhere between 12 to 20 non-production environments. We do have a very robust masking process that we have put in place. We do a lot of checks to ensure that no production data accidentally slips through into non-production. Masking is the center of everything that we do.”

Masking, when automated, can help you avoid data security costs and labor that you would otherwise have to bear. You must still properly secure development environments, but masking test data gives you one less thing to worry about (and pay for).

Opportunity Cost and the Impact on App Dev

There’s one more type of cost that can impact your budget: opportunity cost. Opportunity cost refers to the loss of potential gain from other alternatives when one alternative is chosen. Essentially, it's what you give up in order to do something else. In the case of test data, it’s the revenue lost as a result of slower time to market with new digital capabilities or features. This, of course, is hard to measure. Per the IDC report, “Six of 10 companies saw a revenue gain associated with improved data operations, or DevOps, from Delphix, generating a 1% average annual growth in revenue.”

How does this impact the app dev budget? It’s complicated, but budgets tend to increase as revenue grows. Businesses only have so much to spend on both capital (the cost to create assets) and operations (the cost to run the business). They allocate a certain amount of their revenue to do so and attempt to balance business growth with profitability and risk mitigation. Therefore, getting digital capabilities to market faster will help grow both revenue and the app dev budget. As described above, the IDC report cites a 40 percent reduction in developer time, which arguably translates to a reduced time-to-market. 

Conclusion

A DevOps TDM strategy can significantly reduce both direct and indirect costs in application development. This cost reduction comes through minimizing developer time spent on data-related tasks, reducing the cloud storage and infrastructure expenses, and leveraging automation for engineering and administrative roles. Beyond direct costs, DevOps TDM practices also help in cutting down indirect expenses by improving internal infrastructure and operations efficiency, enhancing IT security, and reducing compliance burdens.

Furthermore, the significance of opportunity costs in the context of TDM cannot be understated. Faster time-to-market, accelerated by efficient TDM practices, can lead to substantial revenue growth and potential increases in application development budgets. Businesses, therefore, must critically evaluate their current TDM approaches and seek to adopt best practices for optimal financial and operational outcomes. By doing so, companies can ensure that they are using their resources effectively, creating a more efficient, productive, and cost-effective development process. Ultimately, the proper management of test data is not just a matter of operational efficiency — it's a matter of smart strategic financial planning.