With data sprawl, compliance, and volume challenging businesses’ agility,
automating the delivery of on-demand, compliant data within data platforms and pipelines is more crucial than ever for enterprises to get full utility from their data.
This article was originally published on the Delphix website here March 15, 2023.
It’s been an exciting few days at SQLBits 2023, full of great networking opportunities and informative sessions. SQLBits is the largest data platform conference, with numerous trainings and general sessions about the many capabilities of data platforms. This year didn’t disappoint, either— if anyone is doubtful about the promises of innovation, they need look no further than data integration platforms. The promise that these platforms and their applications hold for data analytics and business improvement is astounding.
Attending SQLBits is always a pleasure because it’s such a great environment for gaining further insight on how to best leverage data integration platforms for one’s organisational needs. Below are my biggest takeaways and reflections from the conference after attending a multitude of keynotes and sessions, as well as connecting with many peers.
Takeaway 1: Data analytics and insights are enterprise sorcery— and they’re more essential than ever for business success
The theme of the 2023 SQLBits conference was Dungeons & Dragons— the famous tabletop game where players create a fantasy character to participate in a campaign. Magic is a regular occurrence in both Dungeons & Dragons campaigns, and we’re all familiar with the image of witches and wizards using raw magical energy and their specialised magical knowledge to conjure up highly useful spells and tools seemingly from nothing.
Magic, of course, seems unattainable— but enterprise IT leaders do something similar every day. We conjure up actionable insights and meaningful analytics using specialised expertise and vast amounts of raw data. Modern-day organisations depend on data for their success just as much as fictional sorcerers depend upon magic for theirs— a 2022 report from Heap and IDC shows that companies using mature data practices achieved 2.5x better business outcomes “across the board.”
Meanwhile, organisations are only poised to rely on data even more. In a big data-related survey from Capgemini and EMC, 70% of respondents said that obtaining value from big data would be crucial to their future success. And 56% said that their investment in big data over the next three years would surpass previous investments in information management.
Data analytics and insights are more important today than they have ever been. But they present a wealth of opportunities for the enterprises that will harness it well.
Takeaway 2: Well-appointed data platforms are crucial for managing data— but they require collaboration to harness effectively
The various data professions and magic professions share several similarities. For instance, they’re both in high demand— you’d be hard-pressed to find a sorcerer applying for unemployment benefits.
Meanwhile, every witch and wizard has their tools of their trade for harnessing magic— wands, cauldrons, spellbooks, and pointy hats are perhaps the most iconic. IT professionals similarly rely on equipment for working with data– such as APIs, continuous integration and continuous deployment (CI/CD) tools, terminals, and a steady supply of coffee or tea.
Data platforms — and automation within those platforms — remain the most effective tools for harnessing the power of data within enterprises. The breadth of capabilities, operations, and functionality of a well-appointed platform is extensive— and for many enterprises, their platform has become a core component of their data strategy. But using a data platform to its fullest extent requires coordination from many teams, such as database administrators, testing professionals, software developers, compliance specialists, and more.
That’s why I especially appreciate the collaborative, educational, and camaraderie-filled atmosphere at SQLBits. It can be all too easy to lose oneself in the sheer amount of innovation possibilities present in data platforms and their applications. But professionals at this conference are all too eager to share their findings and expertise with their peers and help them become better at their jobs. What’s more, multiple SQLBits sessions about maximising mental health show that we value the wellbeing of our peers and colleagues as we work to advance data management objectives. Whether within our organisations or within industry events, we want to work together to achieve common goals.
Takeaway 3: Data sprawl, volume, and regulations makes leveraging data tougher than ever for enterprises, hurting business agility
We need all the help we can get when working with data– because it’s never more difficult or more rewarding for enterprises to get the full value of their data. Worldwide data volumes are astronomically high, and storing enterprise data in a variety of formats concentrates it within silos. The resulting data sprawl limits enterprise data’s strategic value, stymying enterprises’ ability to leverage its value.
What’s more, data privacy regulations across the world — such as GDPR, HIPAA, GLBA, and PCI DSS — hold organisations accountable to both protecting many forms of sensitive data and fulfilling privacy mandates such as the right to be forgotten. These laws are changing and growing over time— for instance, the European Union passed the Digital Operational Resilience Act (DORA) in November to levy new security requirements for the European financial sector.
Non-compliance with these laws threatens organisations with significant fines and punitive penalties.
The sheer act of making data compliant leads to other issues. When sensitive data is moved from systems of record, it must be made compliant without impacting its business value. But this process can be slow and complex, executed by professionals with specialised knowledge, and it can take time for enterprises to ensure that their data is compliant. This can force enterprises to slow down their data delivery processes, creating a bottleneck for software development, testing, analytics and even the training of machine learning algorithms.
Many businesses can relate to the harms these bottlenecks can cause— 82% of businesses surveyed in Experian’s 2022 Global Data Management Report stated that a lack of data agility would give them a competitive disadvantage. We may have a plethora of data at our fingertips, but we need a strategy to use it efficiently (and securely) to get value in the form of business insights.
Takeaway 4: Access to compliant, on-demand data is key to achieving maximum data agility within data platforms
Unlocking data’s full strategic value requires that data to be movable, dynamic, secure, compliant, and manipulated into a common format. From there, extract, transform, and load (ETL) pipelines must be adapted to each system of record while also scaling to support the massive data sets of modern enterprises.
That’s easier said than done— traditionally, achieving ready, efficient data access while also ensuring that this data is compliant and secure has been a tough balancing act for enterprises to maintain.
But cloud-ready data platform applications can solve this issue. Delphix, for instance, offers a fast, automated, API-driven process for identifying sensitive information across an enterprise and automating data masking wherever that data resides. Delphix masks data to comply with virtually any data privacy regulations while also maintaining the data’s referential integrity.
Delphix also serves as a core component of the Microsoft Intelligent Data Platform (via a partner ecosystem), which supports solutions such as Azure Synapse Pipelines and Azure Data Factory for code-free ETL and enterprise-wide data connectivity. Using these solutions helps avert data silos by facilitating the efficient movement of data across an enterprise while quickly ensuring compliance along the way.
Takeaway 5: Data automation holds the key to future enterprise success
As data continues to grow in importance for enterprises, more and more resources will need to be devoted to data-related work— from budget dollars to employee time. The solution to this issue is automating more aspects of data management, and many companies are exploring automation— 81% of businesses surveyed in the Experian report said that they automate more aspects of their data management programme to free up time to devote to more strategic activities.
At a time when companies are facing a never-ending set of challenges and disruption, they can’t afford to be bogged down by inefficient data processes and operations. But without automated access to the correct data at the correct time, any strategy is doomed to fail from the start.
The many SQLBits sessions on automation prove that it’s top-of-mind for IT professionals. But data automation is easier said than done. While compute and code have been automated, businesses have traditionally held back when it comes to data—considering it riskier due to compliance requirements. For organisations looking to achieve true digital transformation, data is the last automation frontier. And that frontier will only become more essential to explore as time passes.
Looking Forward
SQLBits showed us that data is more important than ever for enterprise success. Employing a robust data platform with effective cloud applications is crucial to gleaning essential analytics and insights from its massive data stores. And as data consumes more corporate resources, legacy data processes become a bottleneck to innovation— making automation of these processes more paramount than ever.
However, data sprawl, compliance, and volume threatens the effectiveness of enterprise data management every day. It’s therefore become essential for enterprises to ensure the steady flow of on-demand, compliant data within ETL pipelines and enterprise-wide connectivity in order to maximise their agility with data — as well as the business as a whole. And integrating applications such as Delphix with data platforms such as the Microsoft Intelligent Data Platform allows organisations to streamline data flow and automate data processes (including compliance) so that they can maximise data efficiency and achieve business objectives quicker.