Consider the contributors to a challenging enterprise data estate:
Data silos and sprawl: Data is locked in legacy systems and information silos, many of which are difficult to modernize. As a result, critical data remains out of reach of many advanced data analytics and AI-enabled initiatives. There also is no single source of “truth,” since data is distributed across different, disconnected data platforms.
With the landscape littered with unorganized data swamps and tool sprawl, the ability to automate the data pipeline, operationalize analytics and decision support workflows, and unlock value from analytics and AI is seriously impaired. “Companies have investments in data platforms, tools, and applications, and are inclined to keep things the way they’re currently structured,” Kamat says. “That stickiness causes a lot of concern about the impact of moving to a different architecture or a more modern data platform.”
Increased complexity: The rise of hybrid and multi-cloud infrastructure has created new layers of IT complexity that make it costly and time-consuming to manage end-to-end data management and AI workflows. With data proliferating across data warehouses and data lakes, and from the cloud to the edge, there is a lack of data lineage and consistent data quality standards across the distributed data estate. This makes it difficult to channel the right data to solve specific business problems.
Click here to learn more
In addition to the sprawl of structured data, there is unprecedented growth of unstructured data from audio, video, clickstreams, and social media, further clouding visibility and increasing the data management burden. “Without a clear inventory of data assets, any data and AI initiatives start on shaky ground,” says Satya Prakash Chinnam, Lead Offering Manager, Data and AI Services at Kyndryl.
Integration in the spotlight: The legacy data estate raises a host of integration challenges, with issues such as “data gravity” making it difficult to easily move information from one platform to another. IT teams need to define an integration blueprint for bridging diverse data platforms and sources while ensuring those connections are forged with the proper security, privacy, and data access controls to remain in compliance and mitigate risks.
Lack of governance: A central tenet for data modernization is governance, and many organizations lack the tools, structure, management commitment, and know-how to establish, evolve, and enforce a formal data governance framework. Without consistent policies, processes, and an organizational structure to support enterprise data management, teams can struggle to scale data initiatives and groom data so it’s actionable for driving analytics initiatives and accelerating growth.
Example: A Deloitte study of AI adopters found that businesses struggle with data management basics, including preparing and cleaning data, integrating data from diverse sources, training AI models, and tackling data governance. The survey found 40% of organizations reported a “low” or “medium” level of sophistication across critical data practices, while close to one-third of executives identified data-related challenges as a top-three inhibitor to AI initiatives.
Last-mile speedbumps: Last-mile delivery in the data supply chain varies for each application, but in many cases, it’s questionable whether the right data is available to the right users at the right time. Without a single version of the truth, formal data lineage, and role-based access, there is no standardized way to prepare and curate data for consumption by applications and users. Poor data quality leads to poor quality of AI models, which in turn erodes the success of data-driven business.
Immature AI workflows: Too often, AI is treated as a one-off use case disconnected from an end-to-end process that operationalizes the AI lifecycle. “A lot of AI models don’t get deployed into production—they are developed and built for a particular purpose and by the time they’re complete, the original business need is delayed, or the model is no longer relevant,” Kamat says.
In contrast, organizations successfully executing AI-enabled business have taken the time to establish processes to continuously optimize model performance and to deploy AI in repeatable and sustainable ways. “There needs to be a level of automation, simplicity, and a curated data set that AI applications and data scientists can use for rapid model development and deployment,” Kamat says.
Organizational alignment: Delivering sustained value throughout a mature data and AI lifecycle requires commitment by multiple teams—not just IT, but across the entire business. Without synchronized strategy and commitment between IT and lines of business (LOB)—i.e., a coordinated change management campaign—data modernization and management efforts can remain unfocused and adrift. “There are typically different LOBs within a large organization, and individually, they don’t see the magnitude of fragmentation within the organization,” explains Chinnam.
Skills and resource challenges: Data modernization and AI initiatives call for new skills and knowledge of diverse platforms and technologies on a scale that tests in-house resources, especially for organizations already grappling with talent shortfalls and limited IT budgets. Many organizations don’t have clear accountings of their current environments, creating gaps in formal plans or visions for modernizing the data estate—and the talent supporting it.