Faced with an ongoing data generation explosion that refuses to quit, organizations are coming to grips with the realization that data silos will be a fixture in their IT estates for the foreseeable future. One design pattern that’s positioned to help these organizations grapple with all that distributed data is the data mesh, which appears poised to spread its wings in the new year.
The data mesh concept, as we have covered several times in these pages, is an architectural design principle that was first put to paper by Zhamak Dehghani, the director of next tech incubation at Thoughtworks North America in 2019. Over the past two years, the idea has garnered significant momentum among organizations trying to cope with the enormous growth of data, much of it unstructured.
Instead of trying to unify data storage and management, which is not practical for enterprises in the exabyte era, Dehghani realized that organizations are better off setting a series of guiding principles to help various teams work with data in an efficiency and effective manner. The data mesh architecture that she devised was marked by four main characteristics, including: domain-oriented decentralized data ownership and architecture; data-as-a-product development; the rise self-serve data infrastructure platforms; and federated computational governance.
This approach is gaining traction among big companies with far-flung operations. One backer of the data mesh approach is Collibra, the developer of data governance tools. According to Collibra co-founder and CEO Felix Van de Maele, the data mesh approach is the right one for enterprises.
(Image courtesy Zhamak Dehghani)
“We are big believer in data mesh,” Van de Maele told Datanami in an interview last month. “We had Zhamak [Dehghani] be part of our keynote at our most recent Data Citizen Conference earlier this year.”
There’s no longer a debate over whether data can be effectively centralized, Van de Maele said, and that is helping to drive interest in data mesh.
“We have to embrace the fact that, to do data well at scale, it will need to be distributed,” he says. “You can’t centralize. We’ve been trying for 30 years to move all data to one place. That’s never going to work, so you have to accept that it’s going to be fragmented.”
One aspect of data meshes that resonates with Collibra is the…
