Barry O’Reilly and this week’s guest, Zhamak Dehghani met 10 years ago when they worked together at ThoughtWorks. Zhamak is currently the Director of Emerging Technologies at ThoughtWorks and the creator of Data Mesh, which Barry describes as “one of the most exciting paradigm shifts in how we manage data at scale.” He and Zhamak discuss why traditional data architecture models are failing and how applying product thinking principles to data management is a way to harvest the data’s full potential. “This show,” Barry remarks, “is for those who are curious to understand how to bring the convergence of product thinking, data management, and distributed systems development together to create platforms and products of the future.” Listen to Decentralizing Data: From Data Monolith to Data Mesh with Zhamak Dehghani.
Early Values
Zhamak has always believed in distribution of responsibility and decentralization of ownership. She finds that these design principles are more compatible with real life. Colleagues taught her the Unix philosophy early in her career which now forms the basis of her data management approach. “They taught me those wonderful ideas to build systems and programs that do one thing and one thing really well. But most importantly they work together really well,” Zhamak says. “‘Simple is beautiful and beauty is the truth’… Reduce systems to their simple principles; then together can emerge complex behaviors.” She saw an opportunity to bring the UNIX principles to data. [Listen from 3:25]
Subscribe
Challenging Assumptions
It often takes someone new to a system to point out obvious flaws to long-time practitioners. Zhamak says that when she came into the world of big data, she was agnostic to the accepted assumptions, so she felt free to challenge them and conceive a different paradigm. For some reason when it comes to data, people eschew UNIX principles and see it as something to be centralized. Unsurprisingly, a data lake becomes monolithic and departments become siloed. Reimagining the world of data requires a new language, she points out: “The moment you need to imagine something different you need to use a very different language.” Instead of seeing data as an asset – which you want to hoard and get more of – Zhamak advocates that data can be seen as a product which should be used to serve internal and external customers. Barry adds that the idea of the single source of golden data makes companies unable to move as they get bigger. [Listen from 10:20]
Move to Product Thinking
Barry comments that the shift towards product thinking started with Amazon. Their monolithic database was preventing them from scaling. “They realized that they needed to create these smaller, more autonomous units that had the capabilities to build things just like product teams. This is where this notion started to emerge from changing the organizational design… both technically and just how teams would work together,” Barry says. In this new way of working, teams could experiment and own outcomes. They could make small, quick changes and see the effects. [Listen from 18:50]
What is Data Mesh?
Instead of trying to fit data into a mold, Zhamak feels that its dynamism should be embraced. “Create an architecture and ownership of data that starts with the assumption that data can be useful and shareable and trustworthy right at the point of origin; and then allow for different domains and different aggregations, different projections to get created as a mesh picture,” she posits. She explains how this new view of data impacts ways of working and the type of platform a company would create. The four principles of the Data Mesh philosophy are, “domain ownership of the data; data as a product; self serve data platform to enable autonomous teams; and a federated computational governance to balance the interoperability of a decentralized world with the trust and security built in.” [Listen from 23:55]
Thinking Big and Starting Small
‘Think big. Start small. Move fast.’ Start with a big vision about where you want to go, Zhamak advises. Next, ask yourself where you can start. She says that you should start with just one or two use cases. Building out these small solutions and testing them will help you build the foundation to go further. Work backwards and see what data you need then seek it out. Then figure out the capabilities you need to build your first generation platform. Build the platform. Move to the next use case and rinse and repeat. “Even though you’re taking the small steps and moving fast you’re still moving towards that big vision,” she says. [Listen from 33:10]
Empower People To Use Data
Barry uses Lipton as an example to illustrate how decentralized data can help a company innovate. He and Zhamak agree that data should be served as a product available to the rest of the company so that they can combine, explore and build new data sets. New technology allows you to synthesize data to find gaps that you can fill with new offerings. Zhamak urges leaders to remove the friction: empower the people closest to the data to experiment. [Listen from 42:10]
Don’t want to miss a post? Sign up for my podcast newsletter and monthly newsletter!
