I am starting to pick up, in discussions with clients and prospective clients, on a very definite trend.
“Data as a Product” (DaaP)
Using data as a product is a reasonably new idea. We as a Data team were thinking about it as early as 2017.
However, it’s how you deliver your data product that is starting to “trend”. In 2017 we had many petabytes of data that we were crunching through to make sense of what we had, what it was and what we could do with the data to make it useful for our clients (internal and external). However, and here is the rub, the “data delivery” cycle looks a bit like this:
- Customer asks for data
- We search and find data across different formats and types
- Understand what we have and how to put the sets together
- Process it for the customer
- Provide it back to the customer in some bespoke form/format.
This works and we’ve all been doing this for ages.
Hard to scale
However, it simply doesn’t scale. Think of it like this, in order to go around the cycle once, you need someone to coordinate with the customer – let’s call them a “Delivery lead” or “Project Manager”. You then need people who understand data and what it means, “Data Analysts”, and you need to figure out how to crunch and produce the final result, “Data Engineers” and finally some oversight to ensure what is produced is monitored, suitably available etc. That’s “Product Support”.
Now imagine this group or team optimistically processes one request every three to four weeks (highly unlikely – for reasons outside of this article). This means a group of six people can effectively produce 12 assets a year. The obvious next step is to scale horizontally – however, this is hard and expensive.
As the saying goes, “Nine women can’t have a baby in one month” therefore there are always SMEs and bottlenecks. In the real world, the delivery of data is cumbersome and slow. This is frustrating for your customers, your delivery teams, and your wallets.
What’s the answer? Well, this brings me to the “trend”.
Many bright people are starting to see that they can standardise the process to drive efficiencies. The most obvious is if you could offer your data in some form of “data store” and allow people to “check out” the data they want. This would solve Step 5 and reduce the support nature of the delivery.
Data factories
Others are even standardising Step 4 by building “pipelines” that allow you to push a given set of data through to ensure it ends up as a single asset, think of this as a sort of “data factory”.
So here we are – data teams all starting to build “Data Stores” or “Data Marketplaces”, places where their users can download assets the team have previously put together.
I have heard this described more times than I can mention as “the Amazon store of data”. Which is great! However, this isn’t going to solve any of the actual underlying issues.
The main issues are buried in Steps 2, 3 and 4. This is where the difficulty is.
Think about Step 2, if you have petabytes of data, how easy do you think it is to find what you really want? Furthermore, is the data the “right data”?
Is it the actual source or a copy? So how can you easily find what you want? Step 3, do you imagine your team is, or has, an expert in the data you are looking for and understanding what it represents and its nuances? Very unlikely.
Shiny new trend
Finally, Step 4 – This seems easy but is actually very hard. How do you produce the data in a “safe” way, ensuring what you produce is allowed to be consumed by the target audience? What if there is data there that shouldn’t be shared, according to some rule? What if that rule only applies to 50% of the intended audience? How do you solve this in a consistent way?
So this trend for a shiny user interface (UI) that has your data assets on a shelf is… just that – a shiny UI.
This trend is a good one but there is so much more to understand and solve.
Firstly, think about how you can search and discover your data and really understand what you have. Second, democratise your data, and allow the people who know the data to tell you what you have and what it means. And finally, find a way to dynamically enforce data security/governance on your data and your data assets.
If you can achieve all of this with your new shiny UI on top, you will give your organisation the service they need and truly unlock the value of your data.
Image: Markus Spiske – Unsplash
