The data domain is about to get so fooked by “AI”

But not the way you would think

The data domain is about to get so fooked by “AI”

I have just finished listening to one of the latest Lenny Rachitsky podcast episodes on my daily walk in the Kapiti sunshine, this one was with the founder of Lovable.

Listen, I was thinking we in the data domain are about to be right royally fooked as a result, but probably not for the reason you think.

I’m not worried about the “AI” replacing humans in the data domain (yet).

I am worried about the application sprawl the new “AI” creator economy is about to spawn and the chaos it will bring to the organisations we try and help using their data.

My thought process goes like this:

A few decades/waves ago, there were monolithic applications. They were slow and expensive to build and change.

In the data domain, our tooling was also slow and expensive, and our ways of working were not much better.

But that was OK, we only needed to get data from one application or a small handful at most.

Then we got a wave where applications became less monolithic. The data tools changed slightly but not much, and the ways of working didn’t change at all.

But that was kinda OK, as our data ways of working were only one standard deviation away from the way those applications were built (so to speak).

And while the applications were more microservices-oriented, from a data domain point of view, we still only had to grab data from a handful of applications.

Then the cloud. mobile and social waves came, and combined with DevOps, our application software brethren took a great step forward, both in technology and ways of working.

In the data domain, we made a few small steps. We removed a lot of the physical hard storage and compute constraints we were bound to, we attempted to “microservices” the data platform with the “modern data stack” (what a fail that was), and we tried to decentralise our data teams and democratise our data work to people who didnt have a data pedigree.

We didn’t move fast enough, and we became two standard deviations away from our software brethren (so to speak). We were walking fast, they were sprinting (yes, there is an agile joke in that last sentence).

And we no longer needed to get data from a handful of applications, we had to collect data from tens, if not hundreds, of SaaS apps.

So back to the podcast epsiode.

It’s clear with tools like Lovable that individuals will, sometime soon, be able to create applications as a team of one and at a great rate of knots.

We are entering a new wave of the creator economy, where individuals will spawn Apps and “AI Agents” at a horrendous pace.

The Smartphone App Store wave will seem like a trickle compared to the tsunami of new Apps and “AI Agents” we are about to experience.

Again, I am not worried about apps and agents that do the data work.

I am worried about the number of small and independent applications that an organisation will start to use that we need to collect, combine and consume data from.

It will be in the thousands.

Our data tooling and our ways of working are not designed to manage this volume. I don’t mean the volume of data but the volume of disparate systems we will need to touch.

Not to mention trying to deal with 1 thousands ways that a Customer is identified, each dealt with different in each of those apps, so we can create the organisations golden “single view of customer”.

Is the move to data teams of one that mirror the “AI” creator economy the answer? Maybe.

Is a data team of one possible? Yes.

Is it feasible and viable right now? Nope.

Is it sustainable right now? Nope.

In the data domain we are three standard deviations away from the rest of the technology world (so to speak) and quickly getting further behind.

What I see is us trying to replay the same technical and ways-of-working patterns that we have regurgitated and brought back to life wave after wave (data catalogs anyone?).

That pattern of behaviour allowed us to survive with the small and big waves we encountered in the past.

But I don’t think it will help us keep our head above the water with the tsunami of targeted, one-use-case applications about to land on our doorstep for collection.

So have you thought about how you are going to ride this latest “AI” wave?