Data Management is still too complex
Time to experiment again on making it simply magical
A long time ago in a galaxy far, not so far away....
Almost 6 years ago, Nigel Vining and I started the journey to try and reduce the complexity of managing data.
How time quickly moves on.
We have iterated many times trying to solve it, but its still not a problem solved.
The attached screenshot is from a McSpikey (experiment) we did in the very early days, where we wanted to know what it would take to make a natural language interface to help us do the data work we do.
The answer was it would take too many people, too much time and too much money to build a natural language engine that solved the problem of the end to end process of wrangling messy (aka real word) data.
So we focussed on other patterns to try and solve it.
When ChatGPT came out we of course tried another McSpikey to see if it is a problem solved.
I posted a blog post on that experiment in Dec 2022
https://agiledata.io/blog/the-language-of-data-is-not-so-natural/
It was of course based on the Text to SQL pattern that was the common experiment. Results of the experiment was it worked a treat on "pretty" demo data but sucked with complex real world data.
Time has moved on again and we now have access to LLM models that are multi modal and have exponentially more power.
We have automated a large number of the data engineering patterns into our core AgileData Platform. But the AgileData App we have built is designed for Nigel and I to do the data work, and it requires tacit knowledge to leverage it.
So its time to have another go at solving that data complexity problem.
This time we are going to experiment with a few things:
What would it take to build an ADI Assist that helps an Analyst conceptually model data?
What would it take to build an ADI Assist that helps an Analyst create the transformation rules that would convert that conceptual data model into a physical data model that was consistently loaded?
What would it take to build an ADI Assist that helps an Analyst do the work to answer their stakeholders business questions with data?
What would it take to provide a virtual Agile Data Coach that coached the Analyst on all the patterns they were using when they were assisted by ADI?
Would any of these be so valuable to somebody that they would pay for it on an ongoing basis?
As I tend to like to do, I want to build in public.
I find creating content that documents as we go cathartic and it also helps me think about what we have done and what we should do next.
But from experience it takes a good few hours to write that content each week, and also its a solo task. This time Im keen to get Nigel Vining to co-create it with me.
Not sure what the best format for that is.
15 Minutes podcast chat between the two of us each week?
15 minute video chart between the two of us each week?
Any better ideas?
Anything you can think of that would be useful for you as we spend the next few months experimenting in this space?
Add a chat and let us know your thoughts!