Patterns why data teams destroy and rebuild the legacy Data Platform
Some are tongue in cheek but maybe not so much
Russell Searle asked a question over on this LinkedIn post:
Why do functioning data solutions get torn down?
I keep seeing ‘in fighting’ between data practitioners. Each with their favorite or preferred method, architecture or technologies.
Invariably, business areas suffer when the experts they trust advise against (or even degrade) a functioning solution because “That’s not how I’d build it”
Anyone got any ideas or have success stories how they’ve overcome this tribalism?
I started to respond off the cuff (and somewhat tongue in cheek) and then I thought some more and went and added more options.
Then over a lunchtime pie I was telling Nigel about them and we both realised these are real patterns we have seen in the real world across the decades. (for good or bad)
Reading Russells question again I can see I have answered with the patterns I see on why they get rebuilt, what I call the “Problem” statements.
Option 1: Because tearing down the old solution and building a new one keeps you very ‘busy’ and you can do it for a while before you get asked what value you have added to your organisation.
Option 2: Its easier to build from scratch than to understand what somebody else built or why.
Option 3: You only know that technology and tools and you can’t be arsed to learn new ones
Option 4: The previous team built on technology that has been brought out by one of the large legacy vendors and has gone to the gods software waiting room to die, and its not a sustainable long term solution.
Option 5: The previous team brought all the cool startup technologies and tools at that time and those startups and their products have gone to start-up heaven (after being in start-up hell for a while)
Option 6: The new vendor will pay for you to go to conferences as a presenter to talk about a “use case” that is really a thinly veiled sales pitch for said vendor.
Option 7: The previous solution doesn’t actually work and you have no choice but to replace it.
Option 8: While the current solution works, the cost to operate it is very high and you have been told to reduce costs. The technology and tools don;t have any easy way to automate the data work being done, so you cant apply DataOps patterns to reduce those costs. So you have no choice but to bite the bullet and wear the cost upfront to replace them to receive the needed cost benefits over time.
Option 9: Your boss hired a very expensive top tier consulting firm to do a “data strategy” and they recommended replacing the current solution with new technologies and tools, that they are “partners” for. You dont want to go against that recommendation because your boss would fire your arse.
Option 10: The technology used was just a database and a shit ton of bespoke code that was not documented and the person who wrote the “solution” has left. You cant actually find the soucre code that is running the platform.
Option 10a: Said previous person is of course willing to provide ongoing support and development of that code as an “ongoing service”.
Nick Pinfold added:
Option 11: The database was actually files in SharePoint with PowerBI sitting over it.
Kris Vandeurzen added:
Option 2b: probably because no documentation excists on tables, ETL or business Logic
But I after reviewing these responses I can see I didn’t actually answer his question:
Anyone got any ideas or have success stories how they’ve overcome this tribalism?
I’m going to have to think hard and come up with the “Solution” patterns.
«oo»
Stakeholder - “Thats not what I wanted!”
Data Team - “But thats what you asked for!”
Struggling to gather data requirements and constantly hearing the conversation above?
Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.