“AI” is the new Excel and Data Governance teams are even more fooked as a result
Episode #4 in the data domain is so fooked because of AI series
This wasn’t meant to become a series, but conversations and thoughts keep popping up weekly so lets see where and how the fooked Data and AI posts go.
Data Governance has already been fooked for a while
IMHO (ok not so humble) Data Governance has always been a pretty broken domain.
In fact in my view its been pretty much a joke for most data teams for quite a while now.
Data Governance is one of those domains data teams try to ignore, and if they can’t ignore it they just smile, nod and wave when they have to, and then get back to doing their busy data work.
this statement is based on most not all Data Governance teams in most not all organisations, there are some organisations and teams that are governing for value
Some common Data Governance anti-patterns I see are:
Endless consulting vendors rehashing the DAMA diagrams to provide yet another Data Governance “strategy” everybody in the organisation will ignore.
Slides decks that spewing forth roles such as Data Owners, Data Sewards etc that makes everybody ‘role’ their eyes.
The Data Owner for a data domain is one of the most senior members of the organisation, yeah right the VP of Finance is really going to approve those data access request themselves.
Governance via a toothless committee, yeah sure I will waste my time yet again writing another another long document that the governance committee will review in a months time and give me feedback on my bad speeling and grandma pistakes but little else.
The problem is they provide ideals not actions
The core problem with the current state of Data Governance is it provides a bunch of ideals, a bunch of ideas, and it rarely provides a bunch of actions.
It rarely provides actions that data teams can take to implement those ideals. And as we all know data teams are all about action and implementation.
Statements like “protect PII data” are great but if that is all that comes out of the Data Governance teams, that leaves all the heavy lifting to the data teams, and they have enough heavy lifting to do already.
Data Governance teams need to meet the data teams at least half way. They need to provide more "‘meat and potatoes’ to help data teams implement things that help improve the governing of data.
“AI” is about to make this problem even bigger
“AI” is the new Excel. People in organisations are uploading their organisations data to tools like ChatGPT willy nilly.
Or they are using new SaaS consumer services, that are fronts for LLM services from OpenAI and Google etc. Again they are sharing their organisations data with these services willy nilly.
As I have said in a previous article, Data Teams have a massive problem coming where their organisations will end up with 1,000s of “AI Agents / Apps” that generate data they will have to collect.
Data Governance teams will have to work out how to govern these 1,000s of systems to ensure they are not used outside of the rules of their organisation.
The current pattern of creating high level policies wont cut it in this new “AI” world.
A real world example
This weeks post was triggered in part based on a conversations in a product community discussion I was part of this week.
That discussion was based on this:
https://www.police.govt.nz/sites/default/files/publications/acceptable-use-generative-ai-policy.pdf
The discussion was around the question, is this level of Policies enough in the new “AI” world?
My thoughts were no, but they are a good start.
Stop giving me problems, bring me solutions….
Ok so given IMHO view the Data Governance domain has failed to deliver in its role as a core part of a Data Teams Way of Working, what needs to change.
My answer is Principles, Polices and Patterns
As I tend to do when I have ranted against a thing in the data domain for a while, I then switch mindset to try to work out how I would innovate and do something different to solve the problem (and therefore stop the ranting).
These days when I talk about valuable Data Governance I talk about:
Principles
Policies
Patterns
The next bit of content was generated by my AgileData.Coach AI Clone. Em dash and ‘delve’ purposely left in:
Principles, policies, and patterns are foundational concepts that guide how teams work, make decisions, and solve problems. They each serve a distinct purpose, and when used together effectively, they create a cohesive and adaptive way of working. Let’s break them down:
Principles
Principles are the why—the fundamental beliefs or values that guide decision-making and behavior. They’re high-level, non-negotiable, and provide the foundation for everything else. Think of them as the compass that ensures you’re always heading in the right direction, no matter the situation.
For example, in the context of data work, a principle might be:
"Deliver value early and often."
This principle ensures that teams focus on delivering incremental value rather than waiting for a big-bang release.
Policies
Policies are the what—the rules or guidelines that define how principles are applied in practice. They’re more specific than principles and often provide boundaries or constraints to ensure consistency and compliance.
For example, a policy that supports the principle above might be:
"All Information Products must be delivered in two-week increments."
This policy ensures that teams adhere to the principle of delivering value early and often by setting a clear expectation for delivery timelines.
Patterns
Patterns are the how—the repeatable solutions or approaches to common problems. They’re practical, actionable, and adaptable to different contexts. Patterns provide teams with a toolkit of proven methods to achieve their goals while aligning with principles and policies.
For example, a pattern that supports the principle and policy above might be:
"Use the Information Product Canvas to define and prioritize deliverables."
This pattern provides a structured way for teams to identify and focus on high-value work, ensuring they can deliver in two-week increments.
How They Work Together
Principles set the direction. They ensure everyone understands the overarching goals and values, creating alignment across the team.
Policies provide the guardrails. They translate principles into actionable rules that guide day-to-day decisions and ensure consistency.
Patterns offer the tools. They provide practical methods for teams to execute their work effectively while staying aligned with principles and policies.
Example in Action
Imagine a data team tasked with improving customer retention:
Principle: "Protect sensitive data to build trust and ensure compliance."
Policy: "Access to PII is restricted based on user roles, and all PII must be masked for non-authorized users."
Pattern: "Implement a dynamic data masking pattern for PII."
The principle ensures the team prioritizes data protection. The policy sets clear rules for how PII should be handled. The pattern provides a practical solution for enforcing the policy. Together, they create a cohesive approach that aligns with the team’s goals and values.
Why This Matters
When principles, policies, and patterns are well-defined and aligned, they create a shared language and understanding across the team. This reduces confusion, improves collaboration, and ensures that everyone is working toward the same goals. It also provides the flexibility to adapt to changing circumstances while maintaining consistency and focus.
Ok human Shagility is back on the keyboard:
Probably a 3 out of 5 (or maybe a 2)
It did an ok job of describing my suggested Way of Working in this domain, but not great. (I will write an article to provide my AI Clone better content as a reinforcement pattern for this one)
Principle: "Protect sensitive data to build trust and ensure compliance."
Im ok with this one, its a principle after all so should be high level and aspirational. I liked the use of the language of a compass.
Policy: "Access to PII is restricted based on user roles, and all PII must be masked for non-authorized users."
This is too fluffy it needs to be more rules based.
It needs to say what is defined as PII data, just peoples names?, their identifiers such as their drivers license number?, their address? ….
It needs to define how it should be masked, is hashing that data ok?, if it is what hashing algorithms are ok and what ones are not?, can the hashed PII data be stored in the same row as the original data or does it have to be stored separately? …..
And who is a non-authorised user for this PII data?, what roles in the organisation can and cannot view this data?, what systems or “AI agents” can or cannot have access to this PII data? ….
Ideally it needs to use a pattern based on Policies defined as Code.
Pattern: "Implement a dynamic data masking pattern for PII."
Again this is to fluffy for me. Patterns are good solutions for common problems provided using a shared language. This statement of "Implement a dynamic data masking pattern for PII." is an ideal, its too high level, it’s not actionable, its:
“all mouth and no trousers”
The Pattern needs to be a ‘the’ not an ‘a’. It needs to be something along the lines of:
"Implement the recommended dynamic data masking pattern for PII."
You can find a set of patterns for PII data over at wiki page …… These patterns provide a code solution for how to detect PII data and how to apply the recommended data masking algorithms to that data …..
So how do Data Governance teams adopt a new Way of Working based on this language of Principles, Policies and Patterns?
Data Governance teams need to become either an ‘Implementator’ or an ‘Auditor’
An Implementator Way of Working means the Data Governance team create Patterns that the data teams can use. They become builders and the data teams become consumers of what they have built.
An Auditor Way of Working means the Data Governance team define the Policies, the rules, that the data teams can be tested against. And they then regularly audit them to make sure they are followed.
Governance as an Implementator
An Implementator Way of Working means the Data Governance team create Patterns that the data teams can use. They become builders and the data teams become consumers of what they have built.
The Data Governance team deliver code or tools that data teams can execute. The Data Governance teams become a form of a “platform as a service” team.
For the PII example the Data Governance team provides code or tools to the data teams that automatically detect data that holds PII records.
The data team don’t have to create the patterns that detect these records they just use a blob of code, or call a micro service, or an “AI Agent” that does the work for them.
The Data Governance team provides the RBAC code or service that then surfaces or hides that PII data to different roles in the organisation.
They do the work that integrates into the organisations HR system or any other system that holds the people and the roles for the organisation. They build the rules based engine that dynamically determines what PII data a role can see.
The data team again use this as a blob of code, a micro service or an “AI Agent” they can use to ensure only the authorised people can see that PII data.
The Data Governance team become accountable, responsible, fireable and all the other ‘able’ words that means they carry the can if PII data is ever inadvertently exposed.
Governance as an Auditor
An Auditor Way of Working means the Data Governance team define the Policies, the rules, that the data teams can be tested against. And they then regularly audit them to make sure they are followed.
So the data teams do the implementation, but the Data Governance teams set the rules and they then audit compliance to those rules on a regular basis.
And these rules cannot be fluffy, they are not the equivalent of:
“a driver of a car should not speed”
They are rules that are specific, rules that ideally could be tested via code. They are the equivalent of:
On the road named Main Road between -40.865759005082296, 175.07426825768684 and -40.84819819938461, 175.08569360625913 a vehicle that has a Tar Weight of less than 2079kg and its not towing a Trailer cannot exceed 80km per hour at anytime.
The Data Governance team then act using the same pattern a financial auditor uses to confirm these rules have ben followed.
For our PII scenario they ask the data team to provide examples of systems, users and queries that have accessed data from the data platform. They then audit that data to determine if it held PII data, and if the people or systems who accessed it were allowed to access it.
And of course in the modern “AI” world systems include any ‘Automated AI’ agents and any ‘Assisted AI’ features.
Yes this is a onerous manual task.
Ideally the Data Governance team will create code that they can run to automate this audit.
Ideally ideally they will use the "‘Push Left’ pattern for this and provide this code as a blob of code, a micro service or an “AI Agent” so the data team can run it themselves to ensure they always meet the “rules”.
Thus is a level of maturity I have yet to see a Data Governance team adopt.
A real world example continues
The NZ Police example I referenced earlier has taken an Auditor pattern for “AI” in their organisation.
They state how the audit will be undertaken and by who.
They state how the audits will take place.
They state what will happen if a team fails the audit.
There is teeth to their bite. They are providing the Trousers so to speak.
My only suggestion is given the speed of change in the “AI space they might need to the audits more often than every 6 months.
“AI” is the new Excel
And Data Governance teams are even more fooked as a result.
The NZ Police example I referenced is for an organisation that has IT restrictions and governance in place, restrictions and governance that are not in place for a lot of other organisations.
(Although I do wonder how many people in NZ Police are able to store PII data in Excel?)
Does your organisation physically restrict you from accessing the new wave of “AI” tooling?
Every organisation should treat “AI” as the new version of Excel.
Its a commodity tool that is easy to use, a tool that most knowledge workers have access to , a tool that makes their data work easier compared to the alternatives.
Lets jump over to ChatGPT to see how it thinks Excel and “AI” are following very similar adoption patterns:
Prompt: how is AI similar to excel in terms of its adoption?
Model: ChatGPT o3-mini
Both Excel and AI have transformed how businesses work, albeit in different eras and contexts. Here are some key similarities in their adoption:
1. Democratisation of Complex Tasks:
Excel made data manipulation accessible to non-technical users, turning complex data analysis into everyday tasks for millions. Similarly, AI is increasingly being packaged into user-friendly tools that empower individuals to extract insights and automate processes without needing specialised technical skills.
2. Gradual Integration into Daily Operations:
Initially, Excel was adopted by a niche group of power users before becoming a standard tool in every office. AI is following a similar trajectory, moving from experimental projects and specialised applications to being integrated into core business functions such as customer service, decision-making, and operational optimisation.
3. Catalyst for Efficiency and Innovation:
Just as Excel revolutionised business efficiency by streamlining data management and reporting, AI is driving innovation by automating repetitive tasks, predicting trends, and enabling more informed strategic decisions. Both tools have opened up new ways of working that were previously too complex or labour-intensive.
4. Overcoming Early Skepticism:
Both technologies initially faced resistance. Excel users had to learn a new way of handling data, while many organisations are still evaluating the impact and ethical considerations of AI. Over time, as users became more familiar with the capabilities and realised the tangible benefits, adoption increased significantly.
These similarities underscore how transformative technologies often follow a similar path: start as specialised tools, gradually become mainstream by making complex tasks simpler, and eventually reshape the way businesses operate.
Ok human Shagility is back on the keyboard:
I have seen few Data Governance teams manage the amount of data stored and used in Excel using the traditional data governance approaches.
I think they have a shit show of those approaches working in the “AI” world we now find ourselves in.
I hope Data Governance teams focus on how to adapt their Way of Working for the new world.
The work they try and do is valuable. And with the emergence of “AI” and what it can potentially do, that work is going to become critical.
«oo»
Stakeholder - “Thats not what I wanted!”
Data Team - “But thats what you asked for!”
Struggling to gather data requirements and constantly hearing the conversation above?
Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.
Great article, in particular the part about data governance needs to become less handwavy and provide specific tools/pattenrs.