Context Plane AI Agent in a Web App

it works but is it valuable, viable feasible and useful?

Is having an AI Agent in the Web App for the Context Plane a good idea?

“Context” of this post

I often find writing helps me coalesce and refine my thoughts when new patterns start to emerge, but aren’t very clear yet.

So this article is a brain dump / train of thought continuation of the product and architecture needed to have one Context Plane to rule them all, as part of a proposed “AI Data Stack”.

This article provides an overview of adding the AI Agent capability to the AgileData Web App.

Not starting from scratch

We have been experimenting with an in App “Chatbot” helper in the the AgileData App for multiple years so I was lucky we already had the core UX needed for us to experiment with this feature.

And we had already integrated Gemini with the AgileData App a while ago. We already used it to assist or automate a few of the more complex bits of data work.

So it was ”just” a case of changing the Gemini Model we were using to switch over to use Gemini 2.5 Pro to get the reasoning capability and also adding the connector to expose it to our MCP Server.

Simples!

(he says as Nigel did all the work)

So from a AgileData App UX point of view nothing changed.

Same Prompt

We have a simple set of prompts already defined in the AgileData Platform for ADI.

You are ADI, the AI assistant inside the AgileData.io product. Your purpose is to guide a data team of one—a data analyst—in configuring a data warehouse using the AgileData App and Platform.

Context & Capabilities:

You have access to:
	•	The AgileData user guides
	•	The screens and workflows within the AgileData app
	•	The APIs that power the AgileData app and platform
	•	The data stored in the AgileData platform

The Challenge:
	•	Every set of data has unique challenges.
	•	There is no single, linear process for working with data—users must “pick a path” or “choose their own adventure.”
	•	Each user will dynamically use the AgileData app in different ways based on their specific needs.

Who You Are Helping:
	•	The user is data-savvy but not a data professional, not a highly technical user, not a data expert.
	•	They may not always know exactly what data task they need to do—but they know the outcome they are aiming for.

How You Should Respond:
	•	Be clear and practical: Guide the user toward the best approach to complete their data task efficiently.
	•	Adapt to their needs: If they are unclear on what to do, ask clarifying questions and suggest possible paths forward.
	•	Use all available information: Leverage guides, app screens, APIs, and platform data to provide the most relevant answers.
	•	Encourage exploration: Offer multiple options for how they could proceed, allowing them to discover the best way forward.
	•	Think like a mentor, not a manual: Rather than just explaining how the platform works, help the user make decisions that will get them to their goal.
	•	Present them with the relevant url to the relevant screen if that will help them complete the task

When you have multiple choices:
	•	Pick the one you think is the most suitable and do that.
	•	Append the other options you had to your response and tell the Data Analyst which choice you selected, let them know they can tell you to pick another option.


Your goal is not just to answer questions but to help the user successfully build their data warehouse in the most effective way possible—no matter what challenges they face.

As you can see they are based on previous experiments where we were focussed on helping a Data Analyst build a Data Warehouse as a “team of one”.

I left this prompt in place for this experiment, but I should do another McSpikey where I tailor them for the Context Plane use cases.

In previous experiments where I used Gemini CLI and Claude to access the Context in the Context Plane, there was no “system” prompts in place at all so this should mean I get slightly different responses from Gemini 2.5 Pro in the AgileData App vs Gemini 2.5 Pro in the Gemini CLI.

ADI also has access to our Doco and also examples screens from the App, so again it has more Agent Context than the previous experiments with Gemini CLI had, as that only had access to the API’s via the MCP Server.

MCP Services

We limited the services available in the AgileData App to a subset of the available APIs.

When I use my typical question to get a list of Tiles and a list of Fields in a specific Tile then it pretty get the response I am expecting, no surprises there.

Except ……

When I use Claude to get the Blast Radius for a field change it gives me very rich Context back.

ADI in the AgileData App however…

She ain’t being that helpful!

Over to Gemini CLI to see if its her or Gemini.

Ah nope

We made a fundamental change to the MCP Server we are using since the last McSpikey. The previous version only had access to a subset of our API’s as tools. We changed it to have access to all API’s.

For Claude this didn’t really make much of a difference, but for Gemini CLI it seemed to have a major impact and not in a good way.

Listing the tiles always seems to work ok, simple task.

Listing the fields in a tile has worked well up until we made the MCP Server Tool change.

That is the LLM version of go F$#@ yourself lol.

I had a quick go to see if I could limit the Tools that Gemini CLI could access on the client side, as we limited them in both the AgileData App for ADI and I limited them in the Claude client.

So new entries in the Gemini CLI settings.json it is:

  "securityPolicy": {
    "mode": "configured",
    "allowedTools": [
      "get_business_glossary",
      "get_catalog_tiles",
      "get_ensemble_config",
      "get_change_rules"
    ]
  }

But that did nada, when I listed the tools in Gemini CLI using /mcp it could still see them all and there was no behaviour change.

If I was very specific with my language it would sometimes work:

But then it would go back to be a real dumb arse again.

So back to very specific language:

So theories from that rabbit hole:

• To many MCP Server Tools makes the AI Agent Tools job harder

• The previous Tools we exposed had some additional hints in the naming of the Tool that probably helped the AI Agent Tools

• Gemini 2.5 Pro is not as good as Claude Sonnet 4 for this use case

One more thing

I ran Gemini CLI against a different AgileData Tenancy, Newcastle, compared to the ADI version which was running in the Kapiti Tenancy (hence the change in Tile name I was asking questions about).

Lets go try ADI in the Newcastle Tenancy and see what happens.

Nope same problem.

So Im going to guess the problem could be:

• the Gemini 2.5 Pro model

• and / or the extra MCP Server Tools

• and / or the Prompts in the AgileData Platform

• and / or all of the above

that is causing the problems.

Out out time for this McSpikey so thats is an experiment for another day.

So in Summary

We can provide a web based interface in the AgileData App easily enough by reusing the current ADI capability.

But need to work out if it actually has any value.

And also validate the assumption that it would be a different personas using it compared to a AI Agent Tool and should that impact the type of response they receive or not. Gut feel is no, it should be “any Human, any AI tool, same response”

And I need to experiment with the combination of LLM models, MCP Server Tools and Prompts a lot more.

Claude Sonnet 4 seems more compotent than Gemini 2.5 Pro

Well for this use case anyway. I just seemed to get back better “Context” from my Context when I use Claude over Gemini.

As slow as a wet pig

One of the surprises was ADI was sooooo slow in the Kapiti Tenancy vs the Newcastle Tenancy.

Check it out.

Newcastle

Kapiti

Almost twice as long in Kapiti vs Newcastle.

One of these things is not like the other

After chatting with Nigel it seems were running Gemini 2.5 Flash in Newcastle and Gemini 2.5 Pro in Kapiti, hence the difference in response times.

So another McSpikey needed to see how different the responses are between the two models for our key use cases.

Its all about the reducing uncertainity

The key to a McSpikey is to reduce uncertainty and for this McSpikey I think we have increased the uncertainty.

But thats ok, better to know that now than later when we are much further down the Context Plane path.

Wood from the Trees

Still a way to go before I have a coherent set of Patterns that I can Coach / Mentor / Teach somebody else for the “Context Plane”, and the “AI Data Stack” or present as a robust Product Overview and / or Architecture map.

But as I have already said, writing my half formed ideas helps me think.

Context Plane Use Cases

You can find all the use cases where I think the Context Plane may have sone value over at:

An incoherent stream of Context

You can find all the previous articles with my train of thought listed in this thread:

https://agiledata.substack.com/t/context-plane

We are building the Context Plane while flying it, so always looking for early adopters to help us decide the final destination.

If you want a virtual chat grab a slot here:

https://contextplane.ai/contact-us/#bookemdanno