We might be focusing on the wrong metric in the enterprise AI debate.
Much of the timeline is arguing over "token budgets"—whether an agent should use 10k or 100k tokens per task. But token burn is usually just a symptom.
The deeper issue is what happens before the model starts reasoning: how we handle context.
Right now, many enterprise deployments rely on live "exploratory retrieval."
An agent connects to a dozen apps via MCP, receives a prompt, and trial-and-errors its way through enterprise data:
1. Querying Jira, getting partial context, and retrying.
2. Searching Slack threads and risking hallucinations.
3. Resolving permissions on the fly—turning a hard security boundary into a probabilistic guess.
That multi-turn loop becomes a recurring tax on latency, cost, and reliability.
### "But how do you pre-compute context for unpredictable queries?"
This is the most common pushback. Real enterprise work is unpredictable, and you can't anticipate every prompt.
The key distinction: you don't pre-compute the destination—you pre-compute the map.
Think of Google Maps. It doesn’t pre-calculate every drive you will ever take. But it does map the roads, link intersections, and flag private, gated streets ahead of time. When you enter an ad-hoc destination, it calculates the optimal route in milliseconds.
Relying purely on live tool-calling is like dropping an agent in a city with no map—forcing it to stop at every intersection to ask external APIs for directions until it finds the answer.
### The Division of Labor
To make agents reliable, we need a clear split:
* Pre-Computed Upstream: Cross-tool identity mapping (Jira user = Slack handle = GitHub ID), relationship linking (Zendesk ticket → Slack debug thread → GitHub PR), and zero-trust permission boundaries.
* Handled at Runtime: When an ad-hoc query arrives, the agent still fetches data in real time—but instead of guessing across APIs, it traverses a pre-structured graph to pull the exact, permission-verified context in one step.
### The Long-Term Moat
Federated tool-calling and MCP serve a purpose for simple, dynamic lookups. But for deep reasoning across tools, leaving identity, security, and relationship mapping to a live LLM reasoning loop introduces too much friction.
The long-term advantage won't just be who has the biggest context window, but who builds permission-aware context layers upstream.
Let frontier models spend their compute doing what they do best: solving complex problems, not rediscovering enterprise structure on every turn.
How is your team balancing live tool-calling with upstream indexing? Are you seeing a shift toward structured context layers?
#EnterpriseAI #AIArchitecture #DataEngineering #LLM #FutureOfWork
