An Alternate Agentic AI Architecture
(It’s About the Data)

Contemporary enterprise copilots and knowledge assistants such as Microsoft 365 Copilot [11], Glean [17], and emerging open agentic systems such as OpenCLAW [14] typically follow an ingestion-and-indexing paradigm. These systems connect to heterogeneous enterprise data sources (e.g., CRM, ERP, file systems), extract documents or records, compute embeddings or sparse indices, and expose a unified semantic search interface over otherwise disconnected silos. While this architecture improves discoverability, it does not integrate the underlying data models. Put differently, a text-oriented architecture is not adept at performing semantic joins.

RUBICON instead treats retrieval as a virtual data integration problem in the classical database sense [9]. It assumes a collection of (often large) data sources that are left insitu with whatever indexing is currently in place. Access control is provided by whatever system manages the data store, as noted in the architectural diagram of Figure 1. Integration occurs at the query layer, not by physically consolidating data into a new store.

In order to join such data sources, the obvious approach (widely used today) is to have a data source-specific wrapper (connector) that will convert the bespoke APIs for the data sources into a common one. In our opinion, a requirement for this interface is to facilitate data integration (joins) and SQL would be a natural choice. However, using Text-to-SQL to convert a human request into SQL is very likely unworkable in an enterprise setting. Instead, we propose a stripped-down SQL-like notation, called AQL (Agentic Query Language) described in Section  2.3. The goal of a wrapper or agent is to translate AQL into the bespoke interface actually supported. We assume that wrappers will be constructed locally by enterprise personnel.

As noted in Figure 1, on top of AQL there is a query processing module, which can run in either of two modes: interactive mode and compiled mode. In interactive mode, a user issues a single AQL command and then inspects the result before proceeding. Since RUBICON can easily “go off the rails,” this allows the user to direct processing toward a final result. Using this mode, we expect the user to execute one command at a time until the desired goal is reached. Each issued command produces an explicit intermediate table that is visible to the user. These intermediate results are first-class artifacts: they can be inspected, saved, reused, or joined with subsequent queries. Because execution proceeds through concrete relational representations rather than hidden conversational state, the resulting workflow is reproducible. The user is not dependent on regenerating an answer through repeated LLM reasoning but instead operates over stable, inspectable query outputs.

In contrast, compiled mode executes an entire sequence of AQL commands as a single composite plan. If the user plans to run the same task again, then RUBICON has the complete command sequence and can invoke a relational-style query optimizer to obtain the answer more rapidly. In effect, interactive mode corresponds to interpreted execution, whereas compiled mode corresponds to relational compilation. Of course, compiled mode can also be invoked directly by specifying a more complex interaction from the outset. One of the strengths of RUBICON is that relational compilation is vastly cheaper and more predictable than repeated LLM processing.

On top of this processing layer sits a user interface. It is possible that this interface is natural language, processed by an LLM. However, our first version will use a graphical user interface layered above AQL. It will build on the form-based relational interfaces of the 1980s, such as Query-by-Forms (QBF) [7] and Query-by-Example (QBE) [21].

In this sense, RUBICON functions as connective tissue across enterprise systems. Rather than constructing a loosely connected graph of embeddings, the system supports a structured relational substrate in which entities, attributes, and relationships are explicitly represented and queryable in a SQL-like notation.

This architectural choice has two important consequences:

• •

  Structured Datastore over Disconnected Indices: Retrieval operates over a logically integrated relational schema constructed at query time via wrappers, enabling joins, constraints, aggregation, provenance tracking, and transactional guarantees.

• •

  Dual Language of Interaction: AQL provides declarative programmability using a GUI and will investigate NL in the future.

Multimodal data sources (e.g. structured records, free text, images, video) may require sophisticated wrappers. For example, a video wrapper would be an object detection pipeline (e.g., YOLO-style detectors [15]) transform visual data into relational tuples (object, attribute, timestamp, source). A textual corpus would likely be supported a word-oriented text system such as Lucene, by a sparse lexical representations [16] or dense vector embeddings [12]. In any case, the wrapper will transform AQL to the native query language of the specific interface.

Crucially, the wrapper must present a relational view, even if none exists natively. For example, an e-mail system can be exposed as a table Message(from, to, subject, date, body, ...) where the underlying search engine evaluates the WHERE clause and returns matching rows. Likewise, Wikipedia can be exposed as a table Page(title, url, snippet, text, categories, ...) even if the underlying interface is keyword or vector search.

These tables are not materialized relations; they are logical views constructed by the wrapper. This normalization step is essential. Without rows and columns, downstream operators such as joins and unions have nothing well-defined to operate on.

Our experience dealing with enterprise data warehouses is that full NL text cannot be translated to SQL with sufficient accuracy to be workable in production. At some point in the future, the outlook may be more rosy, but for now we need to specify a SQL subset. Accordingly, AQL is a restricted query language that serves as the common notation between users and data-source wrappers. Our major simplification is to require the user to say what attributes from what table they desire and then to generate a natural language predicate, as follows:

Find <column(s)>
FROM <table>
WHERE <NL utterance>

Given our experience with data warehouses, this seems a required simplification.

Modern enterprises operate in highly fragmented information environments. Organizational knowledge is distributed across heterogeneous systems, each optimized for a specific operational purpose. These systems typically include:

• •

  Relational Databases: Structured transactional and analytical data stored in data warehouses and OLTP systems.

• •

  Documentation Repositories: Internal knowledge bases, policy documents, technical documentation, and reports stored in content management systems.

• •

  Application Platforms: Enterprise systems such as CRM, ERP, or HR solutions, each exposing application-specific APIs.

• •

  Email and Communication Systems: Semi-structured correspondence that often contains operational decisions, updates, and contextual clarifications.

• •

  External Public Knowledge: Web content and publicly accessible information sources that shape external representations of the organization.

• •

  Geoinformation and CAD Systems: Spatial databases and engineering design repositories that manage geospatial layers, maps, 3D models, and CAD drawings.

Specifically, our benchmark contains the following five data sources:

• •

  Wikipedia, accessed via public API calls.

• •

  University Data Warehouse (DW). We have access to 97 tables from an institutional administration data warehouse. Most of the tables concern facilities (buildings, room use, etc.). We have anonymized the data in these tables and copied it into a SQLite database.

• •

  University research laboratory website. This site has information on lab personnel, research activities, events, etc.

• •

  Email system (Gmail[6]), accessed via API calls,

• •

  The Pile / LLM knowledge, representing general pretrained language knowledge accessible through foundation models.

Each source exposes a distinct interface and data modality (structured tables, web documents, private communication, or pretrained knowledge). No source shares schema-level integration with the others. As a result, cross-source reasoning requires explicit coordination rather than implicit joins or shared schema assumptions.

We constructed a workload of seven expert-designed queries (Q1–Q7). Each query is deliberately designed to require integration of exactly two relevant sources. The remaining three sources are irrelevant for answering the query and serve as distractors. Table 1 shows the ground-truth relevance matrix.

This design enforces three structural properties:

• •

  Mandatory source selection: Correctly answering a query requires identifying both relevant sources.

• •

  Cross-source reasoning: No query can be answered using a single source alone.

• •

  Distractor sensitivity: Invoking irrelevant sources increases cost and may introduce hallucinated or incomplete information.

The benchmark is intentionally small and fully disclosed to allow transparent inspection of system behavior at the per-query level in addition to aggregate metrics.

Ground-truth answers were established through manual expert retrieval across the required sources for each query. For every $q\in Q$, we verified that:

• •

  the required sources contain sufficient information to derive the correct answer, and

• •

  the irrelevant sources do not contain sufficient standalone information to produce a correct result.

Each final answer was validated to ensure consistency across the two required sources. This controlled construction allows systematic diagnosis of coordination failures, including omitted sources and redundant exploration.

We evaluate each system along correctness (accuracy), token usage, provider-reported monetary cost, latency, and coordination complexity.

We evaluate three representative state-of-the-art foundation models and compared their performance to RUBICON:

• •

  OpenAI GPT-5-mini[13],

• •

  Google Gemini-3-flash-preview[5],

• •

  Anthropic Claude-Sonnet-4.6[1].

All models are configured with the highest available reasoning settings (e.g., long-context mode, increased reasoning effort and web search). We deliberately focus on “mini” or mid-tier foundation models rather than the largest available variants. In enterprise settings, query processing must be cost-predictable and scalable. Architectures that require invoking frontier-scale models for every intermediate step incur substantial latency and operating expense, and are therefore unlikely to be viable in production. Our objective is to demonstrate that the proposed system achieves competitive accuracy while reducing reliance on large, opaque models. If acceptable performance can be obtained with smaller models, the resulting solution is more economical, repeatable, and deployable at scale.

We evaluate two prompting styles:

• •

  Natural Language (NL): The query is submitted exactly as written.

• •

  AQL-style Prompting: The query is reformulated into a structured pseudo-query representation intended to guide reasoning.

We compare two orchestration settings against RUBICON: a standard single-shot interaction (“Vanilla LLM”) and a ReAct-style agent[19] implemented using LangChain[3]. Across both settings, all baselines use identical foundation models and model configurations. The only difference is whether structured tool access and iterative reasoning are enabled.

Table 2 reports per-query correctness across all systems and prompting styles. A result is marked as correct (C) only if all required sources are consulted and the final answer fully matches the ground truth. All other outcomes are labeled incorrect (I).

Across all seven benchmark queries, neither vanilla LLMs nor the LangChain ReAct agent achieve a single correct result. This holds across all models (GPT, Gemini, Claude) and both prompting styles (NL and AQL). Overall accuracy for these configurations is therefore 0%.

Importantly, errors are not dominated by hallucinated content. Instead, failures consistently stem from incomplete multi-source coordination: models either omit at least one required source, terminate prematurely, or fail to correctly integrate results across sources. Even when some relevant information is retrieved, required joins are not completed.

Providing tool access via the ReAct agent does not alter this outcome. Despite being able to invoke all available sources, no configuration reliably enforces completeness over the required source set. Tool availability alone does not guarantee correct coordination.

For example, in the “research lab Turing or Nobel laureates” query (Q3), both the vanilla LLM configurations and the LangChain ReAct agent exhibit the same coordination failure. In many runs, the model retrieves award recipients from Wikipedia but never restricts the result correctly to research lab professors. In other runs, the enterprise database is queried but award status is not verified through Wikipedia. In no configuration does the system reliably consult both required sources and complete the necessary join.

In contrast, RUBICON achieves 100% accuracy, correctly answering all seven queries. Because RUBICON enforces explicit source selection and deterministic execution over required sources, it eliminates omission-based failures.

To avoid confounding coordination errors with long-context degradation, we intentionally do not inject enterprise data directly into the context window of vanilla LLMs. Prior long-context evaluations (e.g., BEAVER[4]) demonstrate that excessive context length can degrade reasoning quality due to context dilution. Our setup therefore isolates pretrained knowledge and explicit tool-based coordination, ensuring that observed failures reflect coordination limitations rather than context saturation effects.

Overall, the results demonstrate that the dominant bottleneck is not language understanding but reliable multi-source completeness. Without explicit coordination constraints, even tool-augmented LLM agents fail systematically.

Table 3 reports average input tokens ($\overline{T}_{in}$), output tokens ($\overline{T}_{out}$), tool calls ($\overline{k}$), monetary cost ($\overline{C}$), and time-to-first-token (TTFT, $\overline{t}_{TTFT}$).

Figure 2 characterizes how tool usage is allocated across sources for each query and model.

Even for a single user question, the AQL layer admits multiple semantically similar execution plans with dramatically different cost profiles. Consider the query $Q3$:

NL:

"Which research lab professors at the university have won a Turing Award or a Nobel Prize?"

AQL (award-first):

FIND laureate_full_name, award_name
FROM WIKIPEDIA
WHERE people associated with ’Turing Award’ or ’Nobel Prize’
JOIN
FIND full_name
FROM UNIVERSITY_DW.faculty
WHERE the person is a professor in the research lab

AQL (faculty-first):

FIND full_name
FROM UNIVERSITY_DW.faculty
WHERE the person is a professor in the research lab
JOIN
FIND full_name, award_name
FROM WIKIPEDIA
WHERE for each professor, determine whether their page indicates a ’Turing Award’ or ’Nobel Prize’

This question can be expressed through at least two alternative plans.

Plan A (award-first). First extract the set of Turing and Nobel laureates from Wikipedia, then join this candidate set with research lab faculty from UNIVERSITY_DW.

Plan B (faculty-first). First enumerate all research lab faculty from UNIVERSITY_DW, and for each faculty member retrieve and scan the corresponding Wikipedia page to determine whether it mentions a Turing Award or Nobel Prize.

While both plans aim to compute the same result, their execution characteristics differ fundamentally.

Plan B induces an $\mathcal{O}(|\text{research lab professors}|)$ pattern of external lookups: for every faculty member, a Wikipedia retrieval and long-text scan is required. In an LLM-mediated setting, each probe translates into tool invocations, token consumption, and latency. If faculty pages are long or require multiple retrieval attempts, the cumulative cost increases proportionally with the number of faculty members.

Plan A instead attempts to push down a highly selective predicate (“award recipient”) into Wikipedia before joining with research lab faculty. Since the number of Turing and Nobel laureates is relatively small compared to the number of faculty members, the number of external probes and subsequent joins is substantially reduced. This mirrors classical database join-order optimization: applying selective filters early can shrink intermediate results and avoid expensive nested access patterns.

Importantly, the difference between these plans is structural rather than linguistic. Both can be expressed correctly in AQL; both are logically valid. Yet their token usage, tool-call count, and latency can diverge significantly depending on join order and access path. In LLM-centric multi-source systems, where external retrieval and long-context processing are expensive, such plan sensitivity directly translates into monetary cost and response time.

Our experiments already demonstrate that increasing autonomy expands the cost surface through additional reasoning steps and tool calls. The example above highlights a complementary issue: even under perfect coordination, execution cost remains highly dependent on plan structure. This suggests that multi-source LLM systems require an optimizer (or at minimum a cost-aware planner) that can choose access paths and join orders using source statistics, cardinality estimates, caching strategies, and tool pricing information. Without such mechanisms, greater autonomy risks translating into higher spend and slower responses without corresponding gains in correctness.

We present this example as a motivating illustration of query-plan sensitivity and leave a full empirical cost comparison of alternative AQL plans to future work.

We observe four dominant error modes across systems:

Across all experiments, we find: (i) stronger prompting and increased reasoning effort do not compensate for missing or omitted sources, (ii) higher cost and latency do not reliably predict correctness, and (iii) the dominant limitation is coordination—ensuring source completeness and deterministic multi-source integration rather than increasing model-side deliberation.