Skip to content

feat(tool): run tool calls in the caller's task - #17

Open
jakedipity wants to merge 4 commits into
jakedipity/fix-vector-store-testsfrom
jakedipity/inline-tool-calls
Open

jakedipity wants to merge 4 commits into
jakedipity/fix-vector-store-testsfrom
jakedipity/inline-tool-calls

Conversation

@jakedipity

@jakedipity jakedipity commented Oct 15, 2026 •

Copy link
Copy Markdown

Breaking changes

  • Removed ToolServer::handle_message, ToolServerRequest, ToolServerRequestMessageKind and ToolServerResponse
  • Removed ToolServerError::{Canceled, SendError, InvalidMessage}
  • Added ToolServerError::DefinitionsFailed, and ToolServerError is now #[non_exhaustive]
  • call_tool returns the tool set's error with its variant intact, and its text drops repeated ToolCallError: prefixes (Toolset error: ToolNotFoundError: missing)
  • ToolSet::call and call_tool return a panicking tool's panic as an Err, except on wasm
  • Dropping the caller drops an in-flight tool call

feat(tool): run a tool call in the caller's task

ToolServerHandle::call_tool asks the tool server for the tool and runs it in the calling task. The server task used to run every call itself, one at a time. A call now sees the caller's task-locals and tracing span, calls from different callers run at the same time, and a call stops when its caller stops waiting on it. The server still adds and removes tools and serves definitions.

Tools are held behind Arc so the server can hand one out. A failed call keeps its error text word for word, and a panicking tool fails only its own call, with the panic's message as the error. The span carried on the CallTool message goes away with that message, since the caller's span is now current where the tool runs.

Ref: mezmo/aura#808


refactor(tool): share the tool server's state instead of running a task

ToolServerHandle holds the tool server behind a lock, where it used to send each request over a channel to a task that owned the server. Adding, removing and looking up a tool take the lock briefly. Fetching tool definitions works on a copy taken under the lock and awaits outside it, so a tool call never waits behind a definitions fetch.

With no task there is nothing to stop serving. A caller that stops waiting, or a dynamic tool's vector store returning an error, no longer leaves every later call on the handle failing, and a failed definitions lookup comes back as ToolServerError::DefinitionsFailed. The request and response message types, the error variants for a failed channel, and the wasm-bindgen-futures dependency that spawned the task on wasm are removed.

A failed call returns the tool set's error as it is, so a caller can match its variant or downcast the tool's own error. Its text drops the ToolCallError prefixes the channel added. ToolSet::call shares the call path, so a panicking tool returns an Err there too, except on wasm where a panic aborts. ToolServerError is non_exhaustive.

Fixes: mezmo/aura#811


fix(tool): log a tool call's arguments as readable JSON

The "Calling tool" debug log pretty-printed the arguments as a JSON string, so they showed escaped and quoted, like "{\"x\":2}". They are now parsed and pretty-printed as JSON, or logged as they are when they are not JSON.

Fixes: mezmo/aura#812


fix(tool): list a tool once when it is added again

Registering a tool under a name that was already listed, through ToolServer::tool, rmcp_tool, the agent builder's list, or ToolServerHandle::add_tool, replaced the tool but listed its name a second time. Tool definitions then named it twice, and providers reject a request whose tool names are not unique. Each path now lists a name once.

@jakedipity jakedipity changed the title Jakedipity/inline tool calls feat(tool): run tool calls in the caller's task Oct 15, 2026
@greptile-apps

greptile-apps Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

Tool calls now run in the caller’s task, using shared tool references instead of a background server task.

  • Tool calls run in the task that requested them.
  • Tool definitions load without holding up tool calls.
  • A tool name appears once when the tool is added again.
  • Tool-call logs show valid JSON arguments in readable form.

No new actionable issues were found.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Caller] --> B[Take read lock]
  B --> C[Clone tool reference]
  C --> D[Release lock]
  D --> E[Run tool in caller task]
  E --> F[Return result or caught panic]
  A --> G[Clone server under read lock]
  G --> H[Release lock]
  H --> I[Fetch definitions from copy]
Loading

Reviews (3) · Last reviewed commit: "fix(tool): list a tool once when it is a..." · Reviewed by Greptile

@jakedipity

Copy link
Copy Markdown
Author

The failing rig-mongodb and rig-neo4j integration tests predate this PR, which only touches rig-core's tool server. The test job has not passed on this fork before: neo4j's latest image no longer lists indexProvider, which get_index required, and the mongo test's constant embeddings tie every document at a score of 1.0. #18 fixes both; I'll rebase this onto mezmo once it merges.

ToolServerHandle::call_tool asks the tool server for the tool and
runs it in the calling task. The server task used to run every call
itself, one at a time. A call now sees the caller's task-locals and
tracing span, calls from different callers run at the same time, and
a call stops when its caller stops waiting on it. The server still
adds and removes tools and serves definitions.

Tools are held behind Arc so the server can hand one out. A failed
call keeps its error text word for word, and a panicking tool fails
only its own call, with the panic's message as the error. The span
carried on the CallTool message goes away with that message, since
the caller's span is now current where the tool runs.

Ref: mezmo/aura#808
Signed-off-by: Jacob Hull <jacob@planethull.com>
@jakedipity
jakedipity force-pushed the jakedipity/inline-tool-calls branch from 95ac4bb to e68fc44 Compare October 20, 2026 20:12
@jakedipity
jakedipity changed the base branch from mezmo to jakedipity/fix-vector-store-tests October 20, 2026 20:13
Comment on lines +207 to 221
/// Runs `tool_name` in the caller's task, so the call sees the caller's
/// task-locals and span, and calls from different callers run at the same
/// time.
pub async fn call_tool(&self, tool_name: &str, args: &str) -> Result<String, ToolServerError> {
let (tx, rx) = futures::channel::oneshot::channel();

self.0
.send(ToolServerRequest {
callback_channel: tx,
data: ToolServerRequestMessageKind::CallTool {
name: tool_name.to_string(),
args: args.to_string(),
caller_span: tracing::Span::current(),
},
let tool = self.read().toolset.get(tool_name).cloned();
call_tool_named(tool_name, tool.as_ref(), args.to_string())
.await
.map_err(|err| {
// Callers match on this text. The tool set's error is
// stringified inside a tool call error.
ToolServerError::ToolsetError(ToolSetError::ToolCallError(
ToolError::ToolCallError(err.to_string().into()),
))
})
.await?;

let res = rx.await?;

match res {
ToolServerResponse::ToolExecuted { result, .. } => Ok(result),
ToolServerResponse::ToolError { error } => Err(ToolServerError::ToolsetError(
ToolSetError::ToolCallError(ToolError::ToolCallError(error.into())),
)),
invalid => Err(ToolServerError::InvalidMessage(invalid)),
}
}

@justintime4tea justintime4tea Oct 22, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running the tool future inside the caller's task changes what happens when the caller stops waiting. If a consumer drops the MultiTurnStream, or a timeout fires around call_tool, the tool future is dropped mid-flight. The old server task always ran a call to completion.

Dropping the future stops the local work but not anything already in flight elsewhere. A tool doing multi-step side effects (DB writes, external calls) can be left half-done, with the result lost and no tool_result recorded. For an MCP tool this is true even when the server supports an MCP cancel request, because nothing on our side sends one: rmcp's call_tool goes through send_request, which consumes the RequestHandle, and RequestHandle has no Drop impl, so no notifications/cancelled is ever sent. The server keeps running the call until it finishes on its own.

dropping_the_caller_stops_the_tool locks this in as intended, which is fine, but the Tool docs only say calls may run concurrently.

Suggestion:

Include something along the lines of "a call may be dropped at any await point, so a tool with side effects that must complete should spawn its real work onto a detached task rather than run it inline" to the comment/doc on Tool::call. Call this out in the PR description/commit as a behavior change.

@justintime4tea justintime4tea Oct 26, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jakedipity , @Shearerbeard , coming back to this after talking with Mike and digging further into AURA consuming Rig, the previous suggestion doesn't make sense and would essentially add "dead" code with regards to AURA since AURA never makes calls using McpTool::call. The suggestion of a CancelOnDrop.

It's a bit confusing because I anchored the comment here when it relates to McpTool::call which was never changed. My bad.

I've updated the comment/suggestion and removed the CancelOnDrop suggestion but it could still be useful to add a comment/doc change to Tool::call here:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, the drop introduced does reach AURA in one configuration. A single agent with no HITL, scratchpad, or nudge layer has unwrapped MCP tools, so a dropped stream now drops the call inside AURA's call_tool_tracked with its in-flight entry still registered. That's AURA's to fix and should be tracked on mezmo/aura#808, not something for this PR.

Comment on lines +189 to 192
.collect()
};

drop(replaced);
Ok(())

@justintime4tea justintime4tea Oct 22, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

append_toolset inserts into toolset.tools but never calls list_static, so tools added this way never land in static_tool_names and get_tool_defs never returns them. The model is never told these tools exist.

add_tool just above and ToolServer::tool both list the name. The asymmetry was already there in the old AppendToolset arm, but the rewrite keeps it and the new tests don't cover it. I think the fix is in the filter_map closure that currently borrows server mutably for the insert, so we probably need to call list_static in the same closure or in a separate loop before it.

Perhaps we could call server.list_static(name.clone()) inside the filter_map, and add a test that appends a toolset and asserts get_tool_defs(None) includes it?

@justintime4tea justintime4tea Oct 26, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Downgrading this to optional/nit. AURA only registers tools through the builder and never calls append_toolset or remove_tool on the handle, so the gap can't bite us (AURA). Still a one-liner if you want it; otherwise fine to leave.

Comment thread rig/rig-core/src/tool/server.rs Resolved
Comment on lines +567 to +576
// A panicking tool fails its own call. The caller carries on.
let result = std::panic::AssertUnwindSafe(tool.call(args))
.catch_unwind()
.await
.unwrap_or_else(|panic| {
Err(ToolError::ToolCallError(
format!("tool `{toolname}` panicked: {}", panic_message(&panic)).into(),
))
});
Ok(result?)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

catch_unwind here only wraps polling of the future that ToolDyn::call returns. A hand-written ToolDyn::call that panics synchronously before returning Box::pin (say serde_json::from_str(&args).unwrap() outside the async block) panics on the tool.call(args) expression, before AssertUnwindSafe(...) is even built. That still unwinds the caller's task, which the comment above says can't happen.

What if we catch the construction as well?

let fut = match std::panic::catch_unwind(AssertUnwindSafe(|| tool.call(args))) {
    Ok(fut) => fut,
    Err(panic) => return Err(ToolError::ToolCallError(
        format!("tool `{toolname}` panicked: {}", panic_message(&panic)).into(),
    ).into()),
};
let result = AssertUnwindSafe(fut).catch_unwind().await.unwrap_or_else(/* same */);

Or if we're fine with the gap, how about the comment says "a panic while running the tool" rather than implying construction is covered too? Or maybe include, explicitly, that construction is not covered? The macro-generated ToolDyn impls are safe either way.

Comment on lines +567 to +575
// A panicking tool fails its own call. The caller carries on.
let result = std::panic::AssertUnwindSafe(tool.call(args))
.catch_unwind()
.await
.unwrap_or_else(|panic| {
Err(ToolError::ToolCallError(
format!("tool `{toolname}` panicked: {}", panic_message(&panic)).into(),
))
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

catch_unwind here only wraps polling of the future that ToolDyn::call returns. A hand-written ToolDyn::call that panics synchronously before returning Box::pin (say serde_json::from_str(&args).unwrap() outside the async block) panics on the tool.call(args) expression, before AssertUnwindSafe(...) is even built. That still unwinds the caller's task, which the comment above says can't happen.

How about we move the call inside the block that's already caught?

let result = std::panic::AssertUnwindSafe(async { tool.call(args).await })
    .catch_unwind()
    .await
    .unwrap_or_else(/* same as now */);

Construction then happens on the first poll, under the same catch_unwind, so a panic before Box::pin is handled like a panic after it. No second AssertUnwindSafe needed, and it makes the same promise about shared tool state as the current code does. A test with a tool that panics in ToolDyn::call itself would pin it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Downgrading this to optional/nit. AURA's tools go through rig's Tool blanket impl, which can't panic before Box::pin, so this doesn't affect the fork's consumer.

Comment thread rig/rig-core/src/tool/server.rs Resolved
Comment on lines +609 to +630
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn removing_a_tool_does_not_hold_up_other_calls() {
let handle = ToolServer::new().tool(SlowDrop).tool(Adder).run();
let remover = tokio::spawn({
let handle = handle.clone();
async move { handle.remove_tool("slow_drop").await }
});
tokio::time::sleep(Duration::from_millis(20)).await;

// A blocked lock stalls the thread itself, so a timeout around the
// call would never fire. Measure the call instead.
let start = std::time::Instant::now();
let call = handle.call_tool("add", r#"{"x": 1, "y": 2}"#).await;
let waited = start.elapsed();

assert_eq!(call.unwrap(), "3");
assert!(
waited < Duration::from_millis(100),
"the call waited {waited:?} for the removed tool to drop"
);
remover.await.unwrap().unwrap();
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This asserts a timing bound rather than the property. On a loaded CI runner, scheduling the test body after sleep(20ms) can itself take longer than 100ms, or the remover isn't scheduled yet so the blocking Drop lands on the worker the call needs. Either way waited exceeds the bound with no code change.

How about having SlowDrop::drop block on a std::sync::Barrier or a channel instead of sleeping? Then the test can assert the add call completes while the drop is provably still blocked, and release it afterward. That tests "the lock isn't held during drop" directly with no timing in the assertion.

Comment thread rig/rig-core/src/tool/server.rs Resolved
/// Both the arguments and return value are a String since these values are meant to
/// be the output and input of LLM models (respectively)
///
/// Calls can run at the same time, including calls to the same tool.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ToolServerHandle holds the tool server behind a lock, where it used
to send each request over a channel to a task that owned the server.
Adding, removing and looking up a tool take the lock briefly. Fetching
tool definitions works on a copy taken under the lock and awaits
outside it, so a tool call never waits behind a definitions fetch.

With no task there is nothing to stop serving. A caller that stops
waiting, or a dynamic tool's vector store returning an error, no
longer leaves every later call on the handle failing, and a failed
definitions lookup comes back as ToolServerError::DefinitionsFailed.
The request and response message types, the error variants for a
failed channel, and the wasm-bindgen-futures dependency that spawned
the task on wasm are removed.

A failed call returns the tool set's error as it is, so a caller can
match its variant or downcast the tool's own error. Its text drops the
ToolCallError prefixes the channel added. ToolSet::call shares the
call path, so a panicking tool returns an Err there too, except on
wasm where a panic aborts. ToolServerError is non_exhaustive.

BREAKING-CHANGE: ToolServer::handle_message, ToolServerRequest,
 ToolServerRequestMessageKind, ToolServerResponse, and the Canceled,
 SendError and InvalidMessage variants of ToolServerError are removed.
 ToolServerError gains DefinitionsFailed and is non_exhaustive. A
 failed call's error keeps its type and its text loses repeated
 ToolCallError prefixes. ToolSet::call returns a panicking tool's
 panic as an Err.
Fixes: mezmo/aura#811
Signed-off-by: Jacob Hull <jacob@planethull.com>
The "Calling tool" debug log pretty-printed the arguments as a JSON
string, so they showed escaped and quoted, like "{\"x\":2}". They are
now parsed and pretty-printed as JSON, or logged as they are when
they are not JSON.

Fixes: mezmo/aura#812
Signed-off-by: Jacob Hull <jacob@planethull.com>
Registering a tool under a name that was already listed, through
ToolServer::tool, rmcp_tool, the agent builder's list, or
ToolServerHandle::add_tool, replaced the tool but listed its name a
second time. Tool definitions then named it twice, and providers
reject a request whose tool names are not unique. Each path now lists
a name once.

Signed-off-by: Jacob Hull <jacob@planethull.com>
@jakedipity
jakedipity force-pushed the jakedipity/inline-tool-calls branch from e68fc44 to 12147eb Compare October 27, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants