Repository navigation
feat(tool): run tool calls in the caller's task - #17
jakedipity wants to merge 4 commits into
Conversation
[Medium impact] The PR appears safe to merge; no actionable issues were found. SummaryTool calls now run in the caller’s task, using shared tool references instead of a background server task.
No new actionable issues were found.
|
|
The failing |
ToolServerHandle::call_tool asks the tool server for the tool and runs it in the calling task. The server task used to run every call itself, one at a time. A call now sees the caller's task-locals and tracing span, calls from different callers run at the same time, and a call stops when its caller stops waiting on it. The server still adds and removes tools and serves definitions. Tools are held behind Arc so the server can hand one out. A failed call keeps its error text word for word, and a panicking tool fails only its own call, with the panic's message as the error. The span carried on the CallTool message goes away with that message, since the caller's span is now current where the tool runs. Ref: mezmo/aura#808 Signed-off-by: Jacob Hull <jacob@planethull.com>
95ac4bb to
e68fc44
Compare
| /// Runs `tool_name` in the caller's task, so the call sees the caller's | ||
| /// task-locals and span, and calls from different callers run at the same | ||
| /// time. | ||
| pub async fn call_tool(&self, tool_name: &str, args: &str) -> Result<String, ToolServerError> { | ||
| let (tx, rx) = futures::channel::oneshot::channel(); | ||
|
|
||
| self.0 | ||
| .send(ToolServerRequest { | ||
| callback_channel: tx, | ||
| data: ToolServerRequestMessageKind::CallTool { | ||
| name: tool_name.to_string(), | ||
| args: args.to_string(), | ||
| caller_span: tracing::Span::current(), | ||
| }, | ||
| let tool = self.read().toolset.get(tool_name).cloned(); | ||
| call_tool_named(tool_name, tool.as_ref(), args.to_string()) | ||
| .await | ||
| .map_err(|err| { | ||
| // Callers match on this text. The tool set's error is | ||
| // stringified inside a tool call error. | ||
| ToolServerError::ToolsetError(ToolSetError::ToolCallError( | ||
| ToolError::ToolCallError(err.to_string().into()), | ||
| )) | ||
| }) | ||
| .await?; | ||
|
|
||
| let res = rx.await?; | ||
|
|
||
| match res { | ||
| ToolServerResponse::ToolExecuted { result, .. } => Ok(result), | ||
| ToolServerResponse::ToolError { error } => Err(ToolServerError::ToolsetError( | ||
| ToolSetError::ToolCallError(ToolError::ToolCallError(error.into())), | ||
| )), | ||
| invalid => Err(ToolServerError::InvalidMessage(invalid)), | ||
| } | ||
| } |
There was a problem hiding this comment.
Running the tool future inside the caller's task changes what happens when the caller stops waiting. If a consumer drops the MultiTurnStream, or a timeout fires around call_tool, the tool future is dropped mid-flight. The old server task always ran a call to completion.
Dropping the future stops the local work but not anything already in flight elsewhere. A tool doing multi-step side effects (DB writes, external calls) can be left half-done, with the result lost and no tool_result recorded. For an MCP tool this is true even when the server supports an MCP cancel request, because nothing on our side sends one: rmcp's call_tool goes through send_request, which consumes the RequestHandle, and RequestHandle has no Drop impl, so no notifications/cancelled is ever sent. The server keeps running the call until it finishes on its own.
dropping_the_caller_stops_the_tool locks this in as intended, which is fine, but the Tool docs only say calls may run concurrently.
Suggestion:
Include something along the lines of "a call may be dropped at any await point, so a tool with side effects that must complete should spawn its real work onto a detached task rather than run it inline" to the comment/doc on Tool::call. Call this out in the PR description/commit as a behavior change.
There was a problem hiding this comment.
@jakedipity , @Shearerbeard , coming back to this after talking with Mike and digging further into AURA consuming Rig, the previous suggestion doesn't make sense and would essentially add "dead" code with regards to AURA since AURA never makes calls using McpTool::call. The suggestion of a CancelOnDrop.
It's a bit confusing because I anchored the comment here when it relates to McpTool::call which was never changed. My bad.
I've updated the comment/suggestion and removed the CancelOnDrop suggestion but it could still be useful to add a comment/doc change to Tool::call here:
There was a problem hiding this comment.
For the record, the drop introduced does reach AURA in one configuration. A single agent with no HITL, scratchpad, or nudge layer has unwrapped MCP tools, so a dropped stream now drops the call inside AURA's call_tool_tracked with its in-flight entry still registered. That's AURA's to fix and should be tracked on mezmo/aura#808, not something for this PR.
| .collect() | ||
| }; | ||
|
|
||
| drop(replaced); | ||
| Ok(()) |
There was a problem hiding this comment.
append_toolset inserts into toolset.tools but never calls list_static, so tools added this way never land in static_tool_names and get_tool_defs never returns them. The model is never told these tools exist.
add_tool just above and ToolServer::tool both list the name. The asymmetry was already there in the old AppendToolset arm, but the rewrite keeps it and the new tests don't cover it. I think the fix is in the filter_map closure that currently borrows server mutably for the insert, so we probably need to call list_static in the same closure or in a separate loop before it.
Perhaps we could call server.list_static(name.clone()) inside the filter_map, and add a test that appends a toolset and asserts get_tool_defs(None) includes it?
There was a problem hiding this comment.
Downgrading this to optional/nit. AURA only registers tools through the builder and never calls append_toolset or remove_tool on the handle, so the gap can't bite us (AURA). Still a one-liner if you want it; otherwise fine to leave.
| // A panicking tool fails its own call. The caller carries on. | ||
| let result = std::panic::AssertUnwindSafe(tool.call(args)) | ||
| .catch_unwind() | ||
| .await | ||
| .unwrap_or_else(|panic| { | ||
| Err(ToolError::ToolCallError( | ||
| format!("tool `{toolname}` panicked: {}", panic_message(&panic)).into(), | ||
| )) | ||
| }); | ||
| Ok(result?) |
There was a problem hiding this comment.
catch_unwind here only wraps polling of the future that ToolDyn::call returns. A hand-written ToolDyn::call that panics synchronously before returning Box::pin (say serde_json::from_str(&args).unwrap() outside the async block) panics on the tool.call(args) expression, before AssertUnwindSafe(...) is even built. That still unwinds the caller's task, which the comment above says can't happen.
What if we catch the construction as well?
let fut = match std::panic::catch_unwind(AssertUnwindSafe(|| tool.call(args))) {
Ok(fut) => fut,
Err(panic) => return Err(ToolError::ToolCallError(
format!("tool `{toolname}` panicked: {}", panic_message(&panic)).into(),
).into()),
};
let result = AssertUnwindSafe(fut).catch_unwind().await.unwrap_or_else(/* same */);Or if we're fine with the gap, how about the comment says "a panic while running the tool" rather than implying construction is covered too? Or maybe include, explicitly, that construction is not covered? The macro-generated ToolDyn impls are safe either way.
| // A panicking tool fails its own call. The caller carries on. | ||
| let result = std::panic::AssertUnwindSafe(tool.call(args)) | ||
| .catch_unwind() | ||
| .await | ||
| .unwrap_or_else(|panic| { | ||
| Err(ToolError::ToolCallError( | ||
| format!("tool `{toolname}` panicked: {}", panic_message(&panic)).into(), | ||
| )) | ||
| }); |
There was a problem hiding this comment.
catch_unwind here only wraps polling of the future that ToolDyn::call returns. A hand-written ToolDyn::call that panics synchronously before returning Box::pin (say serde_json::from_str(&args).unwrap() outside the async block) panics on the tool.call(args) expression, before AssertUnwindSafe(...) is even built. That still unwinds the caller's task, which the comment above says can't happen.
How about we move the call inside the block that's already caught?
let result = std::panic::AssertUnwindSafe(async { tool.call(args).await })
.catch_unwind()
.await
.unwrap_or_else(/* same as now */);Construction then happens on the first poll, under the same catch_unwind, so a panic before Box::pin is handled like a panic after it. No second AssertUnwindSafe needed, and it makes the same promise about shared tool state as the current code does. A test with a tool that panics in ToolDyn::call itself would pin it.
There was a problem hiding this comment.
Downgrading this to optional/nit. AURA's tools go through rig's Tool blanket impl, which can't panic before Box::pin, so this doesn't affect the fork's consumer.
| #[tokio::test(flavor = "multi_thread", worker_threads = 2)] | ||
| async fn removing_a_tool_does_not_hold_up_other_calls() { | ||
| let handle = ToolServer::new().tool(SlowDrop).tool(Adder).run(); | ||
| let remover = tokio::spawn({ | ||
| let handle = handle.clone(); | ||
| async move { handle.remove_tool("slow_drop").await } | ||
| }); | ||
| tokio::time::sleep(Duration::from_millis(20)).await; | ||
|
|
||
| // A blocked lock stalls the thread itself, so a timeout around the | ||
| // call would never fire. Measure the call instead. | ||
| let start = std::time::Instant::now(); | ||
| let call = handle.call_tool("add", r#"{"x": 1, "y": 2}"#).await; | ||
| let waited = start.elapsed(); | ||
|
|
||
| assert_eq!(call.unwrap(), "3"); | ||
| assert!( | ||
| waited < Duration::from_millis(100), | ||
| "the call waited {waited:?} for the removed tool to drop" | ||
| ); | ||
| remover.await.unwrap().unwrap(); | ||
| } |
There was a problem hiding this comment.
This asserts a timing bound rather than the property. On a loaded CI runner, scheduling the test body after sleep(20ms) can itself take longer than 100ms, or the remover isn't scheduled yet so the blocking Drop lands on the worker the call needs. Either way waited exceeds the bound with no code change.
How about having SlowDrop::drop block on a std::sync::Barrier or a channel instead of sleeping? Then the test can assert the add call completes while the drop is provably still blocked, and release it afterward. That tests "the lock isn't held during drop" directly with no timing in the assertion.
| /// Both the arguments and return value are a String since these values are meant to | ||
| /// be the output and input of LLM models (respectively) | ||
| /// | ||
| /// Calls can run at the same time, including calls to the same tool. |
There was a problem hiding this comment.
This is where I suggested adding an additional comment.
ToolServerHandle holds the tool server behind a lock, where it used to send each request over a channel to a task that owned the server. Adding, removing and looking up a tool take the lock briefly. Fetching tool definitions works on a copy taken under the lock and awaits outside it, so a tool call never waits behind a definitions fetch. With no task there is nothing to stop serving. A caller that stops waiting, or a dynamic tool's vector store returning an error, no longer leaves every later call on the handle failing, and a failed definitions lookup comes back as ToolServerError::DefinitionsFailed. The request and response message types, the error variants for a failed channel, and the wasm-bindgen-futures dependency that spawned the task on wasm are removed. A failed call returns the tool set's error as it is, so a caller can match its variant or downcast the tool's own error. Its text drops the ToolCallError prefixes the channel added. ToolSet::call shares the call path, so a panicking tool returns an Err there too, except on wasm where a panic aborts. ToolServerError is non_exhaustive. BREAKING-CHANGE: ToolServer::handle_message, ToolServerRequest, ToolServerRequestMessageKind, ToolServerResponse, and the Canceled, SendError and InvalidMessage variants of ToolServerError are removed. ToolServerError gains DefinitionsFailed and is non_exhaustive. A failed call's error keeps its type and its text loses repeated ToolCallError prefixes. ToolSet::call returns a panicking tool's panic as an Err. Fixes: mezmo/aura#811 Signed-off-by: Jacob Hull <jacob@planethull.com>
The "Calling tool" debug log pretty-printed the arguments as a JSON
string, so they showed escaped and quoted, like "{\"x\":2}". They are
now parsed and pretty-printed as JSON, or logged as they are when
they are not JSON.
Fixes: mezmo/aura#812
Signed-off-by: Jacob Hull <jacob@planethull.com>
Registering a tool under a name that was already listed, through ToolServer::tool, rmcp_tool, the agent builder's list, or ToolServerHandle::add_tool, replaced the tool but listed its name a second time. Tool definitions then named it twice, and providers reject a request whose tool names are not unique. Each path now lists a name once. Signed-off-by: Jacob Hull <jacob@planethull.com>
e68fc44 to
12147eb
Compare
Breaking changes
ToolServer::handle_message,ToolServerRequest,ToolServerRequestMessageKindandToolServerResponseToolServerError::{Canceled, SendError, InvalidMessage}ToolServerError::DefinitionsFailed, andToolServerErroris now#[non_exhaustive]call_toolreturns the tool set's error with its variant intact, and its text drops repeatedToolCallError:prefixes (Toolset error: ToolNotFoundError: missing)ToolSet::callandcall_toolreturn a panicking tool's panic as anErr, except on wasmfeat(tool): run a tool call in the caller's task
ToolServerHandle::call_tool asks the tool server for the tool and runs it in the calling task. The server task used to run every call itself, one at a time. A call now sees the caller's task-locals and tracing span, calls from different callers run at the same time, and a call stops when its caller stops waiting on it. The server still adds and removes tools and serves definitions.
Tools are held behind Arc so the server can hand one out. A failed call keeps its error text word for word, and a panicking tool fails only its own call, with the panic's message as the error. The span carried on the CallTool message goes away with that message, since the caller's span is now current where the tool runs.
Ref: mezmo/aura#808
refactor(tool): share the tool server's state instead of running a task
ToolServerHandle holds the tool server behind a lock, where it used to send each request over a channel to a task that owned the server. Adding, removing and looking up a tool take the lock briefly. Fetching tool definitions works on a copy taken under the lock and awaits outside it, so a tool call never waits behind a definitions fetch.
With no task there is nothing to stop serving. A caller that stops waiting, or a dynamic tool's vector store returning an error, no longer leaves every later call on the handle failing, and a failed definitions lookup comes back as ToolServerError::DefinitionsFailed. The request and response message types, the error variants for a failed channel, and the wasm-bindgen-futures dependency that spawned the task on wasm are removed.
A failed call returns the tool set's error as it is, so a caller can match its variant or downcast the tool's own error. Its text drops the ToolCallError prefixes the channel added. ToolSet::call shares the call path, so a panicking tool returns an Err there too, except on wasm where a panic aborts. ToolServerError is non_exhaustive.
Fixes: mezmo/aura#811
fix(tool): log a tool call's arguments as readable JSON
The "Calling tool" debug log pretty-printed the arguments as a JSON string, so they showed escaped and quoted, like
"{\"x\":2}". They are now parsed and pretty-printed as JSON, or logged as they are when they are not JSON.Fixes: mezmo/aura#812
fix(tool): list a tool once when it is added again
Registering a tool under a name that was already listed, through ToolServer::tool, rmcp_tool, the agent builder's list, or ToolServerHandle::add_tool, replaced the tool but listed its name a second time. Tool definitions then named it twice, and providers reject a request whose tool names are not unique. Each path now lists a name once.