Skip to content

Standardise an opt-in SSE progress-streaming response for /evaluate and /chat - #44

Open
m-messer wants to merge 4 commits into
mainfrom
b43-streaming-support
Open

m-messer wants to merge 4 commits into
mainfrom
b43-streaming-support

Conversation

@m-messer

@m-messer m-messer commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Standardises an opt-in Server-Sent Events (SSE) progress-streaming response for /evaluate and /chat, as proposed in #43.

Previously the spec only documented 200 → application/json, while ChatCapabilities.supportsStreaming and LLMConfiguration.stream implied streaming was in scope with no wire contract behind them. This defines that contract once, generically, so both endpoints stream identically and new endpoints get it for free.

Contract:

  • Opt-in via Accept: text/event-stream. HTTP status stays 200 for the life of the stream, including failures.
  • The stream is: zero or more progress frames (event: = stage name, data: = SseProgressStep), optional : keep-alive comments, and exactly one terminal frame.
  • Terminal frame carries the operation's normal 200 body (event: completed) or an ErrorResponse under error (event: failed), plus a steps[] replay of every progress step.
  • X-Request-Id / X-Api-Version are sent once as response headers at stream open.
  • configuration.llm.stream (token-level LLM streaming) is now explicitly documented as distinct from API-layer progress streaming.

Key schema additions (shared, reused by both operations):

Schema Role
SseProgressStep stage / message / timestamp; stage is an informative string, not an enum
SseTerminalSteps shared steps[] replay fragment
StreamingCapabilities supportsStreaming + supportedProgressStages, composed via allOf into ChatCapabilities and EvaluateCapabilities
SseEvaluateTerminalFrame / SseChatTerminalFrame terminal-frame data, reusing each operation's existing 200 body
EvaluateResponse the /evaluate 200 body extracted into a named schema so the terminal frame can reuse it

ChatCapabilities.supportsStreaming moves into the shared fragment; EvaluateCapabilities gains streaming advertisement for the first time. Both /health examples now show supportsStreaming + supportedProgressStages.

Related issue

Closes #43

Scope

  • This pull request is focused on a single concern.
  • The change was started from the latest main branch, or from a fork if direct branch creation is not available.

Validation

  • npm run lint — valid, no new warnings (8 pre-existing warnings unchanged)
  • npm run bundle — succeeds; all new components resolve

Notes for reviewers

  • Compatibility: additive / non-breaking. New optional text/event-stream content, new optional capability fields, description edits. The one refactor is /evaluate's 200 body moving from an inline array to the named EvaluateResponse component (semantically identical).
  • SseChatTerminalFrame.output uses anyOf: [Message, "null"] rather than the repo's usual type: [object, "null"] + allOf: [$ref] idiom, because the failed-frame example sets output: null and must validate. The error fields keep the existing repo idiom (no null example exercises them).
  • Follow-up: the shimmy reference implementation needs a matching PR — terminal-frame field names, /health capability output, its embedded spec copy, and stream tests.

🤖 Generated with Claude Code

@maximiliansoelch

Copy link
Copy Markdown
Contributor

@m-messer Do you plan to actively use this?
The spec changes look reasonable to me, but it's hard to judge from the spec alone whether the contract holds up in practice. A reference implementation that uses it successfully would help confirm the spec covers everything needed. Would it be possible to have the shimmy follow-up you mentioned (or at least a draft of it) before we merge this?

@m-messer

Copy link
Copy Markdown
Contributor Author

@maximiliansoelch Of course, here is the shimmy PR on using SSE: lambda-feedback/shimmy#28

In particular, take a look at: https://github.com/lambda-feedback/shimmy/pull/28/changes#diff-1ee1003a66dcdc217b95324d29e9959e0795e3f021bb886f9b2bf26c149ef59e and the files in the progress directory.

I am happy to answer any questions you have on the implementation.

@maximiliansoelch maximiliansoelch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The overall approach looks good to me 👍
A shared, opt-in SSE progress stream, especially for /chat, is a useful addition.
I left some inline comments on the wire-format examples, event names, terminal payloads, and response selection.
The PR also currently has a merge conflict.

Comment thread paths/chat/chat.yml
model: "gpt-5.2"
temperature: 0.5
outputTokens: 143
text/event-stream:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The schema and examples under text/event-stream appear to describe the JSON in a terminal frame’s data: field, rather than the complete SSE response. That response may also contain progress events and comments, so the framing is missing from the media-type example. Is the intent to describe the full response here as SSE text and document the JSON frame payloads separately?

This also applies to /evalaute.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to OpenAPI 3.2, which has support for SSE using itemSchema

MUST tolerate values they do not recognise. Each service advertises the
stage values it emits via `supportedProgressStages` in its `/health`
capabilities. Common values: "preparing", "starting", "evaluating"
(evaluate), "thinking" (chat), and the terminal "completed" / "failed".

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is unclear whether completed and failed are progress stages or only terminal event names. They are listed here and in supportedProgressStages, while the operation descriptions use them for terminal frames with a different data: shape. If progress frames can use these names, clients need a way to distinguish the two payload types. If the names are terminal-only, the stage description and capability examples currently suggest otherwise.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed completed and failed from the progress states and are reserved for terminal states.


`steps` is always present: the ordered replay of every progress step emitted
during the stream.
allOf:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description defines a completed payload with a result and a failed payload with an error, but the schema requires only steps. It therefore accepts { "steps": [] } and also a payload containing both feedback and error. If these outcomes are intended to be exclusive, separate JSON payload alternatives can express that rule, with the SSE event mapping stated in the description. feedback also repeats Feedback[] instead of referencing EvaluateResponse. The chat terminal schema similarly repeats its normal response fields.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reduced duplication between chat and evaluate, and split the payloads.

Comment thread paths/evaluate/evaluate.yml Outdated
`SseEvaluateTerminalFrame` whose `error` is an `ErrorResponse`.


Failures are always delivered as a terminal `failed` frame, never as an

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 200/failed rule seems to describe failures after the SSE stream opens, but “failures are always delivered” also reads as applying to validation or authorization failures detected beforehand. The operation still lists 400 and 403, leaving that boundary unclear. The response mode is also unspecified when a request includes both callbackUrl and Accept: text/event-stream: one may produce 202, while the other describes a 200 stream. What response-selection rule is intended for these cases?

Also applies to chat.yml.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Errors detected before event streaming are sent as their normal status, and if callback and event streaming are both disabled a 400 error is returned.

m-messer and others added 2 commits November 11, 2026 11:02
Conflict resolutions:
- chat-health.yml / evaluate-health.yml: take main (capabilities moved to
  /info in #34); port the streaming capability examples
  (supportsStreaming, supportedProgressStages) to chat-info.yml and
  evaluate-info.yml.
- evaluate.yml: keep main's MATH matched-case JSON examples alongside the
  text/event-stream SSE content block.

Follow-ups:
- Point streaming docs (openapi.yml, SseProgressStep, LLMConfiguration)
  at /info instead of /health.
- Fix /info examples to use supportedAPIVersions, matching the
  ChatCapabilities/EvaluateCapabilities schemas (was supportedVersions).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oads

- Bump to OpenAPI 3.2 and describe text/event-stream with `itemSchema`
  (per-event SseEvaluateEvent / SseChatEvent) plus raw-wire
  `serializedValue` examples including progress events and keep-alives.
- Replace the terminal-frame schemas with exclusive payloads:
  SseEvaluateCompletedFrame (feedback: $ref EvaluateResponse),
  SseChatCompletedFrame (allOf ChatResponse), and a shared SseFailedFrame
  (required error). Completed frames forbid `error`; failed requires it.
- Reserve `completed` / `failed` as terminal event names only; remove
  them from stage descriptions and supportedProgressStages examples.
- Define an ordered response-selection rule: pre-stream errors use normal
  HTTP statuses; callbackUrl + Accept: text/event-stream -> 400;
  callbackUrl -> 202; streaming -> 200 SSE; otherwise 200 JSON.
- Add the missing chat 202 Accepted response and ChatAcceptedResponse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@m-messer

Copy link
Copy Markdown
Contributor Author

Thanks for the review! I've fixed the merge conflict and addressed your four comments.

additionalProperties was set on the string enum `classification`,
where it has no effect and triggers a Swagger Editor warning. Move it
up to the `dataCategory` object, which is what it was meant to apply to.

Closes #17

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standardise an opt-in SSE progress-streaming response (evaluate, chat, and beyond)

3 participants