[Python] Support multimodal text and image prompts

Goal

Add backward-compatible Python support for text-and-image prompts, matching the existing TypeScript prompt capabilities.

This is an owner-requested tracking item. It is separate from the Python authentication/client-injection work in #516 and the TypeScript authentication parity PR. No Python implementation is included in those changes.

Scope and acceptance criteria

Extend public Python prompt types to accept existing string content and explicitly typed text/image parts, including image URLs and data URLs. Preserve current string-only callers without migration.
Forward supported multimodal content through both the built-in HTTP Chat Completions model and the configured-client adapter from Add dependency-free Python token providers and client adapters #516. Do not silently drop, stringify, or coerce image parts into unconstrained values.
Document how applications provide image context through the existing model/translator prompt surfaces. Keep ordinary translate(str) calls compatible.
Preserve prompt ordering and image context across validation/repair attempts. Do not weaken schema generation, result validation, or explicit error behavior.
Keep authentication, application-owned SDK clients, retries/timeouts, and HTTP response-size controls consistent with the relevant model path.
Add no required SDK, Azure Identity, or image-processing dependencies. TypeChat should forward image references, not fetch images or introduce image generation.
Add deterministic offline tests for existing text prompts, mixed text/images, URL/data-URL images, both model paths, malformed/unsupported prompt parts, and translator repair.
Update the Python documentation with a minimal text-and-image example and supported-format limitations.

Boundaries

This item covers text/image input, not audio/video input, image generation, streaming, or agent orchestration. Responses API support is a separate work item; text/image support should not require that API migration. Add combined-path regression coverage when both capabilities are available.

Context

[Python] Support using keyless auth for Azure OpenAI #216 and Add dependency-free Python token providers and client adapters #516 address reusable authentication and configured model clients, not multimodal prompt typing.
TypeScript already exposes typed text/image prompt content; use it as compatibility/design evidence rather than weakening Python typing or validation.

/typechat propose

Work proposal P1

Proposal for community discussion, not accepted work or permission to build.

Recommendation

I recommend an additive change to Python's existing Chat Completions path, the message-based model API, rather than a new backend or a Responses API migration. PromptSection.content is currently typed as a string, but both built-in model paths already forward message dictionaries. The translator also retains its preamble when it appends correction messages after invalid output, so the existing surfaces are a practical starting point.

Export typed dictionary definitions for tagged text and image_url parts, allowing either existing string content or an ordered list of those parts. Forward image URLs, data URLs, and the optional image-processing detail setting of auto, low, or high unchanged. Use the same structural checks before either path sends a request, reject malformed or unsupported parts without echoing their contents, and keep completion responses text-only. TypeChat would not fetch or decode images.

Use model.complete for message lists and translate with prompt_preamble, the context messages placed before the generated schema request; keep translate's request parameter a string. This avoids another translator API while retaining image context during repair and preserving each path's authentication, retry, and client-ownership behavior. Build on the merged Python adapter in PR 516 and hillary-mutisya's merged TypeScript image contribution in PR 254. PR 519 is an unmerged TypeScript reference, not a Python implementation or a dependency of this proposal.

For the initial Python contract, I recommend images only in user messages and part lists containing explicitly tagged objects rather than bare strings. This keeps the request shape explicit instead of copying the more permissive string-in-array option in the original TypeScript contribution. Do your intended integrations require images in system or assistant messages, or bare strings inside part lists? The issue does not specify those cases, and the answers are needed to finalize these compatibility boundaries.

Changes since the previous proposal

This is the initial recommendation. The supplied conversation contains no preceding public proposal or community feedback to reconcile.
This round turns the recorded requirements into a tagged Python dictionary contract and shared prompt checks. It also separates response typing from prompt typing because the inspected HTTP response annotation currently reuses PromptSection.
The recommendation uses existing Chat Completions and translator preamble interfaces instead of adding a backend or waiting for Responses. It preserves prior contributions and makes the unresolved role and part-list compatibility assumptions explicit rather than claiming agreement.

Direction and readiness

The frozen GOALS.md policy supports this additive Python integration under G1 and G4, reliable input/output validation and repair under G2, compatibility with existing implementations under G3, and documentation and deterministic regressions under G5. Forwarding image references through existing interfaces does not introduce application business logic under N2, a vendor-dependent core under N3, or orchestration under N1. Explicit prompt checks and unchanged schema validation respect N4. The separate Responses API work is not required by this recommendation.

A concrete approach and observable criteria can be recommended, but the issue does not settle whether image parts are needed outside user messages or whether Python must accept bare strings within part lists for TypeScript parity. The recommendation is user-only images and explicitly tagged parts, not an assertion that those boundaries are agreed. These compatibility questions need answers before an owner acceptance decision; the open TypeScript PR does not answer them for Python.

Goal

Python applications can provide typed text and image references through existing Chat Completions model and translator interfaces and obtain validated structured results without changing text-only callers or authentication setup.

Scope

Export PromptContent, TextPromptContent, ImagePromptContent, and ImageUrl using precise dictionary types and literal tags. Widen PromptSection.content to a string or list of tagged text/image parts, with an optional image detail setting of auto, low, or high.
Recommend image_url parts only in user messages for the initial contract, tagged text parts in the existing roles, and no bare strings inside part lists. Preserve whole-string messages in every existing role. These role and list compatibility boundaries remain subject to the listed questions.
Share prompt checking and string-to-user-message normalization between the built-in HTTP and configured-client Chat Completions paths. Preserve nonempty image-reference strings, ordering, and caller data; reject malformed or unsupported parts with a content-free diagnostic before authentication or dispatch.
Keep successful model output text-only, separate response parsing from the widened prompt input types, and explicitly reject non-text response content without changing existing empty-text or missing-text behavior on either path.
Provide image context through the existing model message lists and translator prompt_preamble list. Keep the translator request parameter a string and preserve its generated schema request, diagnostics, repair limit, and original context throughout repair.
Extend the merged Python adapter from PR 516 rather than replacing or duplicating it. Preserve each path's existing authentication, retries, cancellation, timeouts, client ownership, error behavior, and applicable HTTP response-size controls.
Add deterministic offline behavioral and type-check coverage and update Python README and basic-usage documentation. Include combined-path regressions only when the separately implemented Python Responses capability is available.

Non-goals

No Responses API implementation, migration, endpoint auto-detection, or dependency on unmerged PR 519. That TypeScript proposal is comparative evidence, not adopted Python implementation work.
No image fetching, decoding, resizing, generation, asset hosting, or validation of image contents. Applications and providers retain responsibility for image data and provider-specific image-format and size limits.
No audio/video input, streaming, agent orchestration, storage, deployment management, or application-specific business logic.
No new dependencies, authentication backend, configured-client replacement, or application-client lifecycle, retry, or timeout management.
No changes to TypeScript behavior, schema generation, output validation rules, or translation guarantees, and no new multipart request parameter on translate.
No guarantee that every provider or model supports image input, and no local inspection of referenced images to establish provider compatibility.

Acceptance criteria

Public exports describe ordinary dictionaries using TypedDict and literal tags. PromptSection.content accepts a string or an ordered list of text and image_url objects. Image references are nonempty strings with optional detail limited to auto, low, or high. Static checks accept supported examples and reject missing fields, wrong value types, unknown tags, and invalid detail values without unconstrained casts.
Offline HTTP-transport and fake-client tests capture unchanged mixed-part message content, exact hosted-image and data-URL strings, optional detail, and original message and part ordering. Both paths retain existing string requests and leave caller-owned message dictionaries and nested parts unmodified.
Under the recommended initial contract, images are accepted only in user messages and part lists contain only tagged objects; string content remains valid for all three existing roles. Both paths return a Failure identifying an invalid section or part without echoing its contents, before token-provider invocation or request dispatch. Invalid inputs trigger no request retries or translator repair.
Model completion remains Result[str], with response parsing separate from prompt input types. Empty text remains valid, each path preserves its existing missing-text behavior, and non-text response content cannot become a successful string completion merely because prompt inputs now support lists.
Offline translator cases with mixed image preambles cover immediate success, schema-invalid output, malformed JSON, exhausted repair, and model failure during repair. All attempts retain exact original image/text context and ordering. Repair appends rejected assistant text followed by diagnostics and preserves the existing repair limit and rejected-output payload behavior.
Regression cases preserve existing API-key and token-provider HTTP authentication, per-attempt token resolution on retries, cancellation, timeout settings, and response-size boundaries. Configured-client cases preserve the caller's instance, configuration, lifecycle, exceptions, and cancellation without an added retry or timeout layer.
Existing schema-generation errors, runtime validation rules, successfully translated types, and string-only request and repair snapshots remain unchanged. Unsupported schema types are not silently accepted as part of enabling image inputs.
The change adds no dependencies, and TypeChat remains importable without provider SDKs or Azure Identity. Offline cases observe no image retrieval, decoding, generation, streaming, or audio/video handling. Reference structure is checked locally; image contents and provider-specific format and size validity remain the provider's responsibility.
Python README and basic-usage documentation include minimal model.complete and image-bearing prompt_preamble examples. They explain dictionary tags, supported roles and detail values, URL/data-URL forwarding, provider/model limitations, and unchanged authentication and client ownership. Supported examples pass offline type checks without a required provider SDK.
Chat Completions coverage passes independently of Responses support. If the separately implemented Python Responses capability is present in the implementation base, deterministic combined-path coverage verifies text/image forwarding and repair there; otherwise this work adds neither a Responses implementation nor a requirement to wait for it.

Open questions

Does any intended Python integration require image parts in system or assistant messages? The recommendation permits images only in user messages and returns an explicit Failure for images in other roles on both model paths.
Is accepting bare string entries inside content-part lists required for TypeScript parity? The recommendation preserves whole-string content but otherwise accepts only tagged text and image objects, rather than copying the original TypeScript string-in-array option.

Evidence limits and assumptions

No code, reproductions, tests, installations, browser actions, or live provider calls were executed. SDK compatibility and provider-specific image-format and size limits were not independently verified.
The complete current TypeScript model implementation was not inspected. Type-shape evidence came from merged PR 254 and the pinned translator; PR 519's proposed adapter restrictions were read from its unmerged description, not treated as established Python behavior.
The target evidence was limited to the supplied issue body and empty comment list; no later target-thread content was fetched. Related PR metadata was read separately to identify contributions. Unlinked work and unrelated PRs were not reviewed.

Related contributions: #254, #516, #519.

Not eligible for acceptance yet. Discuss the remaining questions and request /typechat propose again.

Proposal run | Approved scope policy

I don't think we need image parts in the system or assistant messages.
We don't necessarily have to have TypeScript parity but flexibility is always good.

Also, we need to consider #241. While the implementation is out of date the idea is a good one and is on the roadmap so we need to consider that future work as part of this update.

/typechat propose

TypeChat: A proposal was not confirmed. Inspect the run and thread before retrying; do not assume nothing was posted.

Owner request | Run

Operational receipt only. This comment does not publish draft text, accept work, authorize implementation, or approve merging.

/typechat reply

Keeping image parts in user messages fits this change; I would leave system and assistant messages text-only. Exact TypeScript parity is not necessary to preserve Python compatibility: keep whole-string messages and use explicitly tagged text/image parts rather than bare strings inside part lists. Python's PromptSection.content is still typed as str, although both the HTTP model and the configured-client adapter already forward message dictionaries. The missing piece is a supported typed contract with consistent input checks on both paths.

Guido van Rossum's PR 241 is relevant to how those messages are assembled. It proposes separating schema instructions from the current user request and placing conversation history between them. The inspected translator instead places prompt_preamble, the application's context messages, before a user message containing both schema and request. It retains those context messages when asking the model to repair invalid output, so image-bearing user messages could use that surface without changing ordinary translate(str) calls.

I recommend keeping structured image parts independent of that layout and preserving the current ordering in this item, rather than adopting PR 241's outdated implementation. Its discussion reported worse results with system prompts and left schema placement and repair-message roles unsettled. Future prompting work should preserve image-bearing user messages and repair context rather than concatenate or stringify them. This would extend the Python client adapter merged in PR 516 and draw on hillary-mutisya's image types from PR 254. PR 519 is now merged too, but provides TypeScript integration code, not Python multimodal support.

I would start with image context through prompt_preamble and leave any change to how the translator accepts the current request for a separate, explicit decision. That avoids coupling image transport to a new prompting scheme while still considering PR 241's direction. Is image context through prompt_preamble sufficient for this version, or do you need translate itself to accept a current request made of text and image parts?

Related contributions: #241, #254, #516, #519.

Automated response requested by the project owner. This does not accept work or authorize implementation.

agreed.

/typechat propose

Work proposal P2

Proposal for community discussion, not accepted work or permission to build.

Recommendation

I recommend keeping this as an additive Python Chat Completions update, using the existing message-based model API. Supply images through user messages in model.complete or prompt_preamble, the context messages passed to translate, and keep translate's current request a string. That follows your user-only guidance and the later agreement with using the preamble, without introducing a second translator request format or waiting for the separate Responses API work.

Use TypedDict definitions, which describe required dictionary fields, for tagged text and image_url parts. Preserve whole-string content in every existing role, allow tagged text parts in those roles, and allow images only in user messages. I recommend tagged objects rather than bare strings inside part lists: Python has no documented list contract to preserve, and exact TypeScript parity is not required. Both built-in paths should apply the same structural checks, preserve image references and ordering, and return explicit failures without echoing prompt contents.

Keep message parts independent of schema placement so Guido van Rossum's PR 241 can later separate schema instructions, conversation history and the current request without redesigning image transport. Concatenating image-bearing history into text would lose that structure. PR 241 remains unmerged: its older reports of worse system-prompt results and later favorable but provisional review do not settle schema placement or repair roles. This item should preserve today's ordering rather than adopt or duplicate that prompting change.

Build on the merged Python client adapter from PR 516 and hillary-mutisya's TypeScript image types and example from PR 254 and PR 258. PR 519 is now merged too, but provides TypeScript integration code, not Python multimodal support. Forward hosted-image references and data URLs, which embed encoded image data, unchanged; do not fetch or decode images. Keep output text-only, preserve authentication and client ownership, and cover both model paths and translator repair with offline regressions. Individual providers and models still determine supported image formats and limits.

Changes since the previous proposal

P1's image-role question is removed because owner feedback explicitly says system and assistant images are unnecessary. The recommendation now fixes images to user messages while preserving text content in all existing roles.
Exact TypeScript parity no longer blocks the contract because the feedback says it is unnecessary. P1's tagged-only part lists remain the Python recommendation: preserving whole-string callers does not require adopting TypeScript's bare-string list entries.
The final agreement follows the reply recommending image context through prompt_preamble. This round therefore keeps translate's current request a string instead of reopening that interface question or inferring acceptance of implementation.
PR 241 is now addressed explicitly through unchanged-layout regressions and an image contract independent of schema placement. Its later favorable but provisional review is considered alongside older regression reports, rather than presenting the older results as the final verdict. The prompting redesign remains separate work with contributor credit preserved.
P1's description of PR 519 as unmerged is updated: it is now merged, but still supplies TypeScript evidence rather than Python multimodal implementation. The proposal continues to extend merged PR 516 and additionally records hillary-mutisya's merged image example in PR 258 alongside the image types in PR 254.

Direction and readiness

Under the supplied approved policy, typed image context for application-defined interfaces fits G1 and G4. Preserving schema checks, strict output validation and controlled repair respects G2 and N4. Additive Python changes and unchanged TypeScript behavior fit G3; documentation and deterministic regressions fit G5. The proposal introduces neither orchestration under N1, application business logic under N2, nor a required vendor SDK under N3. PR 241 and Python Responses work are not adopted here. N5 leaves acceptance to the human direction owner, robgruen.

The recorded user-only guidance, relaxed parity requirement and agreement following the preamble recommendation leave no blocking interface question for these bounded terms. Tagged-only lists remain a concrete recommendation, not a claim of consensus on every design detail. PR 241 and Python Responses remain separate work rather than dependencies. Observable criteria are provided for an owner decision; informal agreement is not treated as work acceptance or permission to implement.

Goal

Python applications can supply typed text and image references through existing Chat Completions model and translator context interfaces and obtain validated structured results without migrating string-only callers or changing authentication setup.

Scope

Export TextPromptContent and ImagePromptContent as tagged TypedDict definitions, ImageUrl with a required nonempty string reference and optional detail of auto, low or high, and PromptContent as a string or ordered list of those tagged parts. Widen PromptSection.content without replacing its dictionary construction interface or changing the model completion signature.
Preserve whole-string messages in every existing role. Support tagged text parts in system, user and assistant messages, but images only in user messages. Do not accept bare strings inside part lists; these tagged-only lists are the recommended Python contract, not a requirement for exact TypeScript parity.
Share structural prompt checks and string-to-user-message normalization between the HTTP and configured-client Chat Completions paths. Preserve supported parts, image references, optional detail, ordering and caller data. Return an explicit content-free Failure for malformed or unsupported input before authentication resolution or dispatch.
Use model.complete message lists and translate's existing prompt_preamble list for images. Keep translate's current request a string and retain current schema/request placement, repair roles, diagnostics, limits and failure payloads. Keep structured context independent of schema placement so future PR 241 work can reuse the same message contract without flattening images.
Separate HTTP response typing from the widened prompt input types and keep successful completions text-only. Preserve empty text, the HTTP null-to-empty behavior and configured-client missing-text failures; explicitly reject other non-text completion content.
Extend the merged Python adapter from PR 516 instead of replacing it. Preserve authentication, environment routing, per-attempt token resolution, HTTP retries/timeouts/response-size controls, cancellation and configured-client ownership and exception behavior. Use merged PR 254 and PR 519 as TypeScript design evidence and preserve credit for PR 258's image example.
Add deterministic offline behavior and static-type coverage using existing test patterns for text-only callers, mixed parts, both image-reference forms, both model paths, invalid input and translator repair. Add combined Responses regressions only when separately implemented Python Responses support is available, without introducing that implementation here.
Update Python README and basic-usage documentation with minimal model and translator-preamble examples, tagged dictionary definitions, role/detail restrictions, unchanged request and ownership behavior, and provider-specific image-format and size limitations.

Non-goals

Do not adopt, replace, close or resolve PR 241. Do not change schema placement, system-message policy, conversation-history layout or repair roles, and do not add a structured current-request parameter to translate.
Do not implement or migrate to the Python Responses API, infer a new backend from endpoint URLs, or require waiting for the separate Responses work.
Do not fetch, decode, resize, validate image contents, generate images, host assets or guarantee that every provider or model supports image input.
Do not add audio/video input, streaming, planning, memory, agent orchestration, storage, deployment management or application-specific business logic.
Do not add dependencies, a new authentication backend, SDK imports, configured-client replacement or application-client lifecycle, retry or timeout management.
Do not change TypeScript behavior, relax schema generation or runtime result validation, or silently accept unsupported prompt or schema types.

Acceptance criteria

The public package exports PromptContent, TextPromptContent, ImagePromptContent and ImageUrl using precise dictionary types and literal tags. PromptSection retains ordinary dictionary construction and accepts whole-string content or ordered lists of tagged text and image_url parts. Existing string-only callers and documented custom model/client examples type-check without migration. Supported new examples type-check without unconstrained casts; missing fields, wrong types, unknown tags and invalid detail literals fail static checks.
Offline HTTP-transport and configured fake-client cases observe exact text, hosted-image and data-URL strings, optional detail values auto, low and high, and original message and part order. Image references must be nonempty strings; omitted detail stays omitted. Neither path mutates caller-owned sections or nested parts, drops content, coerces parts to text or accesses an image.
Both paths accept whole strings in system, user and assistant messages, tagged text lists in those roles, and tagged images only in user messages. They reject unknown roles, bare strings inside part lists, missing or wrongly typed part fields, empty image references, unknown tags and invalid detail values with a Failure locating the section or part without echoing prompt contents. Rejection precedes token-provider invocation or request dispatch and triggers neither request retries nor translator repair.
Model completion remains Result[str], with HTTP response parsing typed separately from widened prompt inputs. Empty text remains successful, HTTP null content retains its existing empty-string result, and configured-client missing-choice or missing-text responses retain explicit failures. Other existing missing-field handling is unchanged. Non-text, non-null response content cannot be returned as a successful string completion.
For each model path, deterministic translator cases with mixed image preambles cover immediate success, invalid JSON, schema-invalid output, exhausted repair and model failure during repair. Every request retains original context and part ordering. Snapshots retain the present preamble-before-schema/request layout, append rejected assistant text followed by user diagnostics, and preserve the repair limit and rejected-output payload rules. These cases do not require PR 241's layout.
Regression cases preserve existing API-key and token-provider authentication, per-attempt token resolution on retries, cancellation, HTTP timeout and response-size boundaries, and environment-based routing. Configured-client cases retain the exact instance, settings and lifecycle; exceptions and cancellation propagate, with no added retry or timeout layer.
Existing schema-conversion errors, strict runtime validation and successful translated types remain unchanged. Current string-only request and repair snapshots still match, including string and list preambles. Unsupported schema types are not silently accepted to enable image input.
Dependency manifests and lockfiles are unchanged, and importing TypeChat still works without provider SDKs or Azure Identity. Offline cases perform no image retrieval, decoding, processing or generation and add no audio/video handling, streaming or orchestration. Only reference structure, not image contents or provider-specific limits, is checked locally.
Python README and basic-usage documentation include minimal mixed text/image examples for model.complete and translate with prompt_preamble. Examples type-check offline and show hosted-image and data-URL references without new required dependencies. Documentation states the supported tags, user-only image rule, detail options, unchanged string request and message layout, caller-owned authentication/client controls, and provider/model format and size limitations.
Chat Completions coverage passes independently of Python Responses support. If separately implemented Python Responses support is present in the implementation base, combined-path offline regressions verify text/image forwarding, malformed-input handling and retained repair context. Otherwise this item adds no Responses backend, migration or dependency on that work.

Evidence limits and assumptions

No code, reproductions, tests, installations, browser actions or live-provider calls were executed. Proposed regression and type-check outcomes are criteria, not verified implementation results.
Python code and documentation were read at the trusted revision. TypeScript evidence was limited to PR 254's image-type diff, PR 519's adapter diff and PR 258's metadata; the full current TypeScript model and image example were not reviewed. SDK compatibility and provider-specific image-format and size limits were not independently verified.
Target evidence is the complete frozen issue and comment snapshot, including P1 and owner feedback; no later target comments were fetched. Related PR metadata and discussions were read separately. Contribution searches do not exclude unlinked work, and PR 241's performance claims were not independently verified.

Related contributions: #241, #254, #258, #516, #519.

The owner may accept these exact terms with /typechat accept P2. Acceptance does not start a build.

Proposal run | Approved scope policy

/typechat accept P2

Accepted work

Approved by the project owner, robgruen, for issue #517. Proposal P2.

Goal

Python applications can supply typed text and image references through existing Chat Completions model and translator context interfaces and obtain validated structured results without migrating string-only callers or changing authentication setup.

Scope

Export TextPromptContent and ImagePromptContent as tagged TypedDict definitions, ImageUrl with a required nonempty string reference and optional detail of auto, low or high, and PromptContent as a string or ordered list of those tagged parts. Widen PromptSection.content without replacing its dictionary construction interface or changing the model completion signature.
Preserve whole-string messages in every existing role. Support tagged text parts in system, user and assistant messages, but images only in user messages. Do not accept bare strings inside part lists; these tagged-only lists are the recommended Python contract, not a requirement for exact TypeScript parity.
Share structural prompt checks and string-to-user-message normalization between the HTTP and configured-client Chat Completions paths. Preserve supported parts, image references, optional detail, ordering and caller data. Return an explicit content-free Failure for malformed or unsupported input before authentication resolution or dispatch.
Use model.complete message lists and translate's existing prompt_preamble list for images. Keep translate's current request a string and retain current schema/request placement, repair roles, diagnostics, limits and failure payloads. Keep structured context independent of schema placement so future PR 241 work can reuse the same message contract without flattening images.
Separate HTTP response typing from the widened prompt input types and keep successful completions text-only. Preserve empty text, the HTTP null-to-empty behavior and configured-client missing-text failures; explicitly reject other non-text completion content.
Extend the merged Python adapter from PR 516 instead of replacing it. Preserve authentication, environment routing, per-attempt token resolution, HTTP retries/timeouts/response-size controls, cancellation and configured-client ownership and exception behavior. Use merged PR 254 and PR 519 as TypeScript design evidence and preserve credit for PR 258's image example.
Add deterministic offline behavior and static-type coverage using existing test patterns for text-only callers, mixed parts, both image-reference forms, both model paths, invalid input and translator repair. Add combined Responses regressions only when separately implemented Python Responses support is available, without introducing that implementation here.
Update Python README and basic-usage documentation with minimal model and translator-preamble examples, tagged dictionary definitions, role/detail restrictions, unchanged request and ownership behavior, and provider-specific image-format and size limitations.

Non-goals

Do not adopt, replace, close or resolve PR 241. Do not change schema placement, system-message policy, conversation-history layout or repair roles, and do not add a structured current-request parameter to translate.
Do not implement or migrate to the Python Responses API, infer a new backend from endpoint URLs, or require waiting for the separate Responses work.
Do not fetch, decode, resize, validate image contents, generate images, host assets or guarantee that every provider or model supports image input.
Do not add audio/video input, streaming, planning, memory, agent orchestration, storage, deployment management or application-specific business logic.
Do not add dependencies, a new authentication backend, SDK imports, configured-client replacement or application-client lifecycle, retry or timeout management.
Do not change TypeScript behavior, relax schema generation or runtime result validation, or silently accept unsupported prompt or schema types.

Acceptance criteria

The public package exports PromptContent, TextPromptContent, ImagePromptContent and ImageUrl using precise dictionary types and literal tags. PromptSection retains ordinary dictionary construction and accepts whole-string content or ordered lists of tagged text and image_url parts. Existing string-only callers and documented custom model/client examples type-check without migration. Supported new examples type-check without unconstrained casts; missing fields, wrong types, unknown tags and invalid detail literals fail static checks.
Offline HTTP-transport and configured fake-client cases observe exact text, hosted-image and data-URL strings, optional detail values auto, low and high, and original message and part order. Image references must be nonempty strings; omitted detail stays omitted. Neither path mutates caller-owned sections or nested parts, drops content, coerces parts to text or accesses an image.
Both paths accept whole strings in system, user and assistant messages, tagged text lists in those roles, and tagged images only in user messages. They reject unknown roles, bare strings inside part lists, missing or wrongly typed part fields, empty image references, unknown tags and invalid detail values with a Failure locating the section or part without echoing prompt contents. Rejection precedes token-provider invocation or request dispatch and triggers neither request retries nor translator repair.
Model completion remains Result[str], with HTTP response parsing typed separately from widened prompt inputs. Empty text remains successful, HTTP null content retains its existing empty-string result, and configured-client missing-choice or missing-text responses retain explicit failures. Other existing missing-field handling is unchanged. Non-text, non-null response content cannot be returned as a successful string completion.
For each model path, deterministic translator cases with mixed image preambles cover immediate success, invalid JSON, schema-invalid output, exhausted repair and model failure during repair. Every request retains original context and part ordering. Snapshots retain the present preamble-before-schema/request layout, append rejected assistant text followed by user diagnostics, and preserve the repair limit and rejected-output payload rules. These cases do not require PR 241's layout.
Regression cases preserve existing API-key and token-provider authentication, per-attempt token resolution on retries, cancellation, HTTP timeout and response-size boundaries, and environment-based routing. Configured-client cases retain the exact instance, settings and lifecycle; exceptions and cancellation propagate, with no added retry or timeout layer.
Existing schema-conversion errors, strict runtime validation and successful translated types remain unchanged. Current string-only request and repair snapshots still match, including string and list preambles. Unsupported schema types are not silently accepted to enable image input.
Dependency manifests and lockfiles are unchanged, and importing TypeChat still works without provider SDKs or Azure Identity. Offline cases perform no image retrieval, decoding, processing or generation and add no audio/video handling, streaming or orchestration. Only reference structure, not image contents or provider-specific limits, is checked locally.
Python README and basic-usage documentation include minimal mixed text/image examples for model.complete and translate with prompt_preamble. Examples type-check offline and show hosted-image and data-URL references without new required dependencies. Documentation states the supported tags, user-only image rule, detail options, unchanged string request and message layout, caller-owned authentication/client controls, and provider/model format and size limitations.
Chat Completions coverage passes independently of Python Responses support. If separately implemented Python Responses support is present in the implementation base, combined-path offline regressions verify text/image forwarding, malformed-input handling and retained repair context. Otherwise this item adds no Responses backend, migration or dependency on that work.

Existing contributions

Proposal run | Approved scope policy

This accepts the scope above. It does not start or authorize an automated implementation run, increase any budget, or approve merging. Material scope changes require a new proposal and owner decision.
After separate owner authorization and pilot activation, use /typechat build P2.

/typechat build P2

Navigation Menu

Repository navigation

[Python] Support multimodal text and image prompts #517

Goal

Scope and acceptance criteria

Boundaries

Context

Work proposal P1

Recommendation

Changes since the previous proposal

Direction and readiness

Goal

Scope

Non-goals

Acceptance criteria

Open questions

Evidence limits and assumptions

Work proposal P2

Recommendation

Changes since the previous proposal

Direction and readiness

Goal

Scope

Non-goals

Acceptance criteria

Evidence limits and assumptions

Accepted work

Goal

Scope

Non-goals

Acceptance criteria

Existing contributions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python] Support multimodal text and image prompts #517

Description

Goal

Scope and acceptance criteria

Boundaries

Context

Activity

robgruen commented on Nov 4, 2026

github-actions commented on Nov 4, 2026

Work proposal P1

Recommendation

Changes since the previous proposal

Direction and readiness

Goal

Scope

Non-goals

Acceptance criteria

Open questions

Evidence limits and assumptions

robgruen commented on Nov 4, 2026

robgruen commented on Nov 4, 2026

github-actions commented on Nov 4, 2026

robgruen commented on Nov 4, 2026

github-actions commented on Nov 4, 2026

robgruen commented on Nov 4, 2026

robgruen commented on Nov 4, 2026

github-actions commented on Nov 4, 2026

Work proposal P2

Recommendation

Changes since the previous proposal

Direction and readiness

Goal

Scope

Non-goals

Acceptance criteria

Evidence limits and assumptions

robgruen commented on Nov 4, 2026

github-actions commented on Nov 4, 2026

Accepted work

Goal

Scope

Non-goals

Acceptance criteria

Existing contributions

robgruen commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions