Repository navigation
SEP-1627: Conformance Testing #1627
Description
Preamble
Title: Conformance Testing
Author(s): Olivier Chafik (@ochafik) , Paul Carleton (@pcarleton)
Sponsor: @pcarleton
Status: Draft
Type: Process
Created: 2025-10-08
Abstract
This SEP proposes the creation of a compliance test suite and harness, aimed primarily at easing up development and feature parity of MCP SDKs by facilitating the comparison of their behaviour across languages in a wide range of scenarios. Associated tooling will also simplify debugging MCP interactions, including authentication and authorization flows.
Motivation
Translating the human readable specification to code requires significant time and energy. Currently, each new implementation (e.g. a new SDK) has to do this work independently. While the specification aims to be unambiguous, this re-translation has resulted in slightly different behavior. The goal of this proposal is to provide tools to reduce or remove these differences to maximize compatibility between independently built implementations.
A secondary goal is to facilitate debugging of any MCP interaction, beyond SDK development use cases: while we can be more prescriptive in how we want the official SDKs to behave, the spec often leaves a large freedom of behavior to implementations, and having a tool that can tell us when the interaction of two arbitrary MCP systems is violating the spec can help users that integrate said systems.
Specification (WIP)
Overview
#948 contains an early prototype for this SEP, which details are pretty much still in flux (feedback welcome). Here are the main ideas:
- Conformance test suite & harness:
- We build a large set of English-written test scenarios
- We implement test cases that exercise each and every of these scenarios in each SDK
- We record the transport layer chatter of client & server connections and store it as human-reviewable protocol trace “goldens”
- Golden for a given test case is are shared across SDKs
- We can develop / update new SDKs against these golden traces
- Protocol Debugger:
- Takes in a trace (set of JSON RPC Messages of an MCP connection), flags any compliance violations (broken MUSTs) warns against unimplemented recommendations (broken SHOULDs).
- Beyond JSON RPC, can analyze transport-specific compliance, and auth compliance.
- Used in the conformance test harness to pre-analyze new goldens before letting humans review them
- Can be used on live traffic (e.g. hard-wired in an SDK, or as a man-in-the-middle debugger)
Activity
[-]SEP-16xx: Conformance Testing[/-][+]SEP-1627: Conformance Testing[/+]pcarleton commentedon Oct 8, 2025
This is very much a WIP, feedback welcome. I'll look to spin up a working group as well to coordinate interested folks.
13 remaining items
pcarleton commentedon Oct 16, 2025
@domdomegg yep! I agree on not blocking on the SEP to get started. We'll want to formalize the processes for conformance testing which we'll want to have a vote on since it will affect future spec changes and all SDKs. Rough plan is to get working on this in a separate repo, and update this issue with design decisions as we go.
younaman commentedon Oct 28, 2025
@pcarleton @domdomegg
Thanks for your reply! I am so sorry for the delay because I had surgery 2 weeks ago, and I just couldn't go back to work until now! I will create a GitHub repository in my account and share the repository with all of you when I finish uploading the code. Please feel free to let me know if you have any concerns or questions. Sorry again for the delay.
github-actions commentedon Jan 27, 2026
Core Maintainer Sponsor Needed
This SEP draft has been inactive for 91 days and doesn't have a core maintainer sponsor assigned.
For a SEP to progress from draft, it needs a sponsor from the core maintainers team who can:
Current assignees: None
If you're a core maintainer interested in sponsoring this SEP, please assign yourself. If you're the author, consider reaching out to the maintainers team to find a sponsor.
This is an automated message from the SEP lifecycle bot.
sep-automation-bot commentedon Feb 23, 2026
Core Maintainer Sponsor Needed
This SEP draft has been inactive for 117 days and doesn't have a core maintainer sponsor assigned.
For a SEP to progress from draft, it needs a sponsor from the core maintainers team who can:
Current assignees: None
If you're a core maintainer interested in sponsoring this SEP, please assign yourself. If you're the author, consider reaching out to the maintainers team to find a sponsor.
This is an automated message from the SEP lifecycle bot.
sep-automation-bot commentedon Mar 16, 2026
Core Maintainer Sponsor Needed
This SEP draft has been inactive for 138 days and doesn't have a core maintainer sponsor assigned.
For a SEP to progress from draft, it needs a sponsor from the core maintainers team who can:
Current assignees: None
If you're a core maintainer interested in sponsoring this SEP, please assign yourself. If you're the author, consider reaching out to the maintainers team to find a sponsor.
This is an automated message from the SEP lifecycle bot.
sep-automation-bot commentedon Mar 30, 2026
Core Maintainer Sponsor Needed
This SEP draft has been inactive for 152 days and doesn't have a core maintainer sponsor assigned.
For a SEP to progress from draft, it needs a sponsor from the core maintainers team who can:
Current assignees: None
If you're a core maintainer interested in sponsoring this SEP, please assign yourself. If you're the author, consider reaching out to the maintainers team to find a sponsor.
This is an automated message from the SEP lifecycle bot.
sep-automation-bot commentedon Apr 13, 2026
Core Maintainer Sponsor Needed
This SEP draft has been inactive for 166 days and doesn't have a core maintainer sponsor assigned.
For a SEP to progress from draft, it needs a sponsor from the core maintainers team who can:
Current assignees: None
If you're a core maintainer interested in sponsoring this SEP, please assign yourself. If you're the author, consider reaching out to the maintainers team to find a sponsor.
This is an automated message from the SEP lifecycle bot.
sep-automation-bot commentedon Apr 27, 2026
Core Maintainer Sponsor Needed
This SEP draft has been inactive for 180 days and doesn't have a core maintainer sponsor assigned.
For a SEP to progress from draft, it needs a sponsor from the core maintainers team who can:
Current assignees: None
If you're a core maintainer interested in sponsoring this SEP, please assign yourself. If you're the author, consider reaching out to the maintainers team to find a sponsor.
This is an automated message from the SEP lifecycle bot.
jashshah999 commentedon May 5, 2026
For the schema quality subset of conformance testing (missing descriptions, unconstrained params, naming anti-patterns), mcpeval handles this today as a standalone CLI.
It checks 13 anti-patterns that cause LLMs to misuse tools, gives a score, and can block CI on failures. Also has an LLM simulation mode that tests whether models actually select the correct tool given your schemas.
Happy to contribute any schema validation rules to the official conformance suite once the spec solidifies.
sep-automation-bot commentedon May 11, 2026
Core Maintainer Sponsor Needed
This SEP draft has been inactive for 194 days and doesn't have a core maintainer sponsor assigned.
For a SEP to progress from draft, it needs a sponsor from the core maintainers team who can:
Current assignees: None
If you're a core maintainer interested in sponsoring this SEP, please assign yourself. If you're the author, consider reaching out to the maintainers team to find a sponsor.
This is an automated message from the SEP lifecycle bot.
sep-automation-bot commentedon May 25, 2026
Core Maintainer Sponsor Needed
This SEP draft has been inactive for 208 days and doesn't have a core maintainer sponsor assigned.
For a SEP to progress from draft, it needs a sponsor from the core maintainers team who can:
Current assignees: None
If you're a core maintainer interested in sponsoring this SEP, please assign yourself. If you're the author, consider reaching out to the maintainers team to find a sponsor.
This is an automated message from the SEP lifecycle bot.
sep-automation-bot commentedon Jun 8, 2026
Core Maintainer Sponsor Needed
This SEP draft has been inactive for 222 days and doesn't have a core maintainer sponsor assigned.
For a SEP to progress from draft, it needs a sponsor from the core maintainers team who can:
Current assignees: None
If you're a core maintainer interested in sponsoring this SEP, please assign yourself. If you're the author, consider reaching out to the maintainers team to find a sponsor.
This is an automated message from the SEP lifecycle bot.
sep-automation-bot commentedon Jun 22, 2026
Core Maintainer Sponsor Needed
This SEP draft has been inactive for 236 days and doesn't have a core maintainer sponsor assigned.
For a SEP to progress from draft, it needs a sponsor from the core maintainers team who can:
Current assignees: None
If you're a core maintainer interested in sponsoring this SEP, please assign yourself. If you're the author, consider reaching out to the maintainers team to find a sponsor.
This is an automated message from the SEP lifecycle bot.
localden commentedon Jun 26, 2026
Superseded by #2484