Skip to content

Official MCP conformance

mcpdev scenarios, Go tests, and the official @modelcontextprotocol/conformance runner answer different questions. Do not treat a pass in one of them as a pass in the others.

What each layer covers

Layer Command What a pass means
Fixture and protocol unit tests go test ./... mcpdev and internal/testkit behave as this repo specifies
Declarative scenarios mcpdev test A target matches mcpdev's YAML/JSON assertions
Official wire-schema conformance xdrun conformance The official requirement set for one spec revision, scored at that revision's wire version

The official runner is an extra standards signal. It is not a substitute for xdrun ci, and xdrun ci does not run it. The catalog HTTP fixtures (legacy-http, modern-http) are primitive labs, not complete MCP servers.

Pin

The npm package and per-revision mapping live in testdata/conformance/pin.json.

Field Value
Package @modelcontextprotocol/conformance
Version 0.2.0-alpha.11 (exact; bump pin.json and the baselines together)
2025-11-25 fixture legacy-http (stateful initialize, session id)
2026-07-28 fixture modern-http (stateless POST, _meta + headers)
Owner mcpdev maintainers

Stable 0.1.16 has no --requirements flag and rejects 2026-07-28. This repo stays on 0.2.0-alpha.11 until a stable 0.2 release publishes both requirement sets.

npx is invoked as npx --yes @modelcontextprotocol/conformance@0.2.0-alpha.11. Do not use @latest. Each revision is a separate process: --requirements 2025-11-25 is never combined with 2026-07-28 on the same argv, and each revision writes to its own artifacts/conformance/<revision>/ directory.

A scenario that exists in both requirement sets must be run twice. A pass under 2025-11-25 is not evidence for 2026-07-28, and the reverse is also true.

Run

Requires Go and Node.js 22 or newer (npx); the pinned runner imports fs.globSync, which Node 20 does not export. .github/workflows/conformance.yml pins Node 24. The task starts testdata/servers/http-lab for the matching fixture, then the pinned runner:

xdrun conformance
xdrun conformance revision=2025-11-25
xdrun conformance revision=2026-07-28
go run ./internal/conformance/cmd/run -revision 2025-11-25

Human logs stay on stderr. Official results/ files land under artifacts/conformance/<revision>/ so the two wires cannot overwrite each other. That tree is gitignored; scheduled CI uploads it as a workflow artifact.

xdrun ci stays the developer gate (vet, lint, tests, build, security, docs). Conformance is workflow_dispatch plus a weekly schedule in .github/workflows/conformance.yml.

Expected-failure baselines

Revision File
2025-11-25 testdata/conformance/baselines/2025-11-25.yml
2026-07-28 testdata/conformance/baselines/2026-07-28.yml

Entries are official scenario names or scenario:check-id pairs. Header comments record owner and rationale. Add a name only when the failure is verified upstream or intentionally deferred for the catalog fixture. Do not copy the other revision's list to make a run green.

The runner's rules still apply: unexpected failures and stale baseline entries fail the process.

HTTP lab

testdata/servers/http-lab serves a catalog HTTP fixture at /mcp and writes ready.url in its working directory. Set MCPDEV_FIXTURE to a catalog HTTP name (legacy-http by default). This is the subprocess counterpart of testkit.StartHTTP; production packages must not import internal/testkit.