Transformers Model Integration Contracts

Hugging Face engineer Tarek Ziadé has outlined Model Integration Contracts, or MICs, a proposed standard for catching regressions in the Transformers library before they land in main. The post's motivating case is a bug introduced on July 30, 2026, when enabling PyTorch's SDPA attention for T5 let UMT5's decoder see future tokens during training. Nothing crashed, and the flaw persisted until a September 28 fix.

Each contract combines a shrunk fixture built from a real checkpoint's configuration, a runnable Markdown usage example, and human-reviewed baseline outputs. Hugging Face says Transformers covers more than 750 model types and that running its 200,000-plus GPU tests on every pull request is impractical, so these fixtures run on cheap CPU machines instead. The UMT5 fixture has 66 MB of weights against 1,227 MB for the source, while Llama 4 Scout drops from 108.6 billion parameters to 44.7 million.

When the team replayed five months of commits, the UMT5 fixture flagged the bug throughout its lifespan, though only on the SDPA path; eager attention missed it. In a CI demonstration, the T5 contract's runner step took 13.4 seconds. An LLM can draft contracts through the tmic command-line tool, but changed expectations need human approval.

The evidence is still limited. Random-weight Whisper fixtures caught 18 of 24 synthetic faults versus 24 for overfit ones, yet both missed an assisted-decoding bug exposed by the real checkpoint. As of November 1, the CI job reports failures without blocking pull requests, since real-world catch rates, false alarms and review costs remain unmeasured.