Your Experts Have Partners. There's HOPE for Keeping Them Together

Researchers at Amazon have introduced HOPE (Higher-Order Pruning of Experts), a method for shrinking Mixture-of-Experts language models by deleting experts while keeping track of how those experts work together. In a Hugging Face blog post published November 9, 2026, authors Alex M. Tseng and Luca Zancato note that MoE models must hold every expert in memory even though only a few run for each token. Qwen3.5-122B, for example, has 256 experts in each of its 48 MoE layers, and those experts make up about 95% of a roughly 250 GB footprint, yet only 8 per layer are active per token.

Existing pruning methods give each expert its own importance score. The authors say this ignores the fact that experts form stable cooperative groups. HOPE instead builds a matrix of pairwise interaction scores from a calibration pass and solves a quadratic program to pick which experts to cut. They report that the extra solving step takes about one to two seconds per layer. The matrix's diagonal nearly reproduces the rankings of REAP, a leading first-order method (Spearman correlation 0.988), so the authors attribute HOPE's gains to the interaction terms.

The team tested Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air against four baselines at pruning rates from 10% to 50%. HOPE had the best average rank overall, and its lead grew at 50% pruning. On SWE-Bench Pro, the 50%-pruned Qwen3.5-35B-A3B scored 42.3 with HOPE, compared with 41.8 for the unpruned model and 36.2 for REAP.

The authors acknowledge limits. The advantage shrinks at low pruning rates, and the method prunes each layer independently. Code for the method has been released by AWS Labs.