Results matrix

*Grader artifact: extract-short-total asserted the comma-formatted "8,388.60"; two models answered the numerically-identical "8388.60". Assertion bug, not model failure - EVAL-DESIGN LESSON: assert on normalized values, or accept both formats. Fixed in the suite for future runs (both formats now documented as acceptable).

The one real capability failure

Qwen3-4B classified BOTH spam emails as "urgent" (prize scam and crypto scam). Business shape: a 4B email triager pages you for scams - urgency-bias in small models is exactly the failure mode a triage deployment must test for. Every 20B+ model got all six triage cases right.

Prediction resolutions (registered in the spec before runs)

• P1 (every model >=80% on short variants): HOLDS after grader correction (worst: Qwen3-4B 10/12 = 83%); on raw grading the 4B lands 75% - the margin is entirely the grader artifact. Recorded both ways.
• P2 (long-extraction separates the field): MISS - nobody failed ANY long variant. Needle-retrieval from 12K tokens of real bench log is below the capability floor of even the 4B. Good news for clients (12K-context extraction is safe across the fleet); the separating eval needs harder long-context tasks (multi-hop, cross-reference, contradiction-finding) - noted for the suite's next iteration.
• P3 (4B shows the largest drop): PARTIAL - it is the only model with real failures, but they are SHORT-context judgment failures (spam), not the predicted long-context failures. The small-model tax is judgment, not context reach.
• P4 (Flash quality vs its audition case): 17/17 - the strongest pilot-level support possible for the Phase A "quality + MIT licence" case. The cost is now measured elsewhere: 7m13s wall vs ~1m25s for every other model (the F8 prefill collapse on 12K prompts, in a real workload), plus F18's anti-scaling. Flash's niche on this box: high-quality SINGLE-USER assistant on short-to- medium context.

Consulting-ready statements this pilot supports

• "Every model 20B+ we tested passed 100% of a mixed short/long task suite; the 4B passed everything except spam judgment."
• "12K-token document extraction is reliable across the whole fleet, including the 2.3GB model."
• "The quality question between the workhorse and GLM-Flash is a tie on this suite - the throughput and concurrency data (8.5x, F18) make the deployment decision instead."
• Machinery: the suite is reusable (spikes/eval-pilot/), runs 5 models unattended in ~13 minutes, and grades deterministically.