Lift tracks the bare floor, not the tier
The widest gaps sit on the lowest floors (glm-5.3 13% → +75; R1 13% → +67). The narrowest sits on the highest (qwen3.8-max 25% → +38). A flagship can still floor low: step-5-preview floors 21% and lifts +63.
Same 8 tasks. Same locked ruler. 11 models from 5 vendors. Deterministic regex grading — no LLM judge, no vibes.
Reading it in one line: we ran every model the same set of tasks twice - once with no reference material (Bare floor), once with the skill handed to it (With skill). The Lift column is just how many more points it scored with the skill than without. Right percentage minus left percentage: one honest before-and-after.
| Model | Bare floor | With skill | Lift |
|---|---|---|---|
| glm-5.3Zhipu | 13% 3/24 | 88% 21/24 | +75 |
| deepseek-reasoner (R1)DeepSeek | 13% 3/24 | 79% 19/24 | +67 |
| mimo-v2.6-proMiMo | 29% 7/24 | 96% 23/24 | +67 |
| mimo-v2.6-flashMiMo | 17% 4/24 | 79% 19/24 | +63 |
| step-5-previewStepFun | 21% 5/24 | 83% 20/24 | +63 |
| deepseek-chatDeepSeek | 29% 7/24 | 88% 21/24 | +58 |
| glm-4-plusZhipu | 8% 2/24 | 67% 16/24 | +58 |
| deepseek-flashDeepSeek | 36% 8/22 | 91% 21/23 | +55 |
| qwen3.8-flashAlibaba | 29% 7/24 | 75% 18/24 | +46 |
| qwen3.8-maxAlibaba | 25% 6/24 | 63% 15/24 | +38 |
| glm-4-flashZhipu | 13% 3/24 | 17% 4/24 | +4 |
Percentages are rounded from the raw counts shown. deepseek-flash has 22/23 valid calls per arm (three transient upstream drops excluded from the denominator, as in every row).
Why a row ran at a reduced or disabled thinking setting - the per-model reasons are recorded in docs/t2-results.md.
The widest gaps sit on the lowest floors (glm-5.3 13% → +75; R1 13% → +67). The narrowest sits on the highest (qwen3.8-max 25% → +38). A flagship can still floor low: step-5-preview floors 21% and lifts +63.
Same skill, same criteria: glm-4-flash lifts +4, mimo-v2.6-flash lifts +63. Both are flash tiers. Tier names are marketing, not a capability contract.
If capability absorbed the skill, lift would compress toward zero as models get stronger. It does not: every current-gen row is still +38 or higher, and most sit at +55 to +75. The ceiling effect is local — one easy task saturates both arms, the domain tasks still miss at least one skill-unique criterion.
# reproduce one row (any OpenAI-compatible endpoint)
T2_BASE_URL=https://api.example.com/v1 \
T2_API_KEY=sk-... \
T2_MODEL=your-model \
node t2/smoke-ab.mjs
# zero-cost check first: the ruler can tell
# good answers from bad, without any API call
node t2/smoke-ab.mjs --dryNot a model ranking. A low bare floor is not “a bad model” — across 8 narrow tasks, the floor measures how well a model happens to land on these specific criteria. That is not a clean proxy for general capability.
Not an activation study. The treated arm injects the skill body directly. Real harnesses load only metadata and let the model decide whether to activate a skill — this board measures the ceiling of “body present in context”, not the activation rate.
Not free of deviations. Some reasoning models blew the harness’s 60 s guard at default thinking and were run at a reduced setting; those rows say so. Their floor is a property of that setting, not native-max capability.
One genre, not all genres. All seven skills tested here are verification-discipline / failure-forensics — each a concrete, checkable procedure. Domain knowledge, creative work, and style are out of scope; the lift is evidence for this genre, not a general claim.