Contents

sparkbench Canonical Benchmark Report

Hardware: GMKtec EVO-X2, AMD Ryzen AI Max+ 395, 128GB LPDDR5X-8000, 2TB NVMe OS: Ubuntu Server 24.04, HWE kernel 6.17.0-35-generic GPU Memory: 105.5GB GTT (in-tree TTM: ttm.pages_limit=27648000); Vulkan heaps 35.82 + 71.65 GiB (Mesa 1/3 host-visible / 2/3 device-local) + 2 GiB BIOS UMA CPU memory bandwidth (STREAM-style, tools/membw.c): read 118.7 / copy 110.4 / triad 107.4 GB/s Inference stack: llama.cpp commit 067de937183141f54c681ed684f540706d2c420a (build b200), Vulkan RADV + Mesa 25.2.8; ROCm 7.2.4 (HIP) added 2026-07-05 for the backend comparison Report generated: 2026-07-05 Benchmark period: 2026-07-03 to 2026-07-05 Licence: © 2026 Alastair McDermott / HumanSpark - CC BY 4.0 (reuse freely with attribution and a link to humanspark.ai)

Bottom Line

A €2,960 mini-PC the size of a large hardback book runs capable AI entirely on your own desk, with no data leaving the building. Everyday chat runs faster than you can read (~93 tokens/sec), a one-hour meeting transcribes in about two minutes, and running costs are a few euros of electricity a month - hundreds of times cheaper per word than a top cloud AI. On routine business work - drafting, summarising, pulling facts from documents - the local models matched the current top cloud model in our test; on the very hardest reasoning the big cloud models are still expected to lead, and we do not claim otherwise. One box comfortably serves a small team: because it processes requests together (batching), extra concurrent users cost almost nothing in energy and total throughput actually rises (§3), and it doubles as an overnight batch worker - only a large, always-on service with many heavy users at the same instant needs more than one box. Setup is not yet plug-and-play; it needs someone technical today. Everything below is the measured evidence, with every number traceable to a raw data file.

Why you can trust these numbers. Three habits set this apart from typical vendor benchmarks. (1) Every prediction was written down before the test ran, so the goalposts couldn't quietly move - and where a prediction missed, the miss is published as a finding. (2) Every AI model was checksum-verified against its original source before use, so it is clear exactly what was tested. (3) All the raw measurements are published alongside this report, so any claim can be independently checked. And we state plainly what we did not test.

How to read this report and verify it

This is a reference document, not a narrative. It is built so that every number, claim, and recommendation can be independently verified from the sparkbench repository - a reader making a purchasing decision should be able to back up any line without asking anyone. Conventions:

Section 0: Headlines

The most decision-relevant numbers, each with a plain-language translation and its source. Fuller detail in the referenced sections.

# Claim Number Human translation Source
1 Everyday chat speed (30B MoE) 92.8 t/s Faster than you can read Qwen3-30B-A3B tg128, bench-step6-qwen3-30b-a3b-2507.md; §2
2 Meeting transcription 28× real-time A 1-hour meeting transcribed in ~2 minutes whisper large-v3-turbo, capability-probes.md §3.1; §6
3 Text-to-speech (CPU only) 26.6× real-time 10 minutes of narration in 23 seconds Piper en_GB-alba, capability-probes.md §3.3; §6
4 Architecture beats family 8.5× Same size + quant, MoE generates 8.5× faster than dense Qwen3-30B-A3B 92.83 vs Qwen3-32B 10.89; F4; §2b
5 Cost of a million output tokens €0.04-0.09 Electricity is a rounding error E12, bench-e12-ws{1,16}.json; §12
6 Fleet serving is energy-free +2.3 W 16 simultaneous users cost 2.3 watts more than 1 E12 idle 4W → 1-slot 80.7W → 16-slot 83.0W; §12
7 Meeting → written summary ~15 s (5.5-min clip) Audio to summary in seconds, nothing leaving the box E17 pipeline; §6
8 Throughput is predictable 6/6 in band Six unseen models priced correctly before download corridor rule F10; §1, §2
9 Local vs frontier API ~285-730× cheaper Per output token vs the current frontier flagship (gpt-5.5); cheaper cloud tiers ~50-240× §12
10 Long-doc prefill on ROCm (pure-MoE) up to 8× 96K-token document ingest ~17 min → ~2-3 min E18a / F23; §7
11 Local image generation 9.5 s / 48 s A finished 512²/1024² image locally, no per-image billing FLUX.1-schnell, capability-probes.md §3.2; §6
12 Can it run Meta's biggest? yes, 18.5 t/s A 109B-parameter Llama runs wholly on one box Llama-4-Scout, bench-survey-scout-*.md; §2

The single most important sentence for a buyer: on this hardware, how a model is built (architecture) matters more for speed than who built it (family) - an 8.5× swing from architecture alone (#4) - and the throughput of any model is predictable from its file size and architecture before you download it (#8).

What this report measures - and what it doesn't. It measures throughput, capability, energy, and cost, on one machine and one software stack. Output quality vs frontier models was tested on a baseline task suite (§13, measured 2026-07-06) where local 20B+ models are competitive - but was not tested on the hardest multi-step reasoning, where frontier APIs are expected to lead. Read the speed and cost figures as "how fast, how cheap, how private," not "as capable as GPT-5.5 on everything." Every number is one machine, one point in time; the constants move with driver releases.

Section 0b: Human-Scale Reference

Maps raw tokens/second to human experience. Used throughout the model cards. (Threshold convention per the report instruction; a separate audience-facing blog may use a 50/20 convention - flagged in §10 as a reconciliation item, it does not affect this report's internal consistency.)

Threshold t/s What it feels like
Instant 70+ No perceptible wait. Output streams faster than you read. Feels like the model is typing for you.
Comfortable 40-70 A slight pause, then smooth streaming. Fine for interactive work.
Reading speed ~24 Output appears at roughly the pace you read it. Usable, not snappy.
Noticeable wait 10-24 You are aware of waiting between chunks. Tolerable for non-urgent work.
Batch only <10 Too slow to sit and watch. Queue it and collect the output later.

For non-LLM capabilities, measured as real-time factor (RTF = output duration ÷ processing time):

RTF Human translation
>10× Effectively instant for practical purposes (an hour of audio in minutes).
1-10× Faster than real-time. Good for batch pipelines.
<1× Slower than real-time. Offline/overnight only.

Section 0c: Quick Pick - which tool for which job

A one-glance decision table; each row links to the measured basis.

Job Use Measured basis
Interactive chat / drafting Qwen3-30B-A3B (Vulkan) 92.8 t/s, instant to ~6,900 tok (§2.1)
Coding assistant Qwen3-Coder-30B (Vulkan) 93.0 t/s (§2.13)
Modest / smaller hardware Qwen3-14B or Phi-4 ~24 t/s dense (§2.2, §2.14)
Meeting transcription whisper large-v3-turbo 28× real-time (§6.1)
Audio narration / accessibility Piper en_GB-alba (CPU) 26.6× real-time (§6.2)
Long-document / RAG ingest pure-MoE on ROCm prefill up to 8× Vulkan at depth (§7)
Overnight batch (many jobs) Qwen3-30B-A3B, 16 slots 193 t/s aggregate (§3)
Blog / illustration images FLUX.1-schnell + upscaler ~9.5-48 s, then 2.9 s to 4K (§6.3)

Not a fit for: 2+ concurrent interactive users on one box (§13); anything needing legible text inside a generated image (§6.3); the hardest multi-step reasoning, where frontier APIs still lead (§13).

Section 0d: Plain-English Glossary

The handful of terms the rest of this report leans on, in plain language.

Token
Roughly a word-piece - the unit AI reads and writes. About 750 words ≈ 1,000 tokens.
Tokens per second (t/s)
The model's typing speed. 70+ feels instant; ~24 is your reading pace; under 10 means "queue it and come back."
Context
How much text the model can hold in mind at once - the conversation or document it's working from. This box handles ~100,000+ tokens.
Prefill vs generation
Two phases of a request: prefill is the model reading your prompt/document; generation is it writing the reply. Long documents are mostly prefill.
Dense vs MoE
Two model builds. "Dense" uses all of itself for every word; "mixture-of-experts" (MoE) wakes only the part it needs - much faster at the same size.
Parameters (e.g. "30B")
The model's size - here, 30 billion internal values. Bigger usually means more capable but slower and heavier.
Quantization (Q4, Q8…)
Compressing a model to save memory and gain speed, at a small quality cost. Q4 is the common sweet spot.
Backend (Vulkan / ROCm)
The software "engine" that drives the AI chip. Two options here, each faster at a different job.
Concurrency
How many people the box serves at the same moment. Interactive speed is a one-at-a-time property.
Unified memory
One 128 GB pool shared by the processor and the AI chip - what lets a small box run large models.
RAG
"Retrieval-augmented generation" - pointing the AI at your own documents so it answers from them.
Local / on-prem
Running the AI on hardware you own, so no data leaves the building.

Section 1: Validated Principles

Each principle states the finding, the data that validates it, and its boundary conditions. Constants carry their n (number of measured points).

1.1 The Corridor Rule (F10)

Generation speed on this hardware is bandwidth arithmetic:

t/s ≈ (220 GB/s GPU-aggregate memory bandwidth ÷ active bytes per token) × architecture factor × quantization factor

where active bytes per token = (file bytes ÷ total params) × active params. Measured architecture and quant factors (fraction of the naive bandwidth ceiling actually achieved):

(Note on the 220 GB/s: this is the GPU-aggregate memory bandwidth the inference engine sustains against the LPDDR5X-8000 bus - not the header's 107-118 GB/s, which is single-thread CPU STREAM and does not saturate the bus. §3's co-residency ledger measures ~207 GB/s of usable aggregate in practice; 220 is the empirical fit that makes the dense factor land at ~1.0.)

Factor Value n Evidence
Dense ~0.99-1.00 8 Qwen3-14B 1.00, Qwen3-8B 0.997, Llama-3.1-8B 0.989, Phi-4 0.99, Mistral-24B 0.98, Qwen3-32B 0.98, R1-Distill-32B 1.00, Coder-32B 1.00 (§2)
Small-dense (<~5 GB artefact) ~0.88 2 Qwen3-4B 0.887, Phi-4-mini 0.877 (overhead onset; §2)
Pure MoE 0.84-0.86 2 Qwen3-30B-A3B 0.84, GLM-4.5-Air 0.86
Classic big-expert MoE (Mixtral) 0.94 1 Mixtral-8x7B (§2)
Chunked-attention MoE (llama4) 0.87 1 Llama-4-Scout
MLA / deepseek2 0.66-0.80 (NOT one class) 2 GLM-4.7-Flash 0.66, DeepSeek-V2-Lite 0.80 (F12)
DeltaNet hybrid ~0.55 1 Qwen3.6-35B-A3B (F7)
MXFP4 quant (multiplier, tg only) ~0.81 1 GLM-4.7-Flash same-model pair (F11); pp unaffected (+4.5%)

Prospective validation: six models never previously run - Llama-4-Scout, Mistral-Small-24B, Gemma-3-27B, Phi-4, Qwen3-32B, DeepSeek-R1-Distill-32B - were priced before download; all 6 landed inside their pre-registered bands (actual/predicted ratios 0.87-1.00). Quant-ladder rungs also landed at prediction (0.98-1.03). Boundary: one machine/stack; factors expected to move with kernel releases (F10 status note) - the method (pre-registered prediction, same-model isolation) is the durable asset, not the exact constants.

What does that mean? You can predict how fast any AI model will run on this box before you download it - from just its size and type. That turns "try it and see" into a quick calculation, so you never waste a day on a model that was always going to be too slow.

1.2 MoE-first (F4)

At matched family / generation / quant / size class, mixture-of-experts generates dramatically faster than dense because it reads far fewer bytes per token: Qwen3-30B-A3B (MoE, ~3.3B active) 92.83 tg / 1137.67 pp vs Qwen3-32B (dense, 32.8B active) 10.89 tg / 198.20 pp - 8.5× generation, 5.8× prefill (bench-step6-qwen3-30b-a3b-2507.md, bench-survey-qwen3-32b.md). The dense model reads all ~18 GB of weights per token; the MoE reads ~2 GB. Earlier Phase-A comparison vs the smaller dense Qwen3-14B was 3.8×; the gap grows with matched parameter count. Boundary: MoE wins throughput; it says nothing about output quality (see §8).

What does that mean? AI models come in two builds. A "dense" model uses all of itself for every word; a "mixture-of-experts" (MoE) model wakes only the small part it needs - a firm of specialists rather than one generalist doing everything. Same size, same maker, but the MoE runs about 8.5× faster here. Which build you pick matters more than which brand.

1.3 Quantization speed cost = the file-size ratio (§5)

tg scales inversely with artefact bytes at every measured rung, within 3% of the corridor prediction. Mistral-Small-24B ladder: Q4_K_M 15.07 → Q5_K_M 12.97 (−13.9%) → Q6_K 11.59 (−23.1%) → Q8_0 8.83 (−41.4%) t/s (bench-ladder-*.md). Implication: quantization is a pure quality decision - the speed cost is known before download because it is the size ratio. (Prefill does not follow the same line; kernel-path dependent - recorded, not attributed.)

1.4 Depth slopes are architecture-determined and quant-invariant (F5, F11)

tg degradation with context depth ranks by architecture: DeltaNet hybrid flattest (−18.5% by 32K), then chunked/SWA, then pure-MoE, with MLA steepest on prefill. The GLM-4.7-Flash Q4/MXFP4 same-model pair shows depth slopes are unchanged by quantization (the MXFP4 tax is a level shift, not a slope change). Special case: Llama-4-Scout is depth-flat - a sawtooth with period 8192 (chunk-boundary resets), net −7% over 2K→16K (F13/E1; §4).

1.5 Concurrency scaling is architecture-dependent (F17, F18, E4)

16-slot ÷ 1-slot aggregate throughput ratios: chunked-attention Scout 2.72× (at 8 slots), pure-MoE workhorse 2.66×, gpt-oss/SWA 1.82× (knee at ~4 slots), MLA GLM-4.7-Flash 0.70× - it ANTI-scales (16 users get less total throughput than one). Proven architectural, not quant, by the same-model quant pair (E2). The single-user speed ranking and the fleet ranking are different orderings - you cannot size one from the other (§3).

1.6 KV-cache quantization is a depth lever; flash attention is free (F16)

Same-model isolation (Qwen3-30B, bench-step8-block3d-*.md): q8_0 KV costs −3.8% at d0, breaks even ~4-6K, and pays +8.5% at 16K / +14.7% at 32K while halving KV memory. Flash attention (-fa on) is free - auto already selects it. Rule of thumb: fresh-chat workloads skip q8 KV; long-context workloads want it.

1.7 Ingest during inference is safe on unified memory (F15)

Contamination experiment (Step 8 Block 3c): a ~17 MB/s download shifted generation −0.6% (null); a sustained ~GB/s disk read+hash left the mean intact (+1.4%) but inflated within-run variance ~4×. UMA bus contention arrives as jitter, not throughput loss - you can ingest data while serving inference; latency variance rises, mean holds.

1.8 Energy: batching is free (E12)

Package power (a lower bound on wall power): idle 4.04 W, one user 80.7 W, sixteen users 83.0 W. Sixteen concurrent users cost 2.3 W more than one - the memory bus is saturated either way. Cost per million output tokens €0.092 (1 user) → €0.036 (16 users) at €0.30/kWh (§12). Concurrency improves the economics exactly as fast as you find demand.

1.9 Backend choice is workload- AND architecture-dependent (F22, F23)

Vulkan wins generation at every depth for every architecture. For prefill, it splits: on pure MoE, ROCm wins and the lead explodes with depth (1.79× at 32K → 8.08× at 96K as Vulkan's prefill collapses); on MLA, Vulkan wins prefill too (ROCm never wins). Vulkan is the safe default; ROCm is a targeted win for pure-MoE long-document work only (§7).

1.10 Measurement integrity is half the result (F15, F20, and campaign-wide)

Repeatedly, an apparent finding was the measuring instrument: single-run noise ±1.5% (F15); the same audio scored 13.5% or 2.8% WER depending only on normalization (E11); eval verdicts flipped on max_tokens for reasoning-channel models (F20); a 114 KB corpus was 40,727 tokens on one tokenizer and under 36,864 on another (E8). Rule enforced throughout this report: no throughput or quality number is comparable to another unless both were measured the same way, and every ruler is stated.

1.11 Operational facts that bite

Section 2: LLM Benchmarks

One card per model, in test order. Canonical spec unless noted: llama-bench, defaults (pp512, tg128, 5 reps, ngl −1 full offload), Vulkan RADV, single invocation -d 0[,4096],8192[,16384,32768]. Models >~60 GiB weights use split depth invocations (F6). Corridor prediction shows the full arithmetic. Provenance (revision/size/SHA256) for every artefact is in manifests/MANIFEST.md.

2.1 Qwen3-30B-A3B-Instruct-2507 - THE WORKHORSE

Field Value
Architecture MoE (qwen3moe)
Total / active params 30.53B / ~3.3B
Quant Q4_K_M
File / size Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf / 18,556,686,752 B (17.28 GiB)
Memory full Vulkan offload; KV ~96 KiB/token
Flags none (variants: -fa on, -ctk/-ctv q8_0)

Performance (bench-step6-qwen3-30b-a3b-2507.md, bench-step8-depths-qwen3-30b.md):

Metric Value ±
pp512 1137.67 (step6) / 1140.72 (step8) 11.04 / 8.53
tg128 92.83 (step6) / 92.28 (step8) 0.46 / 0.41
tg128 @ 4K 76.30 0.94
tg128 @ 8K 66.97 (step6) / 67.06 (step8) 0.53 / 0.44
tg128 @ 16K 53.21 0.47
tg128 @ 32K 38.32 0.25

Corridor check: active bytes ≈ (18.56e9/30.53e9)×3.3e9 = 2.01 GB/token → 220/2.01 = 109 t/s naive × 0.84 pure-MoE = 91.9 predicted; actual 92.83; ratio 1.01. Context depth: instant (70 t/s) crossed at ~6,900 tokens (~7,100 with q8 KV); reading speed (24) not reached by 32K (38.32). Conflict note (rule 8): 92.83 (step-6 dedicated matrix) vs 92.28 (step-8 depth sweep) - both controlled, within the ±1.5% noise floor; 92.83 canonical. Repeatability triplet 90.63/93.41/92.30 (CV 1.52%, F15). Notes: the default recommendation for interactive chat. Concurrency 72.9→193.5 agg (1→16 slots, F17; §3). 100K probe: 84,295-token recall 3/3, first-token 1006 s / cached 2-5 s (E15; §4). Energy reference model (E12; §12).

2.2 Qwen3-14B - dense reference

Field Value
Architecture dense (qwen3)

Performance (bench-step6-qwen3-14b.md, bench-step8-depths-qwen3-14b.md): pp512 633.72/621.66, tg128 24.39/24.34 (±0.04/0.03); tg@4K 22.65, @8K 21.19, @16K 18.85, @32K 15.47. Corridor check: 220/9.0 GB × 1.0 dense = 24.4 predicted; actual 24.39; ratio 1.00 (the dense reference point). Context depth: never instant; reading speed crossed ~1,200 tokens. Notes: validates F4 (the 30B MoE generates 3.8× faster than this while stronger). Cross-run repeatability +0.2%.

2.3 gpt-oss-120b - HOUSE REFERENCE BASELINE

Field Value
Architecture MoE, native MXFP4 (non-expert tensors Q8_0)

Performance (bench-step5-gate.md, bench-step5-longctx.md): pp512 545.65 ±10.01, tg128 53.44 ±0.38; @8K pp 423.71, tg 48.35 ±0.45. (4K/16K/32K not measured.) Corridor check: ~2.6 GB effective active bytes → ~84 naive × MXFP4 → ~54 predicted; actual 53.44; ratio 0.64 of naive (first MXFP4 cluster point, F3→F11). Notes: the Step-5 stack-validation gate (PASS vs community EVO-X2 reference 517.61 pp / 45.54 tg: +5.4% pp, +17.3% tg, attributed to newer Vulkan MoE kernels). This is the regression baseline for stack bumps (WORKPLAN §3). xet-backed repo published no upstream hashes; local SHA256s are its provenance.

2.4 GLM-4.5-Air - quality-leaning middle (memory edge, F6)

Field Value
Architecture MoE (glm4moe 106B.A12B)

Performance (bench-step6-glm45air*.md): pp512 232.38 ±8.14, tg128 23.90 ±0.09; @8K pp 55.24 ±11.68 (21% CV, memory-pressured - low confidence), tg 20.74. Corridor check: ~7.9 GB active → ~28 naive → ~24 predicted; actual 23.90; ratio 0.86. Context depth: never instant; below reading speed from d0. Notes: confirms a 110B model can generate at the same ~24 t/s as a 14B (F5 corridor arithmetic) - a headline surprise (§11). Thinking/non-thinking modes are an eval-time split, not a throughput split. Held from further throughput work per the step-7 brief (Phase-B quality audition candidate).

2.5 gpt-oss-20b

Field Value
Architecture MoE, native MXFP4, alternating/SWA attention

Performance (bench-step7-gptoss20b.md, bench-step8-depths-gptoss20b.md): pp512 1366.71/1321.83, tg128 77.76/75.22; @4K 71.24, @8K 70.65/67.51, @16K 63.57, @32K 56.08. Corridor check: ~113 naive → actual 77.76 = 0.69 (second MXFP4 point, F11). Context depth: instant crossing measurement-sensitive (~5.5K on step-8 numbers, >8K on step-7 - flat curve at the threshold; cross-evening drift −3.3%/−4.4% exceeds the ±1.5% floor, flagged in F15). Notes: concurrency knee at ~4 slots (1.82× at 16, F17; §3). Harmony reasoning channel leaks into content under --jinja, consuming max_tokens (F20; §8).

2.6 Qwen3.6-35B-A3B - DeltaNet hybrid

Field Value
Architecture hybrid MoE (qwen35moe; DeltaNet linear attn + gated attn)

Performance (bench-step7-qwen36-35b.md, bench-step8-depths-qwen36-35b.md): pp512 996.40/982.86, tg128 59.30/58.71; @4K 56.53, @8K 55.14, @16K 52.47, @32K 47.87. Corridor check: pre-registered band 80-90; actual 59.30 - MISS, 26% below floor → the hybrid pays a per-token cost the weight-bytes model misses (F7). Post-hoc factor ~0.55. Context depth: never instant; but the flattest tg slope in the matrix - overtakes the workhorse at ~16K depth (F14; §4) and holds 47.87 at 32K (best deep-context tg). Notes: ubatch sweep (bench-step8-block3a-*.md) found NO Qwen3-Next-style prefill regression at default; a mild +9% optimum at ub1024 for long prompts.

2.7 GLM-4.7-Flash (Q4_K_M) - MLA workhorse hedge

Field Value
Architecture MoE + MLA (llama.cpp deepseek2 graph, "30B.A3B")

Performance (bench-step7-glm47flash.md, bench-step8-depths-glm47flash-q4.md): pp512 943.49/922.24, tg128 71.93/70.93; @4K 57.12, @8K 50.74/50.50, @16K 39.21, @32K 27.91. Corridor check: 0.612 B/param × ~3.3B active → ~1.9 GB/token → ~100-120 naive; actual 71.93 = ~0.66 of naive - the MLA arch tax (F8). Context depth: instant crossed at ~230 tokens - the most fragile instant badge in the matrix (F13; §4). Notes: ANTI-scales under concurrency (0.70× at 16 slots, F18; §3). Vulkan beats ROCm on both phases for this architecture (F23; §7). Quality evals: 17/17 pilot, 6/6 hard-long - but at 5× wall-clock due to MLA prefill decay (§8). MIT licence.

2.8 GLM-4.7-Flash (MXFP4_MOE) - quant-isolation pair

Field Value
Architecture as 2.7

Performance (bench-step7-glm47flash-mxfp4.md, bench-step8-depths-glm47flash-mxfp4.md): pp512 985.57/968.27, tg128 62.94/63.44; @8K 45.43, @32K 26.29. Finding (F11): 7.3% smaller artefact yet 12.5% slower generation (81% of Q4's per-byte efficiency); prefill +4.5% above Q4. The MXFP4 tax is tg-only, isolated on the same model. Depth slopes quant-invariant (MLA behaviour is an arch property, cleanly separated).

2.9 Qwen3-4B-Instruct-2507 - dense boundary case

Field Value
Architecture dense (qwen3)

Performance (bench-step8-block2a-qwen3-4b.md): pp512 2051.57 (highest in matrix), tg128 78.38 ±0.64; @8K 54.12. Corridor check: 0.620×4.02 → 2.49 GB/token → 88.3 naive; actual 78.38 = 0.887 (small-dense overhead onset). Context depth: instant crossed ~2,200 tokens. Notes: quality - passed all 12K retrieval/arithmetic but failed BOTH spam-triage judgments and MISATTRIBUTED table values under enumeration (F19; §8). The judgment tax, not a context-reach tax.

2.10 Qwen3-8B - dense family point

Field Value: dense (qwen3), 8.19B, Q4_K_M, 5,027,784,512 B (4.68 GiB)

Performance (bench-step8-block2b-qwen3-8b.md): pp512 1078.08, tg128 43.56; @8K 35.23. Corridor: band 37-43; actual 43.56 = 99.7% of naive (dense at ceiling). Never instant.

2.11 Meta-Llama-3.1-8B-Instruct - dense family control

Field Value: dense (llama), 8.03B, Q4_K_M (bartowski; meta repos gated), 4,920,739,232 B (4.58 GiB)

Performance (bench-step8-block2c-llama31-8b.md): pp512 1089.93, tg128 44.21; @8K 35.96. Corridor: band 38-44; actual 44.21 = 98.9% of naive. Family-contamination check RESOLVED CLEAN: within 1% ceiling-fraction of Qwen3-8B - dense ~1.0 is a stack property, not a Qwen property (n=4 across families).

2.12 DeepSeek-V2-Lite-Chat - second MLA point

Field Value: MoE+MLA (deepseek2 16B), 15.71B/2.4B active, Q4_K_M (mradermacher), 10,364,416,768 B (9.65 GiB)

Performance (bench-step8-block2d-dsv2lite.md): pp512 1640.63, tg128 110.80 (fastest d0 tg in the matrix); @8K 44.52 (steepest 8K drop, −59.8%). Corridor / F12: amended fork (0.660 B/param → ~139 naive). Threshold ≤92 = MLA constant class, ≥99 = Flash-specific; actual 110.80 = 0.80 of naive → Flash-specific. MLA is NOT a single constant (Flash 0.66 vs this 0.80). Same graph family as GLM-4.7-Flash - the F12 discriminator.

2.13 Qwen3-Coder-30B-A3B-Instruct

Field Value: MoE (qwen3moe, workhorse graph), 30.53B/~3.3B, Q4_K_M, 18,556,689,568 B (within 3 KB of the workhorse)

Performance (bench-step9-coder30b.md): pp512 1108.96, tg128 93.05; @8K 67.09. Corridor: band 90.5-94; actual 93.05 = HIT dead-centre; @8K matches the workhorse to 3 sig figs (67.09 vs 67.06). Fine-tuning changes weight values, not weight bytes - throughput identical to the base sibling. Quality: 17/17 pilot, 5/6 hard-long (sole slip: arithmetic 493.95 for 503.95).

2.14 Phi-4 - "modest hardware floor" (14.7B dense)

Field Value: dense (llama graph, "llama 13B"), 14.66B, Q4_K_M, 8,890,306,112 B (8.28 GiB)

Performance (bench-survey-phi4.md): pp512 611.88, tg128 24.40; @4K 22.36, @8K 20.62. Corridor: band 23.5-25.5; actual 24.40 = HIT (0.99). Reading speed crossed ~800 tokens. Fifth dense point at ~100% of naive.

2.15 Mistral-Small-3.1-24B-Instruct - EU option + quant-ladder base

Field Value: dense (llama), 23.57B, Q4_K_M, 14,333,910,592 B (13.34 GiB)

Performance (bench-survey-mistral24b.md): pp512 333.97, tg128 15.07; @4K 14.42, @8K 13.81. Corridor: band 14.5-16; actual 15.07 = HIT (0.98). Never instant. The EU-sovereignty narrative model; full quant ladder in §5.

2.16 Gemma-3-27B-it

Field Value: dense (gemma3, 5:1 local:global SWA), 27.01B, Q4_K_M, 16,546,688,736 B (15.40 GiB)

Performance (bench-survey-gemma27b.md): pp512 248.43, tg128 12.56; @4K 11.91, @8K 11.65. Corridor: band 12.5-13.8; actual 12.56 = HIT at floor edge (0.94 - low edge of the dense cluster, noted not attributed). Shallowest dense tg slope (−7.2% by 8K, the SWA effect). Vision-capable (§6, E5).

2.17 Qwen3-32B - THE dense-vs-MoE comparison

Field Value: dense (qwen3), 32.76B, Q4_K_M, 19,762,150,048 B (18.40 GiB)

Performance (bench-survey-qwen3-32b.md): pp512 198.20, tg128 10.89; @4K 10.30, @8K 9.76 (an 8K-doc prefill ~95 s). Corridor: band 10-12; actual 10.89 = HIT (0.98). The centrepiece: vs Qwen3-30B-A3B - 8.5× tg, 5.8× pp at matched family/scale/quant (F4; §2b). E13 speculative-decoding target (§3).

2.18 Llama-4-Scout-17B-16E-Instruct - "can I run Llama locally?"

Field Value: MoE, chunked/iRoPE attention (llama4, 8192-token chunks), 107.77B/17B active, Q4_K_M (2 shards, unsloth), 65,359,900,352 B (60.86 GiB), full offload ~10 GiB headroom, F6 split invocations

Performance (bench-survey-scout-*.md, bench-e1-scout-*.md, all −r 3): pp512 163.17, tg128 18.54; sawtooth tg - 2K 17.60, 4K 16.64, 6K 16.75, 7.5K 16.23 (trough), 8K 17.89 (reset), 8.4K 17.63, 12K 16.65, 16K 17.24. Corridor: band 15-20; actual 18.54 = HIT (0.87). Context depth: SAWTOOTH period 8192 confirmed (E1) - depth-flat, tg at 16K within 7% of 2K. Notes: answers "can I run Llama locally?" - yes, Meta's 109B runs wholly on one box at reading speed. Best concurrency scaler (2.72× at 8 slots, E4; §3). But long-range cross-chunk recall is its weakness (E8, 4/6 at 40K; §8) - the chunk-flat speed is paid in long-range reliability.

2.19 DeepSeek-R1-Distill-Qwen-32B - reasoning representative

Field Value: dense (qwen2 graph), 32.76B, Q4_K_M, 19,851,335,584 B (18.48 GiB)

Performance (bench-survey-r1-32b.md): pp512 223.96, tg128 11.05; @4K 10.50, @8K 10.00. Corridor: band 10-12; actual 11.05 = HIT (1.00). Confirms Qwen3-32B within 1.5% (different graph generation). Reasoning cost is in TOKENS (thinking traces), not t/s: 11 t/s × 1000+-token traces = minutes per answer.

2.20 Phi-4-mini-instruct - the absolute floor (3.8B)

Field Value: dense (phi3), 3.84B, Q4_K_M, 2,491,874,272 B (2.31 GiB)

Performance (bench-survey-phi4mini.md): pp512 2149.96 (highest pp in the matrix), tg128 77.44; @4K 63.13, @8K 54.87. Corridor: band 79-90; actual 77.44 = MISS low by 2% → small-dense overhead ~0.88 systematic (n=2 with Qwen3-4B). Instant crossed ~2,100 tokens.

2.21 Qwen2.5-Coder-32B-Instruct

Field Value: dense (qwen2), 32.76B, Q4_K_M, 19,851,335,840 B (18.48 GiB)

Performance (bench-survey-coder32b.md): pp512 223.16, tg128 11.09; @4K 10.53, @8K 10.03. Corridor: band 10-12; actual 11.09 = HIT (1.00). Statistically identical to R1-Distill (same Qwen2.5-32B body).

2.22 Mixtral-8x7B-Instruct-v0.1 - classic MoE + GGUF-vintage finding

Field Value: MoE top-2-of-8 big experts (llama 8x7B), 46.70B/~12.9B active, Q4_K_M (mradermacher re-conversion), 28,448,468,384 B (26.49 GiB)

Vintage finding: the TheBloke Dec-2023 GGUF (26,441,533,376 B) DOES NOT LOAD at commit 067de937 ("failed to load model" - pre-modern MoE tensor layout); kept as evidence. The 2025 re-conversion loads. Performance (bench-survey-mixtral-v2.md): pp512 216.13, tg128 26.45; @4K 24.60, @8K 23.46. Corridor: band 21-27; actual 26.45 = HIT (0.94 of naive - above the modern fine-grained MoE cluster; simpler routing is cheaper per byte). Obsolete on merit (workhorse 3.5× faster on 40% less memory).

2.23 Qwen3-0.6B - smoke/canary model

Field Value: dense, 0.6B, Q8_0, 639,446,688 B; role: step-4a smoke test, DeviceLost canary, E13 draft model

Smoke via llama-cli (not llama-bench): prompt 1173.4 t/s, generation 253.5 t/s (single run). Finding F1: llama-cli is chat-first at this commit - scripted use hangs on EOF stdin; harness runs use llama-server.

Section 2b: Model Rankings

Cross-model comparison tables. All tg128/pp512 are single-stream llama-bench at d0 (sources in §2).

By generation speed (tg128, single-stream)

Rank Model Arch Active B tg128 Human feel
1 DeepSeek-V2-Lite MoE/MLA 2.4 110.80 Instant
2 Qwen3-Coder-30B MoE 3.3 93.05 Instant
3 Qwen3-30B-A3B MoE 3.3 92.83 Instant
4 Qwen3-4B dense 4.0 78.38 Instant
5 gpt-oss-20b MoE 3.6 77.76 Instant
6 Phi-4-mini dense 3.8 77.44 Instant
7 GLM-4.7-Flash Q4 MoE/MLA 3.3 71.93 Comfortable (Instant <~230 tok only, §4)
8 GLM-4.7-Flash MXFP4 MoE/MLA 3.3 62.94 Comfortable
9 Qwen3.6-35B hybrid 3.3 59.30 Comfortable
10 gpt-oss-120b MoE 5.1 53.44 Comfortable
11 Llama-3.1-8B dense 8.0 44.21 Comfortable
12 Qwen3-8B dense 8.2 43.56 Comfortable
13 Mixtral-8x7B MoE 12.9 26.45 Reading speed
14 Phi-4 dense 14.7 24.40 Reading speed
15 Qwen3-14B dense 14.8 24.39 Reading speed
16 GLM-4.5-Air MoE 12 23.90 Reading speed
17 Llama-4-Scout MoE 17 18.54 Noticeable wait
18 Mistral-Small-24B dense 23.6 15.07 Noticeable wait
19 Gemma-3-27B dense 27.0 12.56 Noticeable wait
20 Qwen2.5-Coder-32B dense 32.8 11.09 Noticeable wait
21 R1-Distill-32B dense 32.8 11.05 Noticeable wait
22 Qwen3-32B dense 32.8 10.89 Noticeable wait

The ranking is architecture, not size: the top 10 are dominated by MoE and small-dense models; the bottom 6 are all 24B+ dense. A 2.4B-active MoE (DeepSeek-V2-Lite, #1) beats a 32.8B dense (Qwen3-32B, #22) by 10×.

⚠ Single-user speed ≠ multi-user throughput - do not size a server from this table. These are single-stream numbers. The MLA models near the top (DeepSeek-V2-Lite #1, GLM-4.7-Flash #7) anti-scale under concurrency - 16 users get less total throughput than one (§3, F18). For a shared/multi-user server, read §3 (concurrency) before choosing; the fleet ranking is a different ordering.

By prompt-processing speed (pp512)

Rank Model pp512 Rank Model pp512
1 Phi-4-mini 2149.96 12 Phi-4 611.88
2 Qwen3-4B 2051.57 13 gpt-oss-120b 545.65
3 DeepSeek-V2-Lite 1640.63 14 Mistral-Small-24B 333.97
4 gpt-oss-20b 1366.71 15 Gemma-3-27B 248.43
5 Qwen3-30B-A3B 1137.67 16 GLM-4.5-Air 232.38
6 Qwen3-Coder-30B 1108.96 17 R1-Distill-32B 223.96
7 Llama-3.1-8B 1089.93 18 Qwen2.5-Coder-32B 223.16
8 Qwen3-8B 1078.08 19 Mixtral-8x7B 216.13
9 Qwen3.6-35B 996.40 20 Qwen3-32B 198.20
10 GLM-4.7-Flash 943.49 21 Llama-4-Scout 163.17
11 Qwen3-14B 633.72

By efficiency (tg128 per GiB of file)

Rank Model File GiB tg128 t/s per GiB
1 Qwen3-4B 2.32 78.38 33.8
2 Phi-4-mini 2.31 77.44 33.5
3 DeepSeek-V2-Lite 9.65 110.80 11.5
4 Llama-3.1-8B 4.58 44.21 9.7
5 Qwen3-8B 4.68 43.56 9.3
6 gpt-oss-20b 11.27 77.76 6.9
7 Qwen3-Coder-30B 17.28 93.05 5.4
8 Qwen3-30B-A3B 17.28 92.83 5.4

(This metric rewards small models mechanically - fewer bytes to stream. Useful for "most speed per GB of RAM committed," not for quality-per-token.)

Architecture comparison (MoE vs dense, same family)

Family MoE variant MoE tg Dense variant Dense tg MoE advantage
Qwen3 (matched size) 30B-A3B 92.83 32B 10.89 8.5×
Qwen3 (cross size) 30B-A3B 92.83 14B 24.39 3.8×

This is the single most decision-relevant comparison in the report (F4): at matched size, family, and quant, the only difference is architecture, and it is worth 8.5×.

Section 3: Concurrency Scaling

tools/serve_bench.py, native /completion, temperature 0, unique prompts, cache_prompt=false, 4096 ctx/slot, ~128-token prompt, n_predict 128, 45 s window, one thread per slot. Aggregate = generated tokens ÷ wall (includes per-request prefill = serving throughput); per-stream = mean server-side generation-phase rate. Single-window numbers (no ±); under_sampled flags <3 requests/stream. Raw: results/raw/bench-step9-*.json, bench-e2-*.json, bench-e4-*.json, bench-e6-*.json, bench-e14-*.json.

Qwen3-30B-A3B - f16 KV

Slots Aggregate Per-stream TTFT p50
1 72.85 87.81 287 ms
2 104.53 63.37 390 ms
4 137.08 42.97 695 ms
8 159.79 24.86 1154 ms
16 193.49 14.68 1677 ms

No sharp knee; still rising at 16 slots (2.66×). 193.5 agg beats the ROCm field figure (~168) by ~15%. Instant per-stream (≥70) holds only at 1 slot. Server overhead vs llama-bench: −4.8%.

What does that mean? One box serves one person at full speed. Add more people at the same moment and they share the engine, so each runs a little slower - but because the box handles requests together (batching), a small team using AI on and off through the day sits comfortably on one box, and total throughput actually climbs as users are added (§3). It also clears overnight batches - two hundred jobs done by morning. Only a large team all demanding full speed at the very same instant needs more than one box.

Qwen3-30B-A3B - q8_0 KV (F16)

Slots Aggregate Per-stream TTFT p50
1 67.97 85.00 285 ms
4 133.92 41.84 718 ms
16 192.39 14.79 1720 ms

Within −0.6% to −2.3% of f16 at multi-slot (−6.7% at 1 slot). q8 KV is effectively free under concurrency while halving KV memory.

gpt-oss-20b - f16 KV (F17)

Slots Aggregate Per-stream TTFT p50
1 59.86 71.57 273 ms
4 101.32 31.86 750 ms
16 108.99 8.23 2540 ms

Knee at ~4 slots (+7.6% from 4→16); 1.82× at 16. Architecture-dependent scaling.

GLM-4.7-Flash - Q4 vs MXFP4, the ANTI-scaling proof (E2, F18)

Config 1 slot 4 slots 16 slots ratio
Q4 aggregate 54.07 45.45 38.06 0.70×
MXFP4 aggregate 52.30 31.77 39.49 0.76×
Q4 TTFT p50 366 ms 1206 ms 19,597 ms

MLA ANTI-scales: both quants LOSE aggregate throughput as slots rise (ratios parallel → architecture, not quant). ~20 s first-token at 16 slots. MLA models are single-user machines on this stack.

Llama-4-Scout - best scaler (E4)

Slots 1 4 8
Aggregate 14.86 33.76 40.49

8-slot/1-slot ratio 2.72× - best measured. A 109B model serving 8 concurrent users at 40 t/s aggregate on one box.

Co-residency - fast lane + quality lane (E14)

Server Solo agg Concurrent agg Retained
Qwen3-30B (port 8100) 72.80 36.74 50.5%
Qwen3-4B (port 8101) 65.19 46.02 70.6%

Sum concurrent 82.76 t/s (+13.7% vs best solo). Bandwidth ledger ~92 + ~115 ≈ 207 GB/s = the full bus. The router pattern (two models resident) works without model swapping; plan capacity by adding active-byte budgets.

Mixed workload - one box, whole office (E6)

Workload Solo Mixed Cost
LLM 4-slot aggregate 137.08 55.89 −59%
whisper ×3 (330 s) ~35 s 76 s 2.17×
FLUX 1024² ~60 s 77 s 1.3×

All completed, no errors, TTFT still ~2 s. Chat throughput roughly halves under simultaneous media load; media jobs slow 1.3-2.2×. Schedule media batches off-peak or budget the chat haircut.

Speculative decoding - a workload bet, not a switch (E13, F21)

Qwen3-32B dense target + Qwen3-0.6B draft (--spec-type draft-simple), temperature 0:

Condition t/s vs 10.90 control
repetitive prompt 40.61 3.73×
explanatory prose 12.71 +17%
analytical prompt 6.07 0.56×
creative prompt 5.80 0.53×

Spec decode spans 0.53×-3.73× purely by how predictable the output is. Default OFF; enable per-workload after measuring the production prompt mix. Trap: -md loads the draft but does NOTHING without --spec-type (one silent log line). Fast-MoE targets gain little even on friendly text (+9%).

Section 4: Context-Depth Sweeps

tg128 vs context length (t/s), all Vulkan (raw: bench-step8-depths-*.md, bench-e1-scout-*.md)

Model d0 2K 4K 8K 16K 32K 64K 96K
Qwen3-30B-A3B 92.28 82.35 76.30 67.06 53.21 38.32 24.69 18.40
Qwen3-30B +q8 KV 88.66 79.64 75.34 67.98 57.61 43.75 - -
Qwen3.6-35B (hybrid) 58.71 57.08 56.53 55.14 52.47 47.87 - -
GLM-4.7-Flash Q4 70.93 62.75 57.12 50.50 39.21 27.91 - -
GLM-4.7-Flash MXFP4 63.44 56.44 52.72 45.43 36.47 26.29 - -
gpt-oss-20b 75.22 71.97 71.24 67.51 63.57 56.08 - -
Qwen3-14B 24.34 23.39 22.65 21.19 18.85 15.47 - -
Llama-4-Scout 18.54 17.60 16.64 17.89* 17.24 - - -

(*Scout sawtooth: trough 16.23 @7.5K, resets to 17.89 @8K chunk boundary - depth-flat.)

pp512 vs context length (t/s) - where prefill collapses

Model d0 8K 16K 32K 64K 96K
Qwen3-30B-A3B (Vulkan) 1140.72 561.16 346.59 180.77 53.39 16.57
Qwen3-30B-A3B (ROCm) 1205.15 752.09 528.11 323.68 189.53 133.86

The Vulkan prefill collapse past 32K, and ROCm's resistance to it, is the F23 story (§7).

Instant-death points (interpolated depth where tg crosses below 70 t/s)

Model Instant dies at Notes
GLM-4.7-Flash Q4 ~230 tokens most fragile badge in the matrix
Phi-4-mini ~2,100
Qwen3-4B ~2,200 2-point estimate
DeepSeek-V2-Lite ~3,200 2-point estimate
gpt-oss-20b ~5,500-9,000 measurement-sensitive (flat curve at the line)
Qwen3-30B-A3B ~6,900 (~7,100 with q8 KV) the workhorse
all others never held 70 -

F13: every "instant" claim has a measured expiry depth - never quote one without it. F14: architecture rank is depth-dependent - Qwen3-30B and Qwen3.6-35B tie at ~16K (53.21 vs 52.47) and the hybrid is 25% faster by 32K (47.87 vs 38.32).

100K-context probe (E15)

Workhorse, 84,295-token corpus (exact via /tokenize - not char/4), -c 114688: retrieval 3/3 including a needle at 90% depth; first-question prefill 1006 s (16.8 min) at 84 t/s average; cached follow-ups 4.6 s / 1.8 s. The 262K marketing context is real; its cost structure dictates "ingest once, then converse." (On ROCm, the same prefill would be far faster - §7.)

Section 5: Quantization Comparisons

Mistral-Small-3.1-24B quant ladder (raw: bench-ladder-*.md, bench-survey-mistral24b.md)

Quant File size pp512 tg128 tg vs Q4_K_M Corridor predicted tg
Q4_K_M 14,333,910,592 B (13.34 GiB) 333.97 15.07 baseline ~15.4
Q5_K_M 16,763,985,472 B (15.61 GiB) 236.09 12.97 −13.9% ~12.9
Q6_K 19,345,940,032 B (18.01 GiB) 249.32 11.59 −23.1% ~11.1
Q8_0 25,054,780,992 B (23.33 GiB) 282.89 8.83 −41.4% ~8.6

tg lands at 0.98-1.03 of corridor prediction at every rung - the speed cost IS the size ratio. Near-lossless Q8 runs at 59% of Q4 speed.

What does that mean? "Quantization" is compressing a model so it needs less memory and runs faster, at a small cost to quality - like an MP3 versus a studio master. The "Q4" level most people use is the sweet spot, and the speed you gain back is exactly the memory you save.

pp does NOT follow the size line (Q4 334 > Q8 283 > Q6 249 > Q5 236) - dequant kernel paths differ per quant; recorded, not attributed.

GLM-4.7-Flash Q4_K_M vs MXFP4_MOE (same-model quant isolation, F11)

See §2.8: 7.3% smaller MXFP4 artefact generates 12.5% slower (tg-only tax; prefill +4.5%); depth slopes quant-invariant.

Section 6: Non-LLM Capabilities

Raw: results/capability-probes.md, results/experiments.md (E5/E16/E17). Tool provenance + hashes in docs/TOOLCHAIN.md.

6.1 Speech-to-text (whisper.cpp, Vulkan build)

Model Audio Time RTF Result
large-v3-turbo (1.6 GB) 11 s (jfk.wav) 1.27 s (0.31 load + 0.31 encode) ~11× correct
large-v3-turbo 330 s (30× concat) 11.69 s ~28× correct
small (466 MB) 330 s 11.87 s ~28× correct

Key finding: small and large-v3-turbo run at the SAME speed - quality is free, always use turbo. Vulkan acceleration confirmed (model on Vulkan0). Accuracy round-trip (E11): Piper-read 184-word passage → whisper: naive WER 13.51%, number-normalized 7.34%, semantically-true errors ~2.8% (5 words: "Meta's Llama Scout"→"Metters Lammerscout", etc.). Errors concentrate in compound numbers and novel proper nouns. Ruler caveat: the same audio scored 13.5% or 2.8% depending only on normalization - insist any vendor states theirs. Consulting answer ("real-time meeting transcription?"): yes - a 1-hour meeting in ~2 minutes.

6.2 Text-to-speech (Piper, CPU-only, pre-built binary 2023.11.14-2)

Voice Text Gen time Audio RTF
en_GB-alba-medium 497 chars 1.13 s 30.1 s 26.6×

CPU-only - leaves the GPU free for LLM work. A 10-minute narration in ~23 s. Quality: Alastair-approved 2026-07-05 ("sounded great"), alba is the house default. Consulting answer ("local narration for accessibility/training?"): yes, unqualified.

6.3 Image generation (stable-diffusion.cpp, Vulkan build; FLUX.1-schnell Q4_0 + T5/CLIP/VAE)

Resolution Steps Time Notes
512×512 4 8.52 s sampling + 0.94 s decode = ~9.5 s clean
1024×1024 4 40.25 s + 7.23 s tiled decode = ~48 s needs --vae-tiling; WITHOUT it FAILS after sampling (8.5 GB buffer > RADV allocation cap)
1344×768 (16:9) 4 35.14 s clean
1536×864 (16:9) 4 49.40 s largest verified 16:9
1792×1008 / 1920×1080 / 3840×2160 4 FAIL compute buffer over RADV allocation cap (native 1080p/4K impossible)

50-image style gallery: 50/50 success, ~36 s each, cost style-invariant (photography, illustration, watercolor, oil, cartoon, 3D, sketch, pixel). Binding constraint is the RADV single-allocation cap (~quadratic in pixels), NOT total VRAM - practical ceiling ~1.3-1.5 MP. Upscaler (Real-ESRGAN ncnn-vulkan, E3): 1024→4096 (×4) in 2.9 s; 1536×864→3072×1728 (×2) in 1.4 s. Production 4K pipeline: generate at ceiling + upscale = ~51 s total.

Quality verdict (results/image-quality-verdict.md, 12-image sample review): strong on illustration, oil/watercolor, isometric, comic, and business photography - good enough for blog headers, slide backgrounds, and marketing illustration at local speed. Two real limits: (1) legible text inside images fails - a FLUX-schnell weakness and a hard blocker for anything with words or labels; (2) not photorealism that would fool a professional. No head-to-head vs Midjourney/DALL-E was run - any commercial comparison is qualitative judgement, not a benchmark.

6.4 Vision / image understanding (E5)

Gemma-3-27B + mmproj-F16.gguf (verified) via llama-server --mmproj. Four factual questions on own-gallery 1024px images: 4/4 correct at 2.3-2.5 s/image. Local image understanding (charts, documents, photos) works out of the box. Wire-up probe, not a benchmark; document/chart QA not yet tested.

6.5 Embeddings / RAG feasibility (E16)

Qwen3-Embedding-0.6B-Q8_0 (verified) via llama-server --embedding: 2.9 chunks/sec (~1450 tok/s = prefill physics; unoptimized lower bound); similarity sanity 5/5 triples (related cosine 0.57-0.92 vs unrelated 0.14-0.39). The Phase-A log (38 chunks) embedded in 13 s; a 10K-chunk knowledge base ~1 hour one-off. RAG axis open and viable.

6.6 End-to-end pipeline: meeting → summary (E17)

Audio → whisper large-v3-turbo → workhorse "summarise this meeting", timed end-to-end on the 330 s (5.5-min) clip: transcribe 12.4 s + server load 4.1 s + summarise 2.8 s. Warm (models resident): 15.2 s. Cold: 19.2 s. Conservative 1-hour extrapolation (not measured): transcribe ~2.1 min + summarise a real ~8-12K-token transcript (prefill-bound) → roughly 2.5-3.5 minutes audio-to-summary, nothing leaving the box. (The 2.8 s summarise here understates a real meeting - short repetitive test transcript.)

Real-document validation (2026-07-06): to replace the synthetic clip with genuine varied content, the workhorse ingested a real 75 KB / ~15K-token technical document (the project's own docs/PHASE-A-LOG.md) and answered factual questions. Ingest + first answer (full prefill) 76.7 s on Vulkan; cached follow-up questions 2.9-9.4 s each. All three factual answers were correct and grounded with verbatim quotes from the document (e.g. "gpt-oss-120b tg128 = 53.44 ± 0.38", the corridor formula, the 46-47 °C thermal envelope). This is real document-QA on non-repetitive content. The ~77 s ingest is a Vulkan figure - on ROCm the same prefill would be far faster (F23). Real multi-speaker meeting audio remains untested (no real recording on hand).

Section 7: Vulkan vs ROCm Comparison

ROCm 7.2.4 installed 2026-07-05 (amdgpu-install 30.30.4 driver stack, --no-dkms so the working amdgpu driver is untouched). gfx1151 (Strix Halo) is natively supported - no HSA_OVERRIDE_GFX_VERSION needed (a cleaner result than expected; the render group reaches /dev/kfd, no permission fix). Separate build-rocm/ HIP llama-bench (-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151); Vulkan build/ untouched. Model: Qwen3-30B-A3B (pure MoE) and GLM-4.7-Flash (MLA), both backends same session for an identical ruler. ROCm runs use --mmap 0. Raw: results/raw/bench-e18-*.md, bench-e18a-deep-*.md, bench-e18b-glm-*.md.

7.1 Pure MoE (Qwen3-30B-A3B) - E18 + E18a

Depth Vulkan tg ROCm tg tg winner Vulkan pp ROCm pp pp winner
0 92.70 71.58 Vulkan +29.5% 1130.00 1205.15 ROCm +6.6%
4K 75.91 62.85 Vulkan +20.8% 735.03 912.27 ROCm +24.1%
8K 66.79 56.40 Vulkan +18.4% 557.79 752.09 ROCm +34.8%
16K 53.03 46.81 Vulkan +13.3% 344.68 528.11 ROCm +53.2%
32K 38.06 35.32 Vulkan +7.8% 180.77 323.68 ROCm +79.1%
64K 24.69 23.50 Vulkan +5.1% 53.39 189.53 ROCm +255% (3.55×)
96K 18.40 17.83 Vulkan +3.2% 16.57 133.86 ROCm +708% (8.08×)

Generation: Vulkan wins at every depth, gap shrinking to noise (+29.5% → +3.2%). Prefill: ROCm wins at every depth, advantage EXPLODING (1.79× at 32K → 3.55× at 64K → 8.08× at 96K). Vulkan's prefill collapses past 32K (180→53→17 t/s - at 96K it falls BELOW its own generation rate of 18.40); ROCm holds (324→190→134). No OOM on either backend at 96K (ROCm sees the full 108 GB GTT as VRAM). F22, F23.

7.2 MLA (GLM-4.7-Flash) - E18b, the split INVERTS

Depth Vulkan tg ROCm tg Vulkan pp ROCm pp pp winner
0 71.32 52.98 939.52 924.50 Vulkan +1.6%
8K 50.87 42.43 321.49 300.10 Vulkan +7.1%
32K 27.84 26.20 113.87 95.38 Vulkan +19.4%

For MLA, Vulkan wins BOTH phases at every depth - ROCm never wins, and its prefill deficit widens with depth. ROCm does NOT rescue Vulkan's MLA prefill collapse (F8) - it collapses harder. This proves F22 is architecture-specific. Mechanism candidate: MLA's latent-projection matmul shapes don't favour rocBLAS the way standard-attention GEMMs do (not separable here).

7.3 Mechanism and deployment rule

Why the split: prefill is compute-bound (large batched GEMMs over the whole prompt) - ROCm's mature rocBLAS/hipBLAS kernels win on the standard-attention (pure-MoE) case, and the win compounds at depth as attention gets more compute-heavy; Vulkan's prefill kernels degrade badly past 32K. Generation is memory-bandwidth-bound (one token, streaming weights) - Vulkan's leaner per-token path wins; the HIP runtime carries more per-token overhead.

Deployment rule (business-grade, corrected from the tempting blanket version): - Generation-heavy / interactive chat → Vulkan (always, every architecture). - Prefill-heavy / long-document work (RAG ingest, summarization, extraction) on a PURE-MoE model → ROCm - a large and growing win (the E15 84K ingest ~17 min on Vulkan would drop to ~2-3 min on ROCm). - MLA models (GLM-Flash, DeepSeek) → Vulkan for everything (ROCm never wins). - Vulkan is the safe default; ROCm is a targeted optimization for one model class and one phase. This answers the Part-2 consulting question - yes, use different backends for different workloads, but only in this specific slice.

What does that mean? The AI chip has two interchangeable software "engines." One is faster at reading a long document, the other at writing the reply. You can pick the right one per job, but for everyday chat a single default is fine - this is a tuning lever, not a decision you must get right to benefit.

Section 8: Methodology

Measurement protocol

Error bars

llama-bench ± = stddev over reps. Run-to-run noise floor established by the repeatability triplet: identical canonical Qwen3-30B invocations 90.63 / 93.41 / 92.30 → CV 1.52% (F15). Treat ±1.5% as the single-run error bar. serve_bench windows are single measurements (request counts recorded; under_sampled <3/stream flagged).

Environment controls

Thermal: every leg 46-47 °C start, ~58-59 °C peak, no accumulation over 6 h. Downloads MAY overlap benches (3c: −0.6% null); sustained disk hashing avoided during CV-critical legs (mean-null, variance ×4). Memory edge >~60 GiB → split invocations (F6); DeviceLost → Qwen3-0.6B canary. Not controlled: cross-day drift (one gpt-oss-20b pair −3.3%/−4.4% exceeds the noise floor, unexplained); wall power (package sensor only); network during overlapped downloads (declared per-leg).

Promptfoo integration

User-level (Node v24.18.0 tarball + promptfoo; docs/TOOLCHAIN.md). Used for: capability spike (3/3), pilot suite, hard-long, contradiction, generation, 40K cross-chunk suites via tools/run_eval_pilot.sh (per-model server lifecycle). NOT used for: LLM-judged grading, statistical multi-sample evals, concurrency (serve_bench owns that).

Known limitations

One machine, one stack; constants carry n as listed; re-run trigger banked in WORKPLAN §3. Package power is a lower bound on wall power. Quality evals are single-shot deterministic; no human-judged tier. WER round-trip cannot separate TTS pronunciation from STT hearing. E6/E14 solo baselines are same-day, not same-minute. Scout 16-slot and 32K depth not attempted (memory-edge/time).

Reproducing a single benchmark

Any model card's pp512/tg128 reproduces with one command:

llama.cpp/build/bin/llama-bench -m <model>.gguf -d 0,8192 -o md

Add -fa on -ctk q8_0 -ctv q8_0 for the KV-quant arm; use the build-rocm/bin/llama-bench binary with --mmap 0 for the ROCm figures; extend -d (e.g. -d 0,4096,8192,16384,32768) for a depth curve. Concurrency uses tools/serve_bench.py. Every raw output sits in results/raw/ under the filename cited in each card - compare directly.

Section 9: Timeline and Session Log

Date What was done
2026-07-03 Bring-up (packages, render group, TTM); CPU membw (118.7/110.4/107.4 GB/s); llama.cpp 067de937 Vulkan build; 0.6B + 14B smokes; gpt-oss-120b reference gate PASS (house baseline)
2026-07-03 Step 6 matrix: Qwen3-30B-A3B (92.83), Qwen3-14B (24.39), GLM-4.5-Air (23.90, F6 memory edge); wired network up
2026-07-03 Step 7: gpt-oss-20b (77.76), Qwen3.6-35B (F7), GLM-4.7-Flash Q4 (MLA discovery) + MXFP4 pair (F11); corridor rule form (F10)
2026-07-03/04 Step 8 overnight: 6-model depth curves 0-32K, triplet (F15), 3c contamination, Block 2 (F12), ubatch sweeps, KV-quant (F16); findings register created
2026-07-04 Step 9 concurrency (serve_bench; F17), Coder-30B (93.05), promptfoo spike; capability briefing v2: whisper/Piper/FLUX probes, Tier-1 survey 6/6 band hits, Tier-2, quant ladder
2026-07-04 E1 sawtooth, E2 quant-pair concurrency (F18), E3 upscaler, eval pilot (F19), E4 Scout concurrency, E5 vision, E6 mixed, E7 hard-long
2026-07-04 E8 40K cross-chunk, E9 contradiction, E10 generation (F19/F20), E11 WER; documentation audit (CLAUDE.md, README, raw preservation)
2026-07-04 E12 energy, E13 spec decode (F21), E14 co-residency, E15 84K probe, E16 embeddings
2026-07-05 Pricing resolved (3 sources + OpenAI official); ROCm 7.2.4 installed (gfx1151 native); E17 pipeline; E18 Vulkan-vs-ROCm (F22); E18a deep 64K/96K + E18b MLA (F23); this report
2026-07-06 Publish-gate review + fixes; frontier quality A/B (gpt-5.5, §13); real-document validation (§6.6); image-quality verdict (§6.3); charts; server-context §17; mini-site

Section 10: Gaps and Open Questions

Not yet tested

Open questions

Candidate follow-ups

Scout finer sawtooth + cross-chunk quality; workhorse spec-decode diverse-prompt arm; embeddings throughput tuning; suite v3 (>32K adversarial, long generation, statistical repeats); promptfoo LLM-judged tier; ROCm concurrency + more architectures; soak test; wall-meter energy validation.

Section 11: Surprise Findings

Results that contradicted a prediction or expectation. These are the moments the data taught us something - disproportionately valuable, and each is factual with its finding number.

"MLA is slow" turned out to be model-specific

Expected: a general "MLA architecture tax" (~0.66). Actual: DeepSeek-V2-Lite (same graph family) runs 0.80 of naive, not 0.66 - MLA is NOT a single constant class. F12.

The workhorse has no concurrency knee; gpt-oss does

Expected: a scaling knee for the pure-MoE workhorse. Actual: it scales smoothly to 16 slots (2.66×, still rising); gpt-oss-20b plateaus at ~4. F17.

An MLA model ANTI-scales under load

Expected: MLA would scale like other MoE. Actual: GLM-4.7-Flash gets less total throughput at 16 users than 1 (0.70×), with 20 s first-token. F18.

A smaller model file generated slower

Expected: fewer bytes → faster. Actual: the 7.3%-smaller MXFP4 GLM-Flash generates 12.5% slower - a dequant tax bandwidth arithmetic can't produce. F11.

The best whisper model costs nothing over the smallest

Expected: large-v3-turbo slower than small. Actual: identical speed (~28× RT) - quality is free. §6.1.

A 110B model generates at the same speed as a 14B

Expected: the 110B (GLM-4.5-Air) much slower. Actual: 23.90 vs 24.39 t/s - because MoE active-parameter count, not total, sets generation speed. §2.4 / F10.

Speculative decoding HALVES speed on creative text

Expected: ≥1.5× from a draft model. Actual: 3.73× on predictable text but 0.53× on creative - a workload bet, not a switch. F21.

The small model misattributes values under enumeration

Expected: if a 2.3 GB model retrieves 12K-token facts correctly, tables should be fine. Actual: Qwen3-4B built a plausible table assigning real values to the wrong models - worse than omission. F19.

ROCm's prefill advantage is pure-MoE-only and explodes at depth

Expected: ROCm better at long context generally. Actual: for pure MoE, ROCm prefill hits 8.08× Vulkan at 96K (Vulkan collapses); for MLA, Vulkan wins both phases. The blanket "ROCm for documents" claim is wrong - it's a narrow slice. F22, F23.

Section 12: Cost Estimates

Hardware amortisation

Item Value
Hardware cost (net, Irish, after VAT offset) €2,960
Assumed lifespan 3 years
Annual cost ~€987
Monthly cost ~€82
Electricity (working-day pattern: 8 h @ ~80 W + 16 h @ 4 W ≈ 0.7 kWh/day @ €0.30) ~€6-10/month
Total monthly cost ~€88-92

Token-throughput reference (workhorse Qwen3-30B-A3B)

Cost per million OUTPUT tokens

Pricing checked 2026-07-05, FX 1 USD = 0.874 EUR, sources in docs/briefs/pricing/ (three independent searches reconciled; OpenAI official page settled the flagship). Local from E12.

Scenario € / 1M output Basis
Local, this box (16 users, electricity) €0.036 E12 bench-e12-ws16.json
Local (1 user, electricity) €0.092 E12 bench-e12-ws1.json
Local all-in (+ hardware amort at saturation) ~€0.15 €2,960/3yr ÷ ~16.7M tok/day
OpenRouter - the SAME Qwen3-30B weights ~€0.26 $0.30 × 0.874 (rent vs run)
Anthropic Claude Haiku 4.5 €4.37 $5
Anthropic Claude Sonnet 5 €8.74 $10 (intro; →$15 after ~Aug 31)
OpenAI gpt-5.4 (prev flagship) €13.11 $15
Anthropic Claude Opus 4.8 €21.85 $25
OpenAI gpt-5.5 (current flagship) €26.22 $30

Energy caveat: the local €/token figures use package power - a lower bound on true wall power (§1.8); actual wall cost is likely ~5-15% higher, and the §16 breakeven table inherits the same optimism. Because the cloud-vs-local gap is ~2 orders of magnitude this does not change the conclusion - but do not quote the local absolute cost as exact until a wall-socket meter confirms it (§10).

Two anchors that survive every pricing uncertainty: (1) renting the identical open model is ~3× the local electricity cost and ~1.7× the local all-in cost - running it yourself is cheaper and private; (2) the current frontier flagship (gpt-5.5) is ≈285-730× local per output token (€26.22 ÷ local €0.092 at 1 user = 285×, ÷ €0.036 at 16 users = 728×); the previous-gen gpt-5.4 is ~140-360×, Sonnet 5 ~95-240×, Haiku 4.5 ~50-120× - the ~2-orders-of-magnitude gap holds across every frontier tier.

What does that mean? Running the model on your own box costs a few cents per million words in electricity. A top cloud AI charges tens of euros for the same volume - hundreds of times more - and that meter never stops. The box is a fixed cost no matter how much you use it, so the more you use AI, the more it saves.

Batch APIs are ~50% off; Anthropic prompt-cache hits ~90% off input - these help cloud but do not close two orders of magnitude. NOTE: API pricing changes frequently; re-check before publishing. Caveat: OpenAI's gpt-5-nano ($0.05/$0.40) undercuts local per-token, but it is a nano-class model (Phi-4-mini tier), not a workhorse-quality comparator.

Section 13: Observed Limitations

Factual record of what failed or fell short - data, not editorial.

Models / configs that did not fit or run

Item Detail
GLM-4.5-Air combined -d invocation DeviceLost abort - ~68 GiB weights leave ~3.7 GiB headroom in the 71.65 GiB device-local heap (F6). Split invocations work.
Mixtral TheBloke Dec-2023 GGUF "failed to load model" at commit 067de937 - pre-modern MoE tensor layout. Re-conversion works.
Native image ≥1.4 MP (1792×1008, 1080p, 4K) FLUX compute buffer exceeds RADV single-allocation cap (~4.3 GB). Upscaling is the workaround.
Tier-3 models (Qwen3-235B, DeepSeek-V3) Not tested - not on disk (235B ~133 GB exceeds the 105.5 GiB GTT outright at Q4).

Tasks that produced poor results

Task Model What happened
Spam email triage Qwen3-4B Classified both scam emails as "urgent" (judgment failure; every 20B+ model correct) - F19
Build a table from a document Qwen3-4B Misattributed real values to wrong models - plausible, wrong, invisible without source-checking - F19
Cross-chunk reference at 40K Llama-4-Scout Echoed document instead of answering; arithmetic from approximate retrieval (2 of 6) - E8

Quality vs frontier models (MEASURED 2026-07-06)

A head-to-head was run: the identical deterministic eval-pilot suite sent to the current frontier flagship openai/gpt-5.5 via OpenRouter, temperature 0, same assertions (results/frontier-eval.md; raw results/raw/frontier-eval-*.json; 11 API calls, US$0.0185).

Model Pass rate
gpt-5.5 (frontier) 10/11 - the one miss was a functionally-correct code answer that failed a strict pattern-match
Local 20B+ (gpt-oss-20b, Qwen3-30B-A3B, Coder-30B) 17/17
Local 2.3 GB (Qwen3-4B) 15/17 (2 spam-judgment misses)

Plain reading - this does NOT show local models beating frontier. It shows that on baseline capability tasks (retrieval, extraction, straightforward code and summarization) local 20B+ models and the frontier flagship both sit at the ceiling: the suite has a high floor (even a 2.3 GB model scores 88%). What this suite does not probe - and where frontier models are expected to lead - is deep multi-step reasoning, adversarial robustness, long-horizon coherence, and few-shot generalization. The claim we can stand behind: for routine business tasks (drafting, summarizing, extraction, classification, straightforward code) local 20B+ models on this box are quality-competitive with the frontier flagship. Not claimed: parity on the hardest reasoning - that needs a harder, human-judged suite (Phase B; §10 follow-ups).

What does that mean? On everyday work - drafting, summarising, pulling facts out of documents - the local models matched the top cloud model in our test. On the very hardest reasoning, the big cloud models are still expected to lead; we did not test that and we do not claim it. The fair read: local handles the bulk of real work, and you keep a cloud option for the rare hardest cases (using non-sensitive data only).

Operational constraints (factual)

Section 14: Fleet Scaling Projections

PROJECTIONS from single-box data - no multi-box cluster was tested. Each box is independent (no inter-machine coordination).

Linear scaling (independent boxes, no shared state)

Boxes Hardware cost Interactive users @ instant Aggregate tg (16-slot each)
1 €2,960 1 ~193 t/s
2 €5,920 2 ~387 t/s
4 €11,840 4 ~774 t/s
8 €23,680 8 ~1,548 t/s

Specialisation (4 boxes, mapping measured strengths)

Box Role Model / backend Basis
1 Interactive chat Qwen3-30B-A3B / Vulkan 92.8 t/s, §2.1
2 Meeting transcription whisper large-v3-turbo 28× RT, §6.1
3 Document processing / RAG pure-MoE / ROCm prefill up to 8× Vulkan at depth, §7
4 Code assistant Qwen3-Coder-30B / Vulkan 93.0 t/s, §2.13

4-box fleet: €11,840; monthly amortised (3 yr) ~€329 + electricity. NOTE: cost alone understates it - cloud seats buy frontier quality; local boxes buy transcription + TTS + image + vision + data sovereignty. Different value propositions, not direct substitutes.

Section 15: Alternatives Comparison

Prices checked 2026-07-05 (docs/briefs/pricing/); verify before publishing (hardware pricing is RAM-shortage-volatile).

Hardware (same Ryzen AI Max+ 395 / 128GB chip class)

Device RAM / storage Price Note
GMKtec EVO-X2 (tested) 128GB / 2TB €2,960 (IE net) / ~$3,388 this report
GMKtec EVO-X3 128GB / 2TB ~$3,600 (launch 2026-07-06) same chip, newer chassis
GMKtec EVO-X3 128GB / 4TB ~$3,849
Framework Desktop 128GB $1,999 barebones / ~$2,460-2,605 configured same chip, DIY

The findings apply to the platform (Strix Halo / 128GB unified memory), not the chassis - anyone evaluating this chip class can use these benchmarks (F10 corridor rule generalizes by architecture).

Different-approach alternatives

Approach Cost Throughput (workhorse-class) Privacy Setup
Local box (this report) €2,960 once + ~€8/mo 92.8 t/s solo, 193 agg Full (nothing leaves) High
Cloud API (gpt-5.5) ~€26/1M out, metered frontier Third-party Low
Cloud API (Claude Sonnet 5) ~€8.74/1M out frontier Third-party Low
OpenRouter (same open model) ~€0.26/1M out equivalent weights Third-party Low
Enterprise GPU (A100 80GB) €10-15k + host higher Full (on-prem) Very high

Section 16: Buyer Profile Data

Observable thresholds, not recommendations (derivative documents frame these as guidance).

Usage-pattern thresholds

Pattern This box delivers Fit
Solo, occasional massive overkill poor
Solo, daily interactive 92.8 t/s single-stream (§2.1) strong
Solo + transcription pipeline LLM + whisper both at full speed (§6) strong
2 concurrent interactive 63.4 t/s per stream (§3) - below "instant" marginal
4+ concurrent interactive not at interactive speed on 1 box poor (needs fleet, §14)
Batch overnight (no human waiting) 193 t/s aggregate (§3) strong
Long-document / RAG (pure-MoE) ROCm prefill up to 8× (§7) strong

Confidentiality-constraint profiles (the core market)

Profession Data sensitivity Cloud API viable? Local value
Solicitor privileged no enables AI adoption
Business adviser client strategy risky under GDPR eliminates the risk
Accountant client financials risky compliance assured
HR consultant employee records no under GDPR enables AI adoption
Therapist clinical notes no enables AI adoption
Marketing agency client briefs usually OK cost play, not privacy
General SME low yes cost play only

For confidentiality-constrained buyers the comparison is not "local vs cloud cost" but "local vs no AI at all" - the breakeven question below does not apply to them.

Breakeven (cost-only, for the volume buyer)

Amortisation Monthly hardware Breakeven API spend
2 years ~€123 €123+/month
3 years ~€82 €82+/month
5 years ~€49 €49+/month

Section 17: Running It as a Server, Too - Context for a Business Leader

A fair question: "if the 128 GB of memory is mostly given to the GPU for AI, can the box still run the ordinary services a small business needs - scheduled jobs, a database, a file share, backups?" Short answer: yes, comfortably - with one caveat that is about compute, not memory. This is context, not a benchmark.

The memory is shared on demand, not reserved. The 105.5 GB "GPU memory" figure is a ceiling the GPU is allowed to reach, not a block carved out and locked away. In practice the GPU uses only what the loaded model needs: the everyday workhorse occupies ~20-25 GB (17 GB weights + working cache), leaving roughly 100 GB free for the operating system, databases, web apps, file caches, and background jobs. Even the largest model tested (63 GB) leaves ~60 GB. Underneath it is an ordinary Ubuntu Server - it hosts cron jobs, containers, a small internal web app, or a file share like any other small-business server.

The CPU is barely touched by inference. The processor has 16 cores; AI inference runs on the GPU and uses only a handful of CPU threads. The cores that run your scheduled jobs, database, or backup are mostly idle while the model generates text - they do not compete.

The one real tradeoff is the shared memory bus, and it shows up as timing, not failure. Every part of this box draws from the same memory. We measured what happens when other work runs alongside inference (§1.7, §3): - Light background services (a database, cron jobs, file sync, a download) cost inference essentially nothing - the effect is small latency jitter, not lost throughput (F15). - Heavy simultaneous GPU work (generating images while someone is chatting) does slow the chat - throughput roughly halved under a full media load in testing (§3, E6). That is a scheduling question, not a dealbreaker.

Practical pattern for a small business: run interactive AI plus your light always-on services together all day; schedule the heavy batch jobs (bulk transcription, image generation, overnight document processing) for evenings when no one is chatting. Two AI models can even stay resident at once and serve different jobs without swapping (§3, co-residency).

The limits. It is still a single box - no automatic failover, no redundancy. It is a capable small-business server, not a hardened multi-tenant datacentre node; you would not put a life-or-death always-up service in direct competition with unpredictable heavy AI bursts. But for the realistic case - a professional-services firm running local AI and the handful of background services it already needs - one box does both.

Section 18: FAQ for Business Leaders

Plain answers to the questions a buyer actually asks, each pointing to the measured evidence.

Is my data safe? Yes - everything runs on the box in your office; nothing is sent to any outside service. The one real caveat is physical: if the box is stolen, encrypt the drive so the data is useless to a thief (§13).

Is it as good as ChatGPT? On everyday tasks - drafting, summarising, extracting facts from documents - the local models matched the current top cloud model in our test (§13). On the very hardest reasoning we did not test, and the big cloud models are still expected to lead. The practical split: local for the bulk of real work, a cloud option kept for the rare hardest cases (with non-sensitive data).

Can my whole team use it at the same time? For interactive speed, one box is essentially one-person-at-a-time (§3). It shines as a personal tool or an overnight batch worker (hundreds of jobs done by morning). A team needing simultaneous interactive access needs more boxes (§14).

What does it cost to run? About €2,960 once, plus a few euros a month of electricity (§12). Per million words it is hundreds of times cheaper than a top cloud AI - and the cost is fixed no matter how heavily you use it.

Can it do more than chat? Yes - the same box transcribes meetings, generates speech, creates images, and answers questions about your own documents (§6). A one-hour meeting becomes a searchable transcript in about two minutes.

Do I need to be technical to use it? Day-to-day, no. Initial setup is currently non-trivial and needs someone technical (§13) - though the platform gets simpler each year.

Will it be obsolete soon? The findings apply to the whole hardware class (the chip), not just this box; newer boxes use the same chip (§15). And the predictive method means you can size up any future model before buying it.

Section 19: What It Looks Like in Practice

Short scenarios, each built only from capabilities measured in this report.

A solicitor's morning. She records a client consultation, then drops the audio on the box. A searchable transcript comes back in about two minutes (§6.1), and the local model drafts a plain-language summary (§6.6) - all offline. The privileged material never touches a third-party service.

A financial adviser reviewing a client pack. He feeds a 40-page document to the box and asks pointed questions; the answers come back grounded in the document, with the relevant lines quoted (§6.6). The client's financials stay in the building - the comparison here isn't "local vs cloud cost," it's "local vs not being allowed to use AI at all."

A marketing team on a deadline. The stock-photo budget is gone and they need a blog header and a few social images. The box generates on-brand illustrations in under a minute each (§6.3) - great for headers and backgrounds, though not for anything that needs legible text inside the image.

Overnight, unattended. Two hundred support emails are queued for categorisation and draft replies. The box works through them by morning at batch speed (§3), while no one sits waiting - the per-person speed trade-off - which only bites when several people hit it at the very same moment - is irrelevant when the human is asleep.

Appendix A: Findings Register (F1-F23)

Full text in docs/FINDINGS.md. F1 llama-cli chat-first (harness uses llama-server). F2 download = byte/hash match. F3 quant clusters (→ superseded by F11). F4 MoE-first (8.5× matched). F5 depth slopes = active-set/KV. F6 memory edge ~68 GiB → split invocations. F7 DeltaNet hybrid per-token cost (~0.55). F8 GLM-Flash = MLA/deepseek2 (~0.66). F9 "A3B" is a label not a price (1.57× spread). F10 corridor rule + constants ladder. F11 MXFP4 tg-only tax (~0.81). F12 MLA not a single class (Flash 0.66, DSV2 0.80). F13 instant-death depths measured. F14 depth-dependent rank crossover ~16K. F15 ±1.5% noise; UMA contention = jitter. F16 q8 KV depth lever; FA free. F17 concurrency architecture-dependent. F18 MLA anti-scales. F19 high 12K floor; small-model tax is judgment. F20 eval verdicts depend on serving config. F21 spec decode 0.53×-3.73× by workload. F22 backends split by phase (Vulkan gen, ROCm prefill). F23 that split is architecture-dependent and depth-explosive (pure-MoE ROCm prefill 8× at 96K; MLA inverts).

Appendix B: Raw Data Index

Every table above traces to results/raw/ (llama-bench .md, serve_bench .json) or docs/PHASE-A-LOG.md. Model provenance (repo/revision/size/SHA256) in manifests/MANIFEST.md. Experiment pre-registrations + results in results/experiments.md (E1-E18b). Pricing sources in docs/briefs/pricing/. Non-LLM detail in results/capability-probes.md. Survey detail in results/model-survey.md. Frontier quality A/B in results/frontier-eval.md (raw results/raw/frontier-eval-*.json). Image-quality verdict in results/image-quality-verdict.md. Charts in results/charts/. Full annotated file index in results/INDEX.md.

End of report. © 2026 Alastair McDermott / HumanSpark - CC BY 4.0. Every claim herein is backed by a version-controlled raw file in the sparkbench repository; this document is self-contained for independent verification.