A week of benches on the Olares One: Gemma 4 MTP, Lucebox regression, vLLM no-Genesis hitting the workspace lock

Hi there.

Quick recap for newcomers: on this blog, I benchmark LLMs locally on the Olares One (a mini-PC with an RTX 5090M 24GB — my personal lab). Three names show up a lot: Genesis is Sandermage’s custom vLLM stack that gets me to 88 t/s, Lucebox is a heavily-tuned llama.cpp fork by sandropuppo, and MTP is the speculative decoding head built into Qwen3.6 and Gemma 4. If you want the full backstory, the post on the hardware choice explains the context.

Anyway — a weird week on the Olares One. Upstream everything moves at the same time: Gemma 4 ships its MTP drafters (May 5), vLLM merges native support (May 6), llama.cpp follows (May 7 via a third-party fork), and meanwhile my Lucebox v1.9.0 regresses 22% for no obvious reason. Quick recap of the three benches that took up my afternoons, plus the cleanup of the 8 Qwen3.6 27B apps on the Olares One Market.

1) Gemma 4 E4B + MTP via vLLM = 178 t/s on consumer mobile Blackwell

This one deserves its own post (link at the end), but in two lines: PR #41745 merged May 6 at 14:39 UTC adds native support for Gemma 4 Multi-Token Prediction drafters (E2B/E4B/26B-A4B/31B). On May 7 at 06:13 UTC the nightly Docker drops. At 06:35 UTC, my Olares One spits:

Run 1 (cold): 800 tok in 6.17 s = 129.73 t/s
Run 2:        800 tok in 4.17 s = 191.73 t/s
Run 3:        800 tok in 3.73 s = 214.38 t/s

AVG = 178.6 t/s, 77.3% acceptance. No Genesis, no fork, no patch — just vllm/vllm-openai:nightly-1acd67a795... + the right --speculative-config. Full details in the short post from this afternoon.

The takeaway here: 24 hours between upstream merge and validated bench on consumer mobile Blackwell. A year ago that loop would have taken 2-3 weeks.

2) Lucebox v1.9.0 — the mystery 88 → 69 t/s regression

This one’s trickier. On May 4 I benched Lucebox v1.4.4 (custom DFlash path, target Qwen3.6-27B Q4_K_M, z-lab BF16 drafter) at 88.5 t/s on the same Olares One. When I tried to rebench on May 7 after the v1.9.0 rebuild (which includes PR #94 matched 3.6 SWA draft + PR #99 consumer Blackwell fix), I consistently get 69 t/s — on the exact same hardware, same chart, same drafters cached on disk.

I isolated hypothesis by hypothesis:

• PR #99 (consumer Blackwell fix)? Disable by switching to v1.6.0 image (without the flag) → still 69 t/s. Not the culprit.
• fa_window=2048 default? Override DFLASH27B_FA_WINDOW=0 (full attention, like v1.4.4) → still 69 t/s.
• TQ3_0 KV vs Q4_0 KV (the Lucebox docs left some doubt)? Bench at --cache-type-k q4_0 --cache-type-v q4_0 → 68.96 t/s. Identical to TQ3_0.
• GPU power throttle? nvidia-smi during inference shows 2,200 MHz / 70-83% util — not throttled.
• HAMi 0m parsing bug? Override CUDA_DEVICE_MEMORY_LIMIT_0=24300m → boot OK, daemon ready, still 69 t/s.

Recap table:

Every image/config hypothesis is ruled out. What’s left is the device environment: kernel update, HAMi runtime, libvgpu state, or bench methodology (May 4 I might have benched through an Open WebUI / public auth URL path with HTTP/2, while now I go through kubectl exec localhost). The kubectl exec route adds k8s-stack microseconds, but not 22% in my opinion.

Provisional conclusion: environmental regression that we can’t isolate without device-level access. Documented in long-term memory and Lucebox is removed from the app store (see §4).

3) vLLM no-Genesis with PR #39931 — the half-fix that breaks on the workspace lock

vLLM-side, JartX lands PR #39931 in the official codebase on May 5 at 00:14 UTC: “[Feature] TurboQuant: support hybrid models and uniform quantization”. Big deal: it fixes the crash TurboQuant threw the moment it hit a Mamba layer on Qwen3.5/3.6/Qwen3-Next. Three of the 28 Genesis patches become unnecessary in one go (P60 ngram for GDN, P65 CUDA graph mode downgrade in spec-decode, P66 invalid-size filter at capture).

I tested on the vllm/vllm-openai:gemma4-0505-cu130 image (main HEAD post-merge) with the exact vllmqwen36turbo27bone config, but zero Genesis vars.

First crash, paradoxal: EagleProposer.__init__ → torch.zeros(...) → CUDA driver error: invalid argument. That’s the HAMi 0m bug I already knew Lucebox-side (Driver API). I thought vLLM was immune (Runtime API). Wrong: the MTP drafter allocation goes through a path HAMi-core intercepts. Workaround CUDA_DEVICE_MEMORY_LIMIT_0=24000m → boot.

Second crash, expected: CUDA graph capture on query_start_loc.tolist() in turboquant_attn.py:583. That’s exactly what P65 monkey-patches. Workaround: --enforce-eager (skip ALL CUDA graphs).

Bench:

Run 1: 800 tok in 11.12 s = 71.97 t/s
Run 2: 800 tok in 10.84 s = 73.81 t/s
Run 3: 800 tok in 11.13 s = 71.87 t/s

AVG = 72.55 t/s — −17.5% vs Genesis baseline (88 t/s). That’s the cost of --enforce-eager. Genesis P65 is smarter: it downgrades _cudagraph_support only when speculative_config is active, so 1-token decode batches keep their CUDA graph.

I also tried Patch 65 alone (3 lines inline-patched on main HEAD). Boot OK, clean CUDA graph capture, explicit log: “setting cudagraph_mode=PIECEWISE”. But on the first inference: crash.

File ".../vllm/v1/attention/backends/turboquant_attn.py", line 862, in _decode_attention
    current_workspace_manager().get_simultaneous(...)
AssertionError: Workspace is locked but allocation requires 0.76 MB,
current size is 0.00 MB. Workspace growth is not allowed after locking.

That’s the profiler-invisible torch.empty bug in the “continuation prefill” phase that Sandermage described in the #40807 thread — his patches P22 (shared dequant buffer + 4-D K/V workspace prealloc) and P38 (memory state machine) are the actual fix. P65 alone isn’t enough. To genuinely drop Genesis you need at minimum P22 + P38 + P65.

I posted a comment on #40807 with these two datapoints (4th confirmed hardware after Ampere 3090, A5000, and Blackwell 5080). Still no reply as of May 8 11h.

Conclusion: we can’t drop Genesis today without porting ~5 non-trivial patches upstream. PR #39931 is necessary but not sufficient. The full fix needs Sandermage or a motivated core-team member.

4) Cleanup: 8 Qwen3.6 27B apps on the Olares One Market → 2

After all this stack iteration, my Olares One Market had 8 Qwen3.6 27B variants:

• vllmqwen36turbo27bone (Genesis vLLM, 88 t/s)
• lucedflashqwen36one (Lucebox DFlash, 69 t/s today / 88 ref)
• dflashqwen36one (buun-llama-cpp, 80 t/s)
• llamacppqwen36mtpone (llama.cpp + PR #22673, 65 t/s @ 128K)
• llamacppqwen36dense27bone (llama.cpp + NVFP4 + ngram, not benched recently)
• vllmqwen36dense27bone (vLLM 0.19.1 vanilla — crashloop)
• Plus two others for 35B-A3B.

Too many. I kept two 27B apps, and to help users choose, the titles position them directly:

Removed: lucedflashqwen36one, dflashqwen36one, vllmqwen36dense27bone, llamacppqwen36dense27bone. The 35B-A3B apps (llamacppqwen36a3bone + qwen36a3bvisionone) stay — different model.

Along the way I also caught 2 bugs on the “Fast” version during re-test:

• v2.5.2 → v2.5.3: drop --prefix-caching-hash-algo xxhash. The Genesis custom image doesn’t have the xxhash package installed → crash after 6 requests: ModuleNotFoundError: xxhash.
• v2.5.4 → v2.5.5: revert GENESIS_ENABLE_P5B_KV=1. P5B (KV cache pad-smaller-to-max strategy, ~34% per-block VRAM saving) crashes CUDA OOM on the first inference on 24GB — combined with P38 prealloc we exceed the HAMi 24,300m cap. Memory said P5B was incompatible with turboquant_4bit_nc; turns out it’s also broken with K8V4 on 24GB.

All of that to eventually boot v2.5.6 at stable 128K context.

5) Bonus: Atomic Chat fork for Gemma 4 MTP via llama.cpp

For folks who want Gemma 4 MTP on the llama.cpp side (not vLLM), AtomicChat published the MTP-compatible GGUFs + a fork AtomicBot-ai/atomic-llama-cpp-turboquant that adds:

• The gemma4_assistant MTP drafter arch (with centroid LM head for E2B/E4B)
• TurboQuant KV-cache in llama.cpp (-ctk turbo3 -ctv turbo3)
• Runtime flags --mtp-head / --spec-type mtp

I built their fork for sm_120 → image aamsellem/llamacpp-atomic-mtp:0.1.0 (2.72 GB). AtomicChat bench on M5Max: 97 → 138 t/s on Gemma 4 26B (+40%).

Validated on Olares One on May 8 afternoon:

Gemma 4 E2B + MTP    : 206.56 t/s, 60.9 % acceptance (3,198 tok in 15.48 s)
Gemma 4 26B-A4B + MTP : 140.03 t/s, 78.1 % acceptance (3,238 tok in 23.12 s)

The 26B-A4B delivers 140 t/s with 78% acceptance on the first run (no warm-up). That beats AtomicChat’s M5Max reference bench (138 t/s) — the 5090M sm_120 mobile has ~75% of a 4090’s bandwidth, but the Gemma 4 MoE (3.8B activated out of 26B) takes good advantage of it. The E2B at 206 t/s is essentially the “single-stream max” ceiling for a 5B model on this hardware.

Charts shipped: gemma4e2bone v1.0.2 (E2B, 128K ctx) and gemma426ba4bone v1.0.9 (26B-A4B, 64K ctx, no vision because mmproj isn’t wired with MTP in this fork).

What works, what breaks — cheat sheet

The only honest conclusion

The local-inference ecosystem in 2026 has a release cadence nobody can follow daily. In 7 days I saw: an official Gemma 4 MTP drafter + 2 PRs to support it (vLLM + llama.cpp), a Genesis-compat PR in vLLM main, two reproducible regressions, and a TurboQuant fork ported to llama.cpp. My job on the Olares One Market is to triage: keep what’s validated, remove what’s broken, write down what helps other Olares One owners avoid my crashes.

Gemma 4 MTP via vLLM-side, it’s too fresh to generalize beyond a 5090M. If you run on a 5070, 5080, desktop 4090 / 3090, your numbers matter to me. And if you’ve solved the Lucebox 88 → 69 t/s regression somehow, I want to know.

See you next time!

Disclosure — All the benchmarks in this post run on my own Olares One. If the content was useful and you’re considering one, ordering through this referral link gets you $400 off ($3,599 instead of $3,999) and pays me $200. I’m mentioning this out of transparency — and yes, incidentally, it helps keep the blog alive (hosting, domain, and the time I spend writing here). Link valid until late June 2026.