Running a 26B MoE on an 8 GB Jetson by streaming experts from SSD

My Jetson Orin Nano has 8 GB of unified memory. Gemma 4 26B-A4B is a 13.3 GiB download at Q4. This post is about the llama.cpp patch that lets the first thing run the second thing anyway, producing logits that are bit-for-bit identical to the standard code path, and why mixture-of-experts models make that possible when dense models never could.

The idea is borrowed, and worth borrowing

Credit where it belongs: the architecture comes from TurboFieldfare, a Swift and Metal runtime that runs Gemma 4 26B-A4B in about 2 GB of RAM on Apple Silicon. Its trick is to notice that an MoE model only touches a few experts per token. The dense weights (attention, router, shared expert) stay resident, and the routed experts, which are most of the model, live on the SSD. A small per-layer cache of expert slots is filled by parallel reads the moment the router announces which experts the next token needs, and the reads hide behind compute the GPU was doing anyway. The related academic thread is Eliseev and Mazur’s expert-offloading work (arXiv:2312.17238), which established that MoE routing is stable enough between tokens for caching to pay off.

TurboFieldfare proves the idea beautifully, but it is a bespoke runtime: two supported models, Apple platforms only, custom kernels for everything. I wanted the same idea for the other cheap 8 GB machine on my desk, a Jetson Orin Nano, and I wanted it for any MoE model I could quantize. So instead of porting the runtime, I grafted the idea into llama.cpp, which already runs on the Jetson and already has Ampere-tuned CUDA kernels.

Streaming without touching a single GPU kernel

The part I like most about the design is that no CUDA kernel changes were needed. llama.cpp evaluates the routed-expert matmul with ggml_mul_mat_id, which takes the expert weight tensor plus a small tensor of selected expert ids and addresses each expert as base + id * stride on the device. The kernel does not care whether base points at 128 experts or at 32; it just multiplies ids by strides.

So the patch allocates, per MoE layer, a persistent pool tensor holding N expert-sized slots plus a tiny slot-id tensor, and at load time it records where every expert’s bytes sit in the GGUF file. During decode, a scheduler callback fires on the router’s top-k node, reads the chosen expert ids, and consults a per-layer LFU cache: experts already in slots are hits, misses are pread directly from the file into their assigned slots by a small thread pool. The callback writes the slot indices into the slot-id tensor and lets the graph continue. The matmul reads the pool through slot ids and gets exactly the bytes it would have read from the full tensor.

There is one place to be genuinely careful, and it is the reason this patch has a paranoid test. The same expert ids feed several consumers in the graph: the main matmul, the per-expert scale vectors that quantization-aware checkpoints ship, LoRA adapters, and expert biases. Only the main matmul may see remapped slot ids. Everything else must keep the real expert ids, because those tensors stay fully resident and are indexed by actual expert number. Get this wrong and nothing crashes; the model just gets quietly stupider. The test that guards against it greedy-decodes the same prompt with streaming off and on and compares every logit of every step with memcmp. Same kernels, same bytes, different addressing, therefore bitwise identity is the correct bar, not “close enough”.

On the Jetson there is a bonus that TurboFieldfare’s mmap trick foreshadowed: the Orin’s GPU shares memory with the CPU, so the slot pools live in pinned host memory that the disk reads land in directly and the GPU addresses without any copy.

For the IO-hiding part, Gemma 4’s graph has a shared dense FFN that runs for every token alongside the routed experts. The patch reorders the layer so the router’s top-k comes first, dispatches the disk reads, lets the GPU compute the shared FFN while the NVMe works, and joins right before the expert matmul needs the data. One synchronization per layer, exactly TurboFieldfare’s overlap, expressed as a graph reordering plus a two-phase callback.

Numbers from the actual machine

Measured on a Jetson Orin Nano Super (8 GB, MAXN, NVMe) with Gemma 4 26B-A4B QAT Q4, using llama-bench tg128. The baseline is the best you could previously do in llama.cpp: dense weights on the GPU, experts mmap’d on the CPU with --cpu-moe.

Configuration	Decode tok/s
--cpu-moe mmap baseline	1.94
Streaming, 32 slots x 8 IO threads, overlap on	2.95
Same, overlap off	2.23

A few observations that the table compresses:

• The overlap alone is worth 32 percent, which is the TurboFieldfare thesis confirmed on entirely different hardware: expert IO really does hide behind compute you were already doing.
• The NVMe’s random-read ceiling (about 1.27 GB/s at this block size, per fio) puts the IO-bound limit near 5 tok/s, so there is headroom left in the IO path, not a wall.
• Slot count is a real tuning knob with a cliff. 32 slots per layer (a 3 GB pinned pool) is the sweet spot on 8 GB; 64 slots allocates 6 GB and starves the rest of the system.
• Peak RSS stays at 6.7 GB, and the correctness bar held on device: 32 greedy steps on the real 26B, every logit bitwise identical, with 10.5 GiB streamed through the pools during the comparison.

The honest tradeoff: cold prefill drops to about a quarter of the baseline’s speed, because streaming mode deliberately stops pre-faulting the expert file (pre-faulting 13 GiB into 8 GB of RAM is its own disaster). Prefill-through-the-pool is the obvious next piece of work, along with O_DIRECT reads.

The GIF at the top is one request against the production gemma4-26b entry in llama-swap. I sent a warm-up call first to get past cold-load and cold-prefill, then recorded a streamed completion with tegrastats logging GPU load and power draw as each token arrived, laid out as three regions in the same terminal so the numbers and the text are visible at once. It settled at 2.09 tok/s where the bench measured 2.95, which is about the gap I’d expect between a clean llama-bench run and one live request with a different prompt. The GPU trace is the interesting part: utilization swings between 0 and 95 percent because for stretches of every layer the GPU waits while the NVMe pulls in the next expert. That is the overlap claim from earlier, on screen.

What generalizing bought

Because the hook lives in llama.cpp’s graph layer rather than in any model’s code, it works for every GGUF MoE architecture. I verified bitwise identity on three families: Qwen3-MoE style graphs with per-expert scale tensors, Gemma 4’s merged gate-up projection, and Qwen 3.5/3.6’s hybrid DeltaNet layers. The same flag also gets a 22 GiB Qwen 3.6 35B-A3B onto the same 8 GB board.

Is 3 tokens per second fast? No. It is roughly reading speed, fine for anything asynchronous and painful for chat. But the alternative on this hardware is not a slower 26B, it is no 26B at all. The compromise the patch offers is exactly the one I wanted: model size becomes a choice you pay for in speed instead of a hard out-of-memory wall.

The work lives on the moe-stream branch of my llama.cpp fork: a standalone expert streamer, a --moe-stream flag, pool allocation at load, the dual-id graph integration, the Gemma 4 overlap, and the identity test harness, in seven commits against upstream master.

Thanks again to TurboFieldfare for the architecture. Ideas this good deserve to escape their original hardware.

The run behind that GIF is scrubbable. I put a replay of it on the Learning Lab, with GPU load and power draw lined up against each token.