Large models that do not fit into your GPU's memory fall off a cliff in speed the moment they spill into system RAM. FreeToken takes aim at that specific wall by focusing on Mixture-of-Experts models. It is released under Apache 2.0, with desktop builds available for Windows and Linux.

An inference engine built around the weights MoE models do not use

FreeToken is an inference engine specialized in running MoE (Mixture-of-Experts) models. It is developed by FlashML, and the source code sits on GitHub under the Apache License 2.0. The project describes itself as an edge-native serving engine that treats GPUs, CPUs, host memory, and the interconnects between them as a single elastic inference platform.

An MoE model activates only a subset of its experts on each pass. Total parameter counts may run from tens to hundreds of billions, yet the weights actually touched during one inference step are far smaller. FreeToken builds on that property: it moves only the experts currently needed into GPU memory and evicts the ones that have not been used recently through a global LRU (Least Recently Used) cache.

Conventional local inference usually splits a model across GPU and CPU statically, layer by layer, at load time. Anything that overflows becomes a permanent drag on every pass. FreeToken instead reshuffles the allocation on each inference step, aiming to raise effective throughput within the same amount of VRAM.

Deciding the split by available bandwidth

The technical core is bandwidth-adaptive CPU-GPU co-execution, which the paper calls the q* policy. During prefill, when the input is being read in, full layers are streamed with double buffering so that weight transfers overlap with computation and the wait is hidden. Weights themselves are handled in a dedicated fast format called FTW.

Memory allocation is not fixed either. The split between expert cache and KV cache can be changed at runtime without restarting the engine or reloading weights. Long contexts and heavy expert churn call for different balances, so that flexibility matters in practice.

Caching designed with agents in mind

The second pillar is semantic-aware caching. FreeToken places what it calls semantic anchor checkpoints, storing recurrent state and KV caches. As a result, agent-specific context edits such as tool calls and thinking blocks do not force a full recomputation of the surrounding context.

Anyone who runs coding agents knows the pattern: every insertion mid-conversation triggers another round of recomputation and another wait. Cutting into that is a different target from the usual benchmark-driven optimization.

Numbers measured on consumer RTX hardware

Shuo Yang of the University of California, Berkeley, the lead author of the paper, published measurements showing Qwen3.6-35B-A3B at 39 tokens per second on a GeForce RTX 4060 Laptop (8GB), DeepSeek-V4-Flash 284B at 22 to 25 tokens per second on a GeForce RTX 5090 (32GB), and GLM-5.2 753B at 15 tokens per second on an RTX PRO 6000 Workstation. The clearest headline is a 35-billion-parameter class model running at conversational speed on an 8GB laptop GPU.

He also cited a comparison against Ollama: 3 to 4 times faster decode and 6 to 30 times faster prefill. The larger gain on the prefill side reflects the overlapping of transfer and computation described above. The paper is published on arXiv as FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution, with Kurt Keutzer, Song Han, Matei Zaharia, and Ion Stoica among the co-authors.

Two installation paths, with APIs compatible with existing agents

There are 2 ways in. One is the desktop app available from the official site, covering Windows and Linux. It sets up the engine for you and exposes model execution, chat, and tuning through a GUI. The other is the CLI, installed with uv or pip as freetoken[accel]. Building from source is also supported.

uv pip install "freetoken[accel]"

Supported models include open-weight MoE families such as DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2, across MXFP4, NVFP4, FP8, and BF16 quantization formats. GPU support natively targets NVIDIA RTX 30, RTX 40, and RTX 50 series cards. The APIs are compatible with both the Anthropic and OpenAI formats, so coding agents including Codex, Claude Code, OpenCode, OpenClaw, and DeepSeek Harness can connect directly. The point of that compatibility is to let your own machine serve as the runtime for those agents.

For reference, the GitHub repository has already gathered more than 4,300 stars and 391 forks not long after publication. Interest is clearly high, though it is worth remembering this is a young project with open issues still on the board.

Summary

FreeToken pushes on the fact that MoE models only ever use a fraction of their weights, aiming to run large models at usable speed on machines with limited VRAM. It combines bandwidth-aware division of work between CPU and GPU, LRU-based expert swapping, runtime reallocation between expert and KV caches, and a caching design that holds up under agent-style context edits. A 35-billion-parameter class model at 39 tokens per second on an 8GB laptop GPU may shift what people assume local LLMs can do.