MLX Runner: An Inference Engine You Can Read

MLX Runner is a local LLM inference engine for Apple silicon. It is written from scratch in Swift on Metal, and it ships as a native macOS app, an OpenAI-compatible local server and the mlxr command-line tool.

mlxr pull mlx-community/Qwen3-4B-4bit
mlxr run mlx-community/Qwen3-4B-4bit "Explain RoPE"
mlxr serve mlx-community/Qwen3-4B-4bit --port 1234

It reads the same files mlx-lm reads [3], so any compatible mlx-community repo runs without conversion. It does not link Apple's MLX framework: the name refers to the model format, and the math runs on the engine's own Metal kernels.

This essay makes three claims about it and shows the evidence for each:

1. Dependency free. The Swift standard library and Apple's system frameworks, and nothing else.
2. Highly optimized. Hand-written Metal kernels for every layer, at about 85% of the memory-bandwidth bound for plain decoding on a 7B model.
3. Auditable. Small enough to read, with correctness checked against references, and a release chain you can verify.

---

The stack under a local model

Running a model locally is usually described as a privacy win: the prompt never leaves the machine. That is true, and it moves the trust question instead of closing it. The prompt now passes through whatever stack runs the model.

That stack is often deep. The reference way to run these files, mlx-lm on MLX, is a Python interpreter, a tensor framework with a C++ core, a tokenizer library, a template engine, and the transitive packages each of those pulls in. Every layer is maintained by someone else and updated on someone else's schedule.

Two things follow.

• Few people read it. The code is open, but the path from a weight file to a token is spread across repositories and languages. A security review of "the inference engine" becomes a review of a lockfile.
• Performance is someone else's too. When decode is slow, the cause sits behind a framework boundary: a generic kernel, a graph compiler, a scheduler. You can tune flags. Changing the kernel means working inside someone else's framework.

MLX Runner takes the opposite position, the same one odek takes for agent runtimes. An inference engine is a small, well-defined program: load tensors, run a fixed sequence of matrix products, pick a token. It can be written as one.

Zero dependencies, literally

The package manifest declares no dependencies and the repository has no Package.resolved [1]. There are no SwiftPM packages, no vendored code and no Python. Build scripts are Swift or POSIX shell.

Apple's own mlx-swift is a third-party package, so it is not used either. The engine runs MLX models, not MLX.

The complete import list of the shipped code is: Foundation, Metal, SwiftUI, AppKit, Network, CryptoKit, Security, Observation, and the system's libedit for line editing. Everything a runner normally imports is written in the repository:

Usually a dependency	In MLX Runner
Tensor framework	Metal compute kernels in one file, Kernels.swift (2,762 lines)
Tokenizer library	Tokenizer.swift (593 lines): byte-level BPE and SentencePiece-style BPE with byte fallback, driven by tokenizer.json
Template engine	ChatTemplate.swift: six prompt formats, detected and rendered natively
safetensors reader	A bounds-checked loader that copies weights once into a Metal buffer
HTTP server	OpenAIServer.swift on Network.framework
Hub client	Hub.swift on URLSession, with resume and SHA-256 checks
Update framework	Updater.swift on CryptoKit (Ed25519) and codesign

The size of the whole thing:

Part	Files	Lines
Engine (kernels, model, tokenizer, server, hub, updater)	18	11,980
macOS app (SwiftUI)	12	4,372
CLI	26	3,304
Shipped code	56	19,656
Tests	65	11,767

One person can read the engine in a few days. That is the property the rest of this essay depends on.

Architecture: one command buffer per token

flowchart TD
    F["📦 Model directory
config.json · tokenizer.json · safetensors"] --> V{"1. Validate
headers, offsets, shapes"}
    V -->|"reject"| E["❌ Clear error
nothing reaches the GPU"]
    V -->|"ok"| B["2. One page-aligned
Metal buffer"]

    P["👤 Prompt"] --> T["3. Tokenizer + chat template"]
    T --> PF["4. Prefill
quantized GEMM + flash attention"]
    B --> PF
    PF --> KV["KV cache"]

    KV --> D["5. Decode step
fused matvecs + attention"]
    B --> D
    D --> S["6. Token selected on GPU"]
    S -->|"7. next step already queued"| D
    S --> O["8. Detokenize → app · CLI · server"]

Three design choices carry most of the weight.

Kernels compile from source at launch. The Metal shader source is a string inside Kernels.swift, compiled at startup with makeLibrary(source:). There is no prebuilt metallib. The shader source you read is the source Apple's Metal compiler builds on your machine, with nothing prebuilt in between.

The GPU holds one copy of the weights. Loading copies each tensor once, in file order, into a page-aligned Metal buffer, then releases the file mapping. The copy exists because MLX writes safetensors without the 16-byte alignment the kernels' vector loads need. From the page cache, and including a warm-up pass, it takes about 0.5 s for small models and 3 s for Mistral 7B. Qwen 3.5 is the exception: its scales and biases are stored a second time, interleaved, which adds about a ninth.

The engine runs the weights as shipped. MLX affine quantization stores each group of weights as integer codes plus one scale and one bias: weight = scale × code + bias. The kernels multiply against the packed codes directly, at 2, 3, 4, 5, 6 or 8 bits and group sizes of 32, 64 or 128. No file is converted and no dequantized copy of the weights is ever stored. Only the 4-bit format has a specialized decode kernel so far, and every benchmark below is 4-bit.

Optimized against a known ceiling

"Fast" needs a reference point. For decode there is a hard one.

Generating one token reads nearly every weight once, so plain decoding is bounded by memory bandwidth. The M2 Pro used for every number here reads 197 GB/s from the GPU with a plain read kernel. Mistral 7B at 4 bits is 4.08 GB, which puts the bound at about 48 tokens per second. The only ways past it are fewer bits per weight or checking several tokens per pass.

MLX Runner decodes that model at 41.0 tok/s, about 85% of the bound. mlx-lm reaches 41.2 on the same file: both engines are close to the wall. The remaining work is in making sure nothing else is on the critical path.

Decode, one token per step:

• Fused matvecs. The q, k and v projections are one quantized matrix-vector kernel over three weight matrices, with the bias folded in. The MLP's gate and up projections are another.
• A 4-bit path without shifts. The input vector is pre-scaled by powers of 1/16, so each 4-bit code is extracted with a mask alone. Each GPU lane consumes 32 weights per row per step.
• Fused glue. Per-head RMSNorm, RoPE and the KV-cache append run as a single kernel. The residual add is fused with the next RMSNorm.
• Sampling on the GPU. Repetition penalty, exact top-k, min-p, top-p and one seeded draw run at the end of the same command buffer. The chosen token is written to a buffer that the next step's embedding lookup reads. So the next step is committed before the current one is read back, and the GPU never waits for the CPU.

Prefill, many tokens at once:

• Quantized GEMMs on simdgroup_matrix tiles. Each weight tile is dequantized once and shared by 32 tokens.
• Flash attention on the same matrices, with the softmax rescale done as a diagonal-matrix multiply.

Across turns: prompt tokens shared with the previous request are skipped, so a chat follow-up usually starts almost instantly.

The numbers

Same weight files, same M2 Pro (16 GB), greedy sampling, mlx-lm 0.32.2 on mlx 0.32.4. One run each. Decode is 128 tokens at a short context: mlxr bench for MLX Runner, stream_generate for mlx-lm. Prefill is a raw forward of 512 tokens from an empty cache. This is the README's full table, not a selection [4].

Model (4-bit)	Decode: MLX Runner	mlx-lm	Prefill 512: MLX Runner	mlx-lm
Qwen2.5 0.5B	343 tok/s	296	4,270 tok/s	4,030
Qwen3 0.6B	286	260	3,020	2,340
Qwen3.5 0.8B ¹	271	254	2,780	2,420
Llama 3.2 1B	202	205	1,720	1,720
Gemma 3 1B	189	181	2,070	2,010
Qwen3.5 2B ¹	133	136	1,158	1,030
Gemma 2 2B	94	113	782	791
Gemma 1.1 2B	78	81	763	868
Gemma 4 E2B	73	76	770	639
Phi-3.5 mini	70	66	466	572
Qwen3.5 4B ¹	62.3	65.7
Mistral 7B	41.0	41.2	247	305

¹ Measured differently: medians of one to three alternating rounds against mlx_lm.benchmark, whose prefill figure is not a raw forward and reads higher.

Read it plainly. MLX Runner decodes faster on 5 of the 12 models and slower on 7. The range runs from 16% ahead (Qwen2.5 0.5B) to 17% behind (Gemma 2 2B), and nine of the twelve rows are within 7%. It prefills 512 tokens faster on the Qwen models, Gemma 3 and Gemma 4, is level on Llama 3.2 and Gemma 2, and is slower on Gemma 1.1, Phi-3.5 and Mistral 7B.

Prefill is the weaker side. At 128 tokens, a common chat prompt size, mlx-lm prefills 4–36% faster on every model except Gemma 4. At 30,000 tokens it prefills 1.4–2× faster, because prefill attention grows with the prompt.

A third engine gives the same picture. On Qwen 3.5 4B with a 490-token prompt, the median of three runs is 62.1 tok/s for MLX Runner, 64.4 for Ollama's MLX runner and 64.3 for mlx-lm: about 3.5% behind both. The script that produces this comparison, scripts/bench-compare.swift, is in the repository.

These figures were measured on engine 2.0.0, which 2.1.0 leaves unchanged. So the claim is not "fastest". The claim is that a single-maintainer engine with no framework under it runs within a few percent of a vendor-backed one on its own file format, and that every MLX Runner number above can be reproduced with mlxr bench.

Long contexts

At long contexts, decode reads the whole KV cache of every layer. From 5,128 tokens of context, full-attention layers attend sparsely instead: each new token attends to the first keys, a 512-key recent window, and the 2,048 earlier keys whose 4-key blocks score highest [5].

At 30K tokens this is 1.4–3× faster than the engine's own dense decode. Against mlx-lm, which decodes exactly, it is about 1.45× faster on 7–8B models and 2.4× on Qwen3 0.6B, from single runs. With sparse attention off, long-context decode is 3–11% behind mlx-lm on 7–14B models.

It is also the one place where the engine knowingly changes its output for speed. The next section says what that costs.

Auditable means checkable

A small codebase is necessary for an audit and not sufficient. The reader also needs a way to check that the code is correct. MLX Runner has four checks. Two are automated tests and two are manual gates, and the difference matters.

1. The compute-heavy kernels have CPU references (automated). The quantized matmul kernels are tested for every bit width, group size and scale dtype, across all three code paths (decode matvec, small batch, prefill GEMM). Attention, the sparse kernels and Qwen 3.5's linear-attention kernels have references too. GPU token selection is tested against the CPU sampler, including ties, tiny vocabularies and non-finite logits. The small glue kernels, such as RMSNorm, RoPE and the activations, are covered only end to end, by the model tests.

2. Prefill and decode must agree (automated, local). They are different kernels computing the same function. On real checkpoints, a test checks that batch prefill and token-by-token decode pick the same next token, with logits within 0.1. It covers the Qwen3, Qwen 3.5, Llama, Phi, Mistral and Gemma checkpoints. Qwen2.5 is not in that list yet.

3. Greedy output stays token-identical across changes (manual gate). The contributor rules require that this command keeps producing the same tokens:

mlxr run <model> "Explain RoPE" --temp 0 --repeat-penalty 1 --seed 42 --max 256

An optimization that changes a token is treated as a bug, with the one documented exception below. No golden-output test enforces this yet; it is a rule a contributor follows.

4. Logits match mlx-lm (manual measurement). For the Gemma and Qwen 3.5 checkpoints, teacher-forced logits were compared with a float32 mlx-lm run position by position: the same argmax everywhere, within 0.05. The comparison is reported in the README and is not a test in the repository.

The exception is sparse decode attention, which is on by default and changes output past 5,128 tokens. Perplexity on real text rises by at most about 3% on every model measured, except Gemma 4 E2B (+8% at 30K) and Gemma 1.1 (+3.6% at 7K). Those are mostly single runs. With a budget covering the whole context it is tested to be bit-identical to dense decode, and --sparse off removes it.

The suite is about 650 tests in 11,767 lines. Roughly 410 cover the CLI, 145 the server, updater, Hub client and tool templates, and the rest the kernels, tokenizer and models. CI runs everything that needs no downloaded model, on Apple silicon. The model tests, including check 2, run locally once a model is installed.

Auditable is not audited. The repository records no independent review yet. What it offers is a codebase small enough to make one practical.

The dead ends are in the source. Performance changes need numbers before and after, and a change is kept only if it wins. The losers are recorded where the next reader will look. The comment above the fused decode kernel lists three variants that were built, measured and rejected: for example, a gate/up variant that also folded the activation into the kernel, 4–11% slower on every model. An auditor can see what was tried, as well as what was kept.

Untrusted input, verified output

An inference engine downloads large binary files from the internet and feeds them to a GPU. It often also opens an HTTP port. MLX Runner treats both as attack surface [6].

Model files are untrusted. config.json, tokenizer.json and .safetensors are validated before anything reaches the GPU. Header sizes and tensor offsets are bounds- and overflow-checked, every dimension is range-checked, token ids must fit the embedding table, and every weight's shape must match the config.

This is not optional politeness. The whole engine target, which includes the server, the Hub client and the updater, builds with -Ounchecked. That removes Swift's overflow and bounds traps, so a missed check is undefined behaviour instead of a crash. The checks have to be explicit in the code, and that is where a reviewer should look first.

Chat messages cannot forge turns. Message text is tokenized as plain text, so a message containing <|im_start|> is a string, not a control token. The raw completions endpoint parses special tokens by design.

The server is closed by default. It binds to 127.0.0.1, rejects any Host header that is not a loopback name (DNS rebinding) and any browser Origin outside its allowlist, accepts JSON bodies only, and enforces deadlines, connection caps and size limits. Serving on the LAN requires an API key. Features it cannot honour, such as JSON mode or images, return an OpenAI-style 400 instead of being silently ignored.

No telemetry. Outbound requests go to Hugging Face, for searches and downloads you start, and to GitHub's releases API for the update check. That check is on by default, at launch and every six hours, and a setting turns it off. The security policy asks for any other network access to be reported as a vulnerability.

The release chain is verifiable end to end:

• Releases are built from a version tag on a clean GitHub-hosted runner. The workflow refuses a tag that does not match the version in the source.
• SHA256SUMS is signed with Ed25519. The private key exists only as a release-workflow secret; the public key is compiled into the app.
• Each file carries Sigstore build provenance, checked with gh attestation verify.
• The updater installs a release only if the signature, the SHA-256, a strictly newer version and the app's code signature all verify. The swap is atomic, with rollback, and nothing downloaded runs before every check passes. The app is ad-hoc signed, so the code signature proves the bundle is intact, not who built it. Authenticity rests on the Ed25519 key.

With no package dependencies, there is no lockfile behind this chain. What remains to trust is named and short: Apple's toolchain and OS, GitHub's hosted runner and the two GitHub Actions the release workflow uses, and Hugging Face for the models.

What it does not do

The cost of writing everything yourself is that everything you have not written is missing.

• Platform. Apple silicon and macOS 15 or newer. Nothing else.
• Architectures. Qwen2.5, Qwen3, Qwen 3.5, Llama 3, Mistral, Phi-3.5 and Gemma 1 through 4 run. Mixture-of-experts models and Gemma 3n do not, and multimodal exports run as text only.
• Chat templates are detected, not executed. There is no Jinja engine, so a model with an unusual template may get a near-match format. Tool-calling prompts are checked byte for byte against the official templates.
• Prefill is behind mlx-lm on short prompts, on long prompts and on the larger models, as the numbers above show.
• Dense long-context decode is 3–11% behind mlx-lm on 7–14B models. Sparse attention closes that gap by changing the output.
• Speculative decoding is exact but slower. Both prompt lookup and a draft model verify exactly, yet verification costs more than it saves on this GPU. Qwen3 8B generates at 28 tok/s with a draft, against 38 without. It stays off by default.
• The app is ad-hoc signed, not notarized by Apple. The first launch needs a right-click and Open.

The README lists more, each with its measurement: no tool calling on Gemma, no presence penalty, an untested gated-model path. An engine you are asked to trust should tell you where it loses.

Quick start

V=2.1.0   # the latest version is on the landing page
BASE=https://github.com/BackendStack21/mlx-runner/releases/download/v$V

# 1. Download
curl -LO $BASE/mlxr-$V-macos-arm64.tar.gz
curl -LO $BASE/SHA256SUMS

# 2. Verify before installing
shasum -a 256 -c SHA256SUMS --ignore-missing
gh attestation verify mlxr-$V-macos-arm64.tar.gz --repo BackendStack21/mlx-runner

# 3. Install
tar -xzf mlxr-$V-macos-arm64.tar.gz
sudo mkdir -p /usr/local/bin
sudo install -m 755 mlxr-$V-macos-arm64/mlxr /usr/local/bin/mlxr

# 4. Run
mlxr search qwen3                                   # search MLX models on the Hub
mlxr run mlx-community/Qwen3-4B-4bit                # interactive chat, pulls if missing
mlxr serve mlx-community/Qwen3-4B-4bit --port 1234  # OpenAI-compatible server
mlxr bench mlx-community/Qwen3-4B-4bit              # prefill and decode speed on your Mac

The checksum only proves the download is intact. The origin check in this block is gh attestation verify. The Ed25519 signature on SHA256SUMS is checked by mlxr upgrade and by the app's updater, on every later update.

The CLI follows Ollama's verbs and flags [7]. The server speaks the Responses API, Chat Completions and legacy Completions, with tool calling for Qwen, Llama 3.1+ and Mistral models. The macOS app, with model discovery, downloads and chat, is a .dmg at mlx-runner.21no.de [2].

To build from source you need Xcode or its command-line tools (Swift 6) and nothing else: make app, make cli, make test.

Takeaways

An inference engine is a small program. A transformer forward pass, a tokenizer, a sampler and a server fit in about 12,000 lines, comments included. The frameworks underneath most runners are general-purpose; a runner needs a few dozen kernels.

Owning the kernels is what makes the speed explainable. The decode bound is a bandwidth number. The gap to it is a list of specific kernels. Every optimization, kept or rejected, has a number next to it.

Auditability is a design constraint, not a feature. It comes from three decisions made early and kept: no third-party code, correctness checked against references, and a release chain that proves what was built from what.

MLX Runner does not prove that frameworks are unnecessary. Research needs autograd, and a team shipping to every platform needs a portable core. It shows that for one job, running a quantized model on one family of chips, the default can be something you can read end to end.

---

References

[1] 21no.de. (2026). MLX Runner. GitHub. https://github.com/BackendStack21/mlx-runner

[2] 21no.de. (2026). MLX Runner. https://mlx-runner.21no.de/

[3] Apple ML Explore. mlx-lm. GitHub. https://github.com/ml-explore/mlx-lm

[4] 21no.de. (2026). MLX Runner README: Performance. GitHub. https://github.com/BackendStack21/mlx-runner#performance

[5] 21no.de. (2026). Sparse decode attention. GitHub. https://github.com/BackendStack21/mlx-runner/blob/main/docs/sparse-decode-attention.md

[6] 21no.de. (2026). MLX Runner security policy. GitHub. https://github.com/BackendStack21/mlx-runner/blob/main/SECURITY.md

[7] Ollama. Ollama. GitHub. https://github.com/ollama/ollama

---

Kyberneees, November 2026