Open Brain Surgery on a 4-bit LLM
One night, one 16 GB laptop and one AI agent. A 4-bit Qwen3 0.6B came out scoring 38.9% on MMLU instead of 32.0%, with the same size and the same speed.
The question
In early November, Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab (MIT CSAIL) published Sparks of Weight-Space Programming [1]. It claims that a frontier LLM can read and write neural-network weights directly, "not through an optimizer". In their words, the model "uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly."
One result stuck with me. Qwen3.5-0.8B kept looping on math problems. Their LLM traced the loops to one direction written by two MLP layers, and removed it from 131,072 weights, 0.016% of the model. MATH-500 went from 7.9% to 48.1%.
That raised a narrower, practical question for me. Every small model I run locally is a 4-bit copy of a bf16 original, and quantization costs it intelligence. Could an AI agent operate on those 4-bit weights and buy the lost intelligence back, without changing the file's format, size or speed?
I gave Claude Code one night, a MacBook M2 Pro with 16 GB, and Qwen3 0.6B. A lead agent (Opus) planned the work and sent it out to subagents. This post covers the method, the numbers, and what didn't work.
The patient
The patient was mlx-community/Qwen3-0.6B-4bit [2], a 335 MB file. Its original, Qwen/Qwen3-0.6B [3] in bf16, weighs 1.5 GB and is noticeably smarter. On my dev set, the 4-bit copy scored 29.2% on MMLU against the original's 39.6%. Its average KL divergence from the original's next-token distribution was 0.285 nats.
The format explains the gap. Every row of every weight matrix is cut into groups of 64 weights, and each group stores:
- 64 integer codes from 0 to 15;
- one scale and one bias, so that weight = scale × code + bias.
The converter picks each code by rounding to the nearest step, and each scale and bias from the group's min and max. That is fast and blind: it never asks what the layer's output should be.
The one hard constraint: same format, same speed. I run these models in MLX Runner [4], my own Swift/Metal engine. It reads 2- to 8-bit files, but the agent measured the alternatives early, and each decoded slower than the shipped 4-bit format (single runs):
- 3-bit, 19–22% slower;
- a 6- or 8-bit embedding and head, 7–14% slower.
So the shape of the file was frozen. Only the numbers inside it could change: the codes, the scales and biases, and the normalization weights.
The surgeon's reference. The bf16 original shows exactly what every layer should output. Every change below was judged against it: the original is the teacher, and the 4-bit copy is the student.
Rules before results
An agent that can launch a training run every ten minutes will produce dozens of candidates by morning. With that many, it is easy to find one that looks better by luck. So the metrics and the dev question sets were frozen before the first candidate existed. The acceptance rule and the holdout followed two hours in, when the overnight run was planned.
What counts as intelligence. I used two kinds of measurement.
- Multiple choice: zero-shot log-likelihood scoring on ARC-Easy, ARC-Challenge, WinoGrande, PIQA, HellaSwag and MMLU, 10,653 questions in all.
- Book text: 48 windows of 1,024 tokens from a public-domain novel, teacher-forced. For each window: the KL from the original, the gap in text loss, and how often both models' top guess matches.
Everything is paired. Every candidate sees the same questions and the same windows as the baseline. Differences are reported per item, with a standard error and an exact McNemar p-value. A 1-point gain on 300 questions is noise; the paired test says so.
A holdout, scored once. At the start of the long run, a subagent built a second evaluation set and checked that it was disjoint from the first. It held 2,000 fresh MMLU questions, 2,000 fresh HellaSwag items, the ARC validation splits, and a different book (Moby-Dick, against Pride and Prejudice for dev). Nothing was tuned on it, and every finalist was scored on it once, at the end. The winner was then picked among five finalists on this same holdout, which flatters its numbers slightly.
Acceptance rule, per iteration. A candidate became the new "current best" only if all three held:
- It was closer to the teacher on book KL, by more than two standard errors.
- Its dev MMLU was not worse beyond two standard errors.
- It passed an engine gate. The weights must load in MLX Runner, generate coherent text, agree with mlx-lm's generation, and decode at full speed.
The final model would be picked among all accepted ones on the holdout. That rule was added mid-run, and the lessons section explains why.
The surgery: three cuts
Unlike weight-space programming, the agent did not write weight values by reasoning alone. It reasoned about which numbers to change, toward what target, and on what data. Small optimizers did the cutting, under the rules above. Three cuts mattered.
1. Re-distill the scales (DWQ). mlx-lm [5] ships distilled weight quantization. It keeps the 4-bit codes and trains only each group's scale and bias, plus the normalization weights: 18.7M numbers out of 0.6B. The objective is to make the student's next-token distribution match the teacher's on real text.
- 512 chat samples took 10 minutes, and KL fell from 0.285 to 0.179.
- 1,536 more took 30 minutes, reaching KL 0.159 and dev MMLU 36.3%, from 29.2%.
Most of the night's accuracy gain was already here.
2. Re-choose the codes (learned rounding). Round-to-nearest is the obvious way to pick each code, not the best one. A Sonnet subagent wrote an AutoRound-style quantizer [6] from scratch. For every weight it learns whether to round up or down, and for every group of 64 it learns how much to clip the range.
It works block by block: the 4-bit block's output must match the teacher block's, given the actual 4-bit input. That way errors are compensated instead of passed down the layers.
- It took 14 minutes for 28 blocks and flipped 18–23% of the codes.
- Alone, it matched the KL of 2,048 samples of DWQ: 0.154 against 0.159. Its text loss was worse, though.
- Followed by the same DWQ, it beat round-to-nearest codes by 0.026–0.045 KL.
3. Feed it the right food (calibration data). DWQ's default data is chat transcripts. A Haiku subagent built a mix of three parts:
- 1,500 chat transcripts;
- 1,500 Wikipedia passages;
- 107 chunks of code, math and multilingual text.
It dropped every row sharing a 12-word sequence with any eval question or either book. From the same starting point, the run on the mix lowered KL by 0.0176; the identical run on chat alone, by 0.0041. It also undid a drift in plain-prose loss that chat-only training had caused.
One more knob: the learning rate. In a continuation pass, 1e-6 beat 2e-6, and 5e-6 made the model worse.
The winner's lineage: five steps from the bf16 original.
Every step after the first is trained to reproduce the original's outputs. Only the numbers inside the file change, never its shape.
Results
On the holdout, the operated 4-bit model scored 38.9% on MMLU, up from 32.0%. That closes 81% of the gap to the bf16 original's 40.5%. It decodes at the same speed, within measurement noise.
| Holdout metric | 4-bit, as shipped | 4-bit, after surgery | bf16 original |
|---|---|---|---|
| MMLU (2,000 questions) | 32.0% | 38.9% (+6.9, se 1.0, p < .001) | 40.5% |
| HellaSwag, length-normalized (2,000) | 43.2% | 45.6% (+2.4, se 0.7, p = .001) | 47.6% |
| ARC-Easy (570) | 56.1% | 58.1% (+1.9, not significant) | 59.1% |
| ARC-Challenge (297) | 28.3% | 30.6% (+2.4, not significant) | 33.3% |
| Mean accuracy, 4 tasks (plain accuracy, HellaSwag unnormalized) | 38.0% | 41.0% | 42.6% |
| KL from the original (Moby-Dick) | 0.291 | 0.115 (−61%) | 0 |
| Top-1 agreement with the original | 69.7% | 80.3% | 100% |
| Decode tok/s at 16 / 1,024 tokens (median of 3 alternated runs) | 285.8 / 234.1 | 285.9 / 234.5 | n/a |
Partway through, I asked for "20% better performance". On intelligence, two numbers met it:
- MMLU rose 21% relative.
- KL fell 61%.
Mean accuracy rose 7.7%, and could not have reached 20%. The bf16 original is itself only 12% better than the 4-bit copy on that measure. Surgery can approach the teacher; it cannot pass it.
Speed did not improve, and could not, by design. The file has the same format and runs the same kernels. Every faster-looking format the agent tried was slower in practice.
The winner is published on Hugging Face as 21no-de/Qwen3-0.6B-4bit-LR-DWQ [7]. It is a drop-in replacement for the original 4-bit file, in MLX Runner and in mlx-lm:
# MLX Runner
mlxr download 21no-de/Qwen3-0.6B-4bit-LR-DWQ
mlxr run 21no-de/Qwen3-0.6B-4bit-LR-DWQ -p "What is RoPE?"
# mlx-lm
mlx_lm.generate --model 21no-de/Qwen3-0.6B-4bit-LR-DWQ -p "What is RoPE?"What didn't work, and where the gain really came from
The overnight run did five complete iterations, each with one hypothesis.
| # | Hypothesis | Verdict |
|---|---|---|
| 1 | More DWQ data still pays | Rejected: KL improved slightly, dev MMLU fell 2.7 points |
| 2 | A higher learning rate trains faster | Wrong direction: 5e-6 hurt; 1e-6 won and was accepted |
| 3 | Better codes beat rounded ones under DWQ | Accepted: learned rounding, then DWQ |
| 4 | Calibration data matters | Accepted: the mixed data, the eventual winner |
| 5 | Twice the rounding effort helps | Marginal: no gain from the rounding itself |
Tried and dropped along the way:
- 6- and 8-bit embeddings: a little better on KL, flat or worse on MMLU, 7–14% slower.
- 3-bit: smaller but 19–22% slower.
- A higher-precision input embedding: real but tiny, 0.004 KL for about 165 MB more.
The honest attribution is less flattering than the headline. The evening's first two DWQ runs, 40 minutes of GPU time, already reached 37.1% on holdout MMLU. Against that model, the whole overnight run added:
- a lot of fidelity: KL fell from 0.162 to 0.115;
- little accuracy: +1.8 MMLU points (se 0.9), with the other tasks level.
The chart below shows the pattern on dev. KL fell at every step; MMLU zig-zagged.
Dev set: 48 book windows, 2,000 MMLU questions. From the run's eval results, rounded.
The final pick was not the lowest-KL model. It tied with another on holdout accuracy, and its lower holdout KL broke the tie.
Lessons
Zero-shot MMLU on a small model mostly measures a letter bias. The gold answers are spread evenly across A to D. Yet nearly every model here answered "A" more than half the time, the bf16 original included (1,115 of 2,000). The shipped 4-bit model answered A 1,410 times.
Small shifts in that bias moved MMLU by several points between checkpoints whose KL barely differed. The winner's answer distribution happened to be the closest to the original's. Read MMLU gains on models this size with that in mind. KL is the steadier signal.
Per-step rules can ratchet. "MMLU not worse beyond noise", checked only against the current best, let dev MMLU slide 36.3 → 35.2 → 33.8 over two accepted steps. Each step was inside noise; the total, −2.5 points, was larger than either step's noise.
The fix was to pick the final model across all accepted ones, on the holdout. That rule was written mid-run, after the slide was visible, but before any candidate's holdout score existed. That is the line to hold: never write a rule after seeing the numbers it will judge.
Contamination hides in standard files. mlx-lm's default calibration text for GPTQ contains passages of Moby-Dick, the holdout book. The filter caught it, and no candidate trained on it. A 12-word-overlap check on every calibration source is now a fixed step.
The agents made mistakes too, and the log kept them.
- One learning-rate trial was deleted before it was scored. Its validation loss suggests it was the worst, but it has no score.
- A subagent deleted its own failing verdict from the shared lab notebook. The lead restored it with an audit note, and the notebook became strictly append-only.
- The engine gate's "garbage output" check failed the original model itself, on a greedy phrase loop that mlx-lm reproduces exactly. It was loosened. That change was justified, but nobody formally closed its review.
- The first GPU lock let waiting jobs starve, once for 36 minutes. A first-come-first-served ticket queue replaced it.
How the night ran
I asked for subagents matched to task complexity. Two hours in, I asked for Sonnet and Haiku only. The lead agent (Opus) kept the plan, the rules and the judgment, and handed out work like this:
| Role | Model | Job |
|---|---|---|
| Lead | Opus | Plan, acceptance rules, iteration scripts, verdicts, final pick, install |
| Setup | Haiku | Python environment, teacher download, smoke test |
| Eval | Sonnet | Evaluation harness, frozen question sets, baselines |
| Quant | Fable | Format and speed probes, first DWQ runs |
| Gate | Sonnet | Engine check: MLX Runner against mlx-lm |
| Holdout | Sonnet | Disjoint holdout sets |
| Round | Sonnet | The learned-rounding quantizer |
| Mix | Haiku | Mixed calibration data with contamination filtering |
Four pieces of plumbing kept it honest and unattended:
- A shared lab notebook, append-only. One timestamped line per start, finish and headline number.
- A first-come-first-served GPU lock. One GPU job at a time on a 16 GB machine, so timings stay valid and nothing swaps.
- Shell chains for iterations. Each chain trains, screens, evaluates, gates and applies the acceptance rule, logging every number as it goes. The agent writes the chain; the machine runs it.
- A report every ten minutes. Each report also advanced the plan, so a finished step never sat idle.
The whole thing ran from 17:15 to 01:37, about eight hours and twenty minutes.
Takeaways
There is free intelligence in your quantized files. Distilling the scales of a 4-bit model against its own original recovered most of the lost MMLU, at zero cost in size or speed. Forty minutes on a laptop got most of it. Learned codes and clean data then pushed fidelity much further.
The agent's real contribution was judgment, not arithmetic. It chose what to change and against which target. It wrote a quantizer, built clean data, and refused to fool itself with noise. Weight-space programming points at something more direct: a model that understands a network well enough to edit it by reasoning. The two could meet. Reasoning finds what a quantized model got wrong, such as a looping direction or a letter bias, and distillation repairs everything else.
What I would try next:
- the same procedure on other families, Llama and Gemma;
- larger teachers, through teacher log-probabilities precomputed to disk, so an 8B model can teach a 0.6B one on 16 GB;
- learned rounding trained on end-to-end KL instead of block-by-block error.
References
[1] Takehi, R., Teoh, J., Zhang, A., & Khattab, O. (2026). Sparks of Weight-Space Programming. MIT CSAIL. https://rikiyat.github.io/blog/weight-space-programming/
[2] MLX Community. Qwen3-0.6B-4bit. Hugging Face. https://huggingface.co/mlx-community/Qwen3-0.6B-4bit
[3] Qwen Team. Qwen3-0.6B. Hugging Face. https://huggingface.co/Qwen/Qwen3-0.6B
[4] 21no.de. (2026). MLX Runner. https://mlx-runner.21no.de/
[5] Apple ML Explore. mlx-lm, including mlx_lm.dwq. GitHub. https://github.com/ml-explore/mlx-lm
[6] Cheng, W., et al. (2023). Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs. arXiv:2309.05516. https://arxiv.org/abs/2309.05516
[7] 21no.de. (2026). Qwen3-0.6B-4bit-LR-DWQ. Hugging Face. https://huggingface.co/21no-de/Qwen3-0.6B-4bit-LR-DWQ
Kyberneees, November 2026