Can a Small Model Improve Itself? Testing Continual Harness on a Laptop
I re-implemented Continual Harness for two small local models. Self-editing made both of them worse — and only outside checks brought the small one back.
A language-model agent works inside a harness: its prompt, memory, tools and helper code. Continual Harness (Karten et al., 2026) lets an agent rewrite its own harness while it works, and it improves large frontier models. Its authors suggest that weaker models fall below a “capability floor” and cannot improve themselves this way.
I wanted to test that directly, with models that run on a laptop.
The setup
I re-implemented Continual Harness for local models running through Ollama and tested two of them: qwen3.5:0.8b and gemma4:e2b. Each ran 20 short computer tasks — coding, files, shell and data — and every task is checked automatically, pass or fail.
On top of the original method I added five techniques, each aimed at a known weakness of small models:
- A separate, larger refiner that writes the edits.
- A regression gate that keeps an edit only if recent tasks do not get worse.
- A memory database with retrieval.
- Skills that are kept only if their code actually runs.
- Structured error messages.
What happened
Continual Harness as-is lowered success for both models compared with a fixed harness: qwen went from 32% to 15%, and gemma from 92% to 77%, in 5 of 6 seeds. The drop for gemma stays significant after correcting for multiple comparisons; the drop for qwen is significant only before correction.
No single added technique brought qwen back. All five together — where a larger local model writes the edits and outside checks decide which edits stay — reached 35%: not different from the fixed harness, and above Continual Harness as-is in all three seeds.
This combined system is a form of safeguarded adaptation with a stronger refiner, not self-improvement by the small model.
On held-out tasks, the main test of whether learning transfers, it passed 60% against 27% for the fixed harness. With 15 runs per condition, that is only a suggestive result.
A simple Reflexion-style memory made the smaller model much worse (5%). For the stronger model, the regression gate rejected most edits, which reduced the harm but gave no gain.
Why it goes wrong
The logs show small models writing wrong lessons into their own prompts and keeping edits they cannot judge, while every edit makes the prompt longer. These observations are consistent with the drop, but I did not test them as causes.
The results fit the view that, at this scale, unsupervised self-editing is harmful by default, and that outside checks mainly limit the harm.
How it was measured
Every comparison uses exact paired tests with Holm correction, results are reported per seed, and there is a determinism check. The released data covers 1,256 task runs, every harness edit and gate decision, and the server version and model digest for every model. One command rebuilds every table in the paper:
uv sync
uv run python -m scripts.make_tablesWhat next
Longer tasks with more experience to learn from, more held-out tasks and seeds, stricter gates, a leave-one-out study of the five techniques, and more models between 1B and 5B parameters. The code, tasks and per-run outcomes are open, so every run can be repeated on any laptop.
Adapted from the project write-up. See the code and full details on GitHub ↗