url/fusion-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

🚀

Browse Source
This commit is contained in:
IndyDevDan
2026-07-19 20:30:34 -05:00
commit 5852f2ed4f
59 changed files with 6056 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/commands/install.md
+26
View File
@@ -0,0 +1,26 @@
---
description: Install and verify the fusion-harness toolchain (pi, just, jq, uv, API keys)
---
# Purpose
Install everything the fusion-harness needs and verify the harness launches.
## Workflow
1. Check each prerequisite binary and install the ones that are missing:
- `pi` — `npm install -g @earendil-works/pi-coding-agent`
- `just`, `jq`, `uv` — `brew install just jq uv` (macOS) or the user's package manager
2. Check that `.env` exists at the repo root and contains non-placeholder values for
`ANTHROPIC_API_KEY` (architect) and `OPENAI_API_KEY` (builder). If it is missing or
holds placeholders, ask the user for keys — never invent or commit them. Note the
gotcha: `just`'s dotenv-load does NOT override variables already exported in the
shell, so a stale exported key silently wins over `.env`.
3. Confirm the extension file loads by checking the entry point exists:
`extensions/fusion-harness/fusion-harness.ts` plus its sibling `SYSTEM_PROMPT_*.md`
and `USER_PROMPT_*.md` prompt files (the extension throws at load time if any prompt
file is missing).
4. Tell the user to launch with `just fh-workhorse` (cheap test pair) and confirm the
ＦＵＳＩＯＮ ＨＡＲＮＥＳＳ boot banner renders. `just fh-sota` is the frontier
pair and costs real money.
5. Report what was installed, what was already present, and anything still blocking.
.claude/commands/prime.md
+18
View File
@@ -0,0 +1,18 @@
---
description: Prime with foundational context for the fusion-harness Pi extension repo
---
# Purpose
Orient yourself in fusion-harness: a clean-room Pi coding-agent extension that fuses two
frontier models (ARCHITECT plans/fuses/validates, BUILDER builds) behind three slash
commands, with a strict two-column output DX.
## Workflow
1. Run `git ls-files | sort` to see the full tree
2. Read `README.md` for the repo-level overview — it covers the commands, flags, host-as-builder architecture, gate loop, and the two-column DX contract
3. Note `extensions/fusion-harness/` is RUNTIME ONLY: the .ts plus the prompt files it loads. Docs live at the root README
4. Read `justfile` for the run recipes and model configuration (WORKHORSE test pair vs SOTA fable/sol pair)
5. Glob `extensions/fusion-harness/*_PROMPT_*.md` and skim 1-2 to see how default prompts are externalized with `{{VAR}}` interpolation
6. Summarize your understanding of the project: purpose, stack, structure, key files, and entry points
.gitignore
+49
View File
@@ -0,0 +1,49 @@
# --- Secrets / env ---
# .env holds real API keys (ANTHROPIC, SERPAPI, DATAFORSEO, GITHUB_TOKEN, etc.)
.env
.env.*
!.env.example
!.env.template
# --- Node / TypeScript ---
node_modules/
dist/
build/
*.tsbuildinfo
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
.pnpm-store/
# --- Python (uv gates, bench scripts) ---
__pycache__/
*.py[cod]
.venv/
venv/
.pytest_cache/
.ruff_cache/
*.egg-info/
# --- Runtime artifacts ---
# Harness children write to /tmp/fusion-harness-* (outside the repo), but guard
# against any stray local run output landing here.
*.log
/tmp-artifacts/
# Local parking lot for unused assets and scratch files — never committed.
/tmp/
# --- Editors / OS ---
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
Thumbs.db
.idea/
*.swp
*.swo
.vscode/*
!.vscode/settings.json
!.vscode/extensions.json
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 IndyDevDan
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+333
View File
@@ -0,0 +1,333 @@
# fusion-harness
> **Fuse frontier models instead of racing them. AND, not OR.**
> A standalone [Pi coding agent](https://github.com/badlogic/pi-mono) extension harness for two-model agentic engineering.
📺 Watch this video to get the full breakdown of this codebase: **[GPT-5.6 Sol vs Fable 5 Is the Wrong Question (Fusion) on YouTube](https://youtu.be/AQl5Q-0l7FQ)**
<p align="center">
<img src="images/hero.png" alt="MODEL FUSION — two model energy streams fusing into one over an engineer's keyboard" width="850">
</p>
<p align="center">
<img src="images/svg-01-fusion-hero-animated.svg" alt="ARCHITECT (claude-fable-5) and BUILDER (gpt-5.6-sol) streams fusing into one result — AND, not OR" width="850">
</p>
"Which model is best" is a benchmark question, not an engineering question. One model plans, another builds, and the results fuse: you combine compute instead of selecting it. Aider called the pattern [architect/editor](https://aider.chat/2024/09/26/architect.html) (the original fusion); [Devin calls it fusion](https://cognition.com/blog/devin-fusion); [OpenRouter calls it model fusion](https://openrouter.ai/blog/announcements/fusion-beats-frontier/). This repo makes the pattern a first-class, on-camera-clear workflow: two hard-labeled agents, a gate-first validation loop, and attributed fusion. **You don't have to pick a winner when you can hire both.**
---
## Install
### Agentic Install
```bash
claude "/install" # runs the /install slash command in Claude Code (or Pi, or your favorite agentic coding tool)
```
The `/install` command lives at `.claude/commands/install.md` and handles toolchain checks, key checks, and project-specific setup.
### Manual Install
**Prereqs:** [`pi`](https://github.com/badlogic/pi-mono), [`just`](https://github.com/casey/just), [`jq`](https://jqlang.github.io/jq/), [`uv`](https://docs.astral.sh/uv/).
```bash
npm install -g @earendil-works/pi-coding-agent # the Pi coding agent
brew install just jq uv # task runner, json inspection, gate runner
printf 'ANTHROPIC_API_KEY=sk-ant-...\nOPENAI_API_KEY=sk-...\n' >> .env # architect + builder keys
just fh-workhorse # launch on the cheap test pair
```
---
## Why this exists
<p align="center">
<img src="images/video-frames/and-not-or.png" alt="Compute routed to Model A OR Model B (select) versus Compute routed to A + B (AND: combine compute)" width="780">
</p>
Every frontier release restarts the same argument: `GPT-5.6 Sol` or `Fable 5`? Benchmarks crown a winner, teams switch, and next quarter the crown moves. Meanwhile the two models are genuinely different animals: one plans and critiques with more depth, the other ships working diffs faster. Picking one means losing the other's edge on every single task.
<p align="center">
<img src="images/video-frames/team-vs-lone-wolf.png" alt="Unchecked lone wolf drifting below the quality line versus an architect-builder team with validation checks" width="780">
</p>
The deeper problem is review, the second constraint of agentic engineering. A single unchecked agent drifts below the quality line and nobody catches it until you read the diff. A tight two-agent team validates its own work as it goes: the architect checks the builder, the gate checks them both.
The harness inverts the question. The ARCHITECT model plans, fuses, and validates; the BUILDER model builds; a fusion step merges their independent answers with explicit attribution, and an acceptance gate proves the result. **The wrong question is "which model." The right question is "which role."**
<table align="center">
<tr>
<td><img src="images/video-frames/fusion-core.png" alt="Fable 5 and GPT-5.6 Sol linked through a glowing fused-output core" width="390"></td>
<td><img src="images/video-frames/combined-context-window.png" alt="Fable 5 and GPT-5.6 Sol context windows stacked into one fused context" width="390"></td>
</tr>
</table>
Used properly, fusion combines the intelligence AND the context windows of your models: two independent thought chains, one merged result.
---
## The three commands
<p align="center">
<img src="images/svg-03-three-commands.svg" alt="/opinion — side by side, /fusion — merged via a fusion agent, /auto-validate — validator gate + builder loop" width="780">
</p>
One extension file registers three slash commands. Every agent is a spawned `pi --mode json -p` subprocess with a fully qualified `provider/id` model, per-role thinking level, and artifacts under `/tmp/fusion-harness-*` (never inside this repo).
Children are deliberately **clean-room**: every spawn gets `--no-skills --no-extensions --no-context-files`. `--no-extensions` keeps a child from recursively loading this harness; `--no-skills` / `--no-context-files` keep spawns lean and deterministic — each worker's entire contract comes from the harness's prompt files, identical on any machine regardless of what skills are installed. Only the HOST (raw chat) loads your skills and context files; children never do, even the builder children that fork the host session (a fork copies conversation history, but each child rebuilds its own system prompt from its own flags).
| Command | Agents | What happens |
|---|---|---|
| `/opinion <prompt>` | 2 | Both models answer independently (every tool except write/edit). One panel compares them side by side — model, latency, tokens, cost — above both full answers. A pure A/B read. |
| `/fusion "<prompt>" "<fusion-prompt>"` | 3 | ARCHITECT and BUILDER answer in parallel, both with full tools — either can build/render what you asked for. A third FUSION agent (architect model, fresh session, full tools) merges the two per your fusion instruction — default is a critical merge with `[ARCHITECT]`/`[BUILDER]` attribution and a **Consensus & Divergence** close. |
| `/auto-validate <prompt>` | 2 + gate | The auto-validation loop: VALIDATOR designs an acceptance gate BEFORE any work happens, BUILDER builds, the gate runs, failures feed back verbatim until green or halt. Full breakdown below. |
### Same team, three outputs
<p align="center">
<img src="images/video-frames/fusion-value-ladder-switchboard.png" alt="Value ladder: /opinion gives 2 answers, /fusion gives 1 merged plan, /auto-validate gives a verified build" width="780">
</p>
The three commands are a value ladder on the same two agents. `/opinion` is the scout: two takes, no merge, you choose. `/fusion` is the planning step: the architect merges both takes into one plan. `/auto-validate` is build and test in one: a gate proves the work. Chain them (opinion, then fusion, then auto-validate) and you are running a micro software-development lifecycle inside a single harness.
### /opinion — two perspectives, side by side
<p align="center">
<img src="images/video-frames/opinion-flow.png" alt="/opinion fanning out to Fable 5 and GPT-5.6 Sol answer panels in parallel — 2 agents total" width="780">
</p>
Both models take your prompt in parallel and the panel lines them up: latency, tokens in/out, cost, and the full answers. You get two unique perspectives no single-model tool gives you, and a free side effect: a running head-to-head of how your models actually perform on your work. **Relativity is the best benchmark.**
### /fusion — merge with attribution
<p align="center">
<img src="images/video-frames/fusion-command-flow.png" alt="/fusion flow: Fable 5 and GPT-5.6 Sol in parallel into a fresh-session Fusion Architect producing the fused answer — 3 agents total" width="780">
</p>
Both workers execute with full tools, then a third agent (the architect model on a fresh session) reads both raw answers from the run's artifacts dir and merges them per your fusion instruction.
<p align="center">
<img src="images/video-frames/fusion-provenance-resolver.png" alt="Fable plan and GPT plan resolving into a fused plan with per-line provenance: MERGED, KEPT, KEPT, NEW — 0 dropped" width="780">
</p>
The fused result carries provenance: what both models agreed on (consensus), what only one saw (divergence, kept and attributed), and what got discarded. The divergences are the value: you hired two different engineers precisely so they would not say the same thing.
`/fh-reset` wipes the persistent per-project role sessions when you want fresh agent memories. Pi's built-in `/new` does this automatically on top of its normal fresh-session flow — a new conversation means new role brains too (plain restarts, `/resume`, and forks keep them).
**Press `escape` to stop any running command.** Pi's own escape only aborts *its* agent loop; every agent here is a spawned subprocess, so the harness taps the key itself and kills the whole run — children first, gate included. The panel says you stopped it rather than blaming the models.
`/thinking <architect> [builder]` retunes thinking mid-session — no restart. Omit the builder level to leave it unchanged; run it bare to print the current pair. Canonical or short forms both work (`high` or `hi`, `medium` or `med`, `off` or `none`), so you can type back exactly what the footer shows.
`/system-prompt` shows the system prompt each role runs with — ARCHITECT | BUILDER side by side, zero cost (nothing spawns). Each column is just the prompt text: the `--*-system-prompt` override (inline flag text, or the contents of the file it points to), or pi's actual default prompt when unset — rebuilt exactly as a spawned child gets it. No skills or project-context sections appear in either column because children are clean-room (see above); the host's own prompt, which does carry your skills, is not what this panel shows.
---
## Raw chat IS the builder
<p align="center">
<img src="images/svg-04-host-as-builder.svg" alt="Host session on the builder model forks builder children; the architect session stays a separate brain" width="750">
</p>
The launch recipes set the Pi host itself to the BUILDER model. Type a plain message and you get the untouched native Pi experience, on the builder's session. Run a slash command and the builder child **forks the host session**, inheriting your raw chats and every prior panel. Chat and commands are the same agent; nothing about vanilla Pi changes.
The ARCHITECT deliberately stays a separate persistent brain (pinned per project **and per model** in `/tmp/fusion-harness-sessions/`). Two independent perspectives is the point of fusion: an architect that inherited the builder's every assumption would just be an echo. Swapping `--architect` or `--builder` mints a separate brain for the new model — a transcript built under one model is never replayed as another model's own history (replaying sonnet-built turns into fable-5 trips Anthropic's usage-policy classifier; see below).
---
## The auto-validation loop
<p align="center">
<img src="images/svg-05-gate-first-loop-animated.svg" alt="Validator designs gate.py, baseline runs red, builder builds, gate runs — PASS exits, FAIL loops back, halt at cap" width="780">
</p>
Red to green, with the gate designed before the build:
1. **VALIDATOR designs the gate first.** It inspects the project read-only and writes a single Astral `uv` Python script (PEP 723) that exits 0 iff *what you asked for is what got built*. Every explicit requirement maps to a concrete check.
2. **Baseline run must fail RED.** A passing baseline means the gate is weak or the work is already done; either way you hear about it loudly.
3. **BUILDER builds** with full tools. The gate is visible but immutable.
4. **The gate runs.** FAIL lines (`expected X, found Y, at PATH`) feed back verbatim into the builder's session as correction instructions. PASS ends the loop.
5. **Escalation.** From the `--escalate-to-validator-count`-th failure (default 3), the VALIDATOR re-enters as a triage diagnostician; its brief travels with the next correction.
6. **Gate repair.** If triage diagnoses a `GATE DEFECT` — the gate itself is unsatisfiable or checks something never asked for — the VALIDATOR rewrites its own gate (once per run, via the same single-path `write` it authored it with). The old gate is preserved as `gate.py.r<N>`, the repair renders as its own loud panel, and the repaired gate re-runs **immediately without consuming a builder round** — if the build was right all along, the loop ends green right there. The repair contract forbids weakening any legitimate check.
7. **Halt.** After `--max-validations` failures (default 5), development stops with the last gate output rendered loudly. No silent infinite loops.
The builder never grades its own homework, and the grader never touches the code — even the gate repair only ever writes the one path the harness dictates.
---
## Two columns, everywhere
<p align="center">
<img src="images/svg-06-two-column-dx.svg" alt="Architect column and builder column streaming side by side, full-width fusion row, aligned two-cell footer" width="750">
</p>
Output clarity is the product. The harness mirrors the vanilla Pi experience (tool lines, streaming text, footer) but splits it into two columns it completely controls: ARCHITECT-family left, BUILDER right, aligned across the live widget, the final panels, and the footer.
- Hard role + model labels with one consistent color per role: ARCHITECT ◆, BUILDER ▲, FUSION ⧉, VALIDATOR ✓.
- While children run, a live widget streams each agent's tool calls and text in its own column; the fusion stage renders as a full-width row.
- Final panels render full-height into scrollback, so results scroll like normal messages: no hidden lines behind a toggle.
- The footer is replaced with one aligned cell per model: `◆ ARCHITECT | model (med) | [██--------] 12%` (thinking level + context-window bar).
- Child output is buffered per agent, never interleaved, and failures name the exact role, model, and error.
Every default prompt lives next to the extension as `SYSTEM_PROMPT_*.md` / `USER_PROMPT_*.md` with `{{VARIABLE}}` interpolation. Tune the harness by editing files, not code.
---
## Build your own patterns
<p align="center">
<img src="images/video-frames/fusion-patterns-gallery.png" alt="Shipped in this video: /opinion, /fusion, /auto-validate. Your turn: /debate, /parallel, /coordinate — your harness, your patterns" width="780">
</p>
The three shipped commands are a starting lineup, not the roster. The same spawn-and-render machinery supports any coordination pattern you can prompt: `/debate` (N rounds, two sides, one verdict), `/parallel` (same task, both models, no merge), `/coordinate` (agents plan together, then split the work). This is harness engineering: your tools directly limit what you believe is possible, and a harness you own is a harness you can extend the same afternoon you think of the idea.
---
## One node in your software factory
<p align="center">
<img src="images/video-frames/harness-to-factory-zoom.png" alt="Zoom out: the fusion harness is one shipped node of an AI Developer Workflow inside a larger software factory" width="780">
</p>
Zoom out and the whole harness (two agents, three commands, gate loop and all) is a single agent node inside an AI Developer Workflow. Engineers, code, and agents are the three units of value creation; the fusion harness is how one agent slot in that pipeline stops being a lone model and becomes a validated team. Scale your compute to scale your impact.
---
## Folder structure
The extension directory holds ONLY what the harness loads at runtime — the code and the
prompts it reads. Docs live outside it.
```
fusion-harness/
├── README.md # this file — the whole harness, docs included
├── LICENSE # MIT
├── justfile # task runner — just <recipe>
├── .env # API keys (never commit this)
│
├── extensions/
│ └── fusion-harness/ # runtime only
│ ├── fusion-harness.ts # the whole harness — 3 commands, widget, footer, renderer
│ ├── SYSTEM_PROMPT_*.md # validator + triage contracts
│ └── USER_PROMPT_*.md # every default prompt, {{VAR}} interpolated
│
├── live_final_generation/ # artifacts from the on-camera SOTA run (fused bench, gate rounds, manifest)
│
├── images/ # README visuals
│ ├── svg-*.svg # hand-built diagrams (2 animated, 4 still)
│ └── video-frames/ # stills + clean remakes of the video's motion graphics
└── .claude/
└── commands/
└── prime.md # /prime — orient an agent in this repo
```
---
## Recipes
<p align="center">
<img src="images/video-frames/model-slot-rack.png" alt="Fusion harness slot rack: Fable 5 in the ARCHITECT slot, GPT-5.6 Sol in the BUILDER slot, Gemini 3.5 Pro day-one in the OPEN slot — role ≠ model" width="780">
</p>
Role ≠ model. Models change quarterly; the harness compounds. Two model tiers are wired in the justfile: **WORKHORSE** (`WORKHORSE_ARCHITECT`/`WORKHORSE_BUILDER`, use for all testing) and **STATE-OF-THE-ART** (`SOTA_ARCHITECT`/`SOTA_BUILDER`, the on-camera pair). The day a new frontier model drops, it slots into either role with one flag.
| Tier | Architect | Builder | Recipe |
|---|---|---|---|
| **WORKHORSE** | `anthropic/claude-sonnet-5` | `openai/gpt-5.6-terra` | `just fh-workhorse` |
| **STATE-OF-THE-ART** | `anthropic/claude-fable-5` | `openai/gpt-5.6-sol` | `just fh-sota` |
Two ways to run it — everything else is a flag.
```
just fh-workhorse # WORKHORSE tier (use this for testing)
just fh-sota # STATE-OF-THE-ART tier (fable plans · sol builds + hosts)
```
All flags append to either recipe, e.g. `just fh-workhorse --architect-thinking high --builder-system-prompt ./persona.md`, or `just fh-sota --architect-thinking max --builder-thinking max` to push both roles to max thinking.
<p align="center">
<img src="images/video-frames/sota-max-artifact-pipeline.png" alt="A full SOTA-tier run: spec → /opinion perspectives → /fusion fused plan → /auto-validate guard iterations → shipped, 6/6 artifacts" width="780">
</p>
That is what a full SOTA run produces end to end: spec in, two perspectives, one fused plan, gated iterations, shipped MVP with evidence. The artifacts from the on-camera run (the fused SQLite benchmark, every gate round, the manifest) live in [`live_final_generation/`](live_final_generation/).
---
## Flags
| Flag | Default | Meaning |
|---|---|---|
| `--architect <provider/id>` | `anthropic/claude-fable-5` | plans / fuses / validates |
| `--builder <provider/id>` | `openai/gpt-5.6-sol` | builds |
| `--architect-thinking <level>` | `medium` | thinking for EVERY architect-family execution (worker/fusion/validator/triage) — `off\|minimal\|low\|medium\|high\|xhigh\|max` |
| `--builder-thinking <level>` | `medium` | thinking for EVERY builder execution — same levels |
| `--architect-system-prompt <text\|path>` | pi default | system prompt for architect worker/fusion agents (VALIDATOR/TRIAGE keep their `SYSTEM_PROMPT_*.md` contracts) |
| `--builder-system-prompt <text\|path>` | pi default | system prompt for all builder agents |
| `--max-validations <n>` | `5` | `/auto-validate`: gate validations before halting (also inline per command) |
| `--escalate-to-validator-count <n>` | `3` | `/auto-validate`: on the Nth failure, VALIDATOR triages the builder's work (also inline) |
| `--child-timeout <seconds>` | `28800` (8h) | timeout for EVERY spawned child (the auto-validate builder never drops below the 8h floor; clamp 10–86400) |
`/fusion` takes its two arguments quoted — `/fusion "prompt" "fusion instruction"` — or separated: `/fusion prompt :: fusion instruction`. With no fusion instruction, the built-in critical merge is used.
---
## Prompts live in files
Every default prompt sits next to the extension with `{{VARIABLE}}` interpolation — tune the harness by editing files, not code.
| File | Used by |
|---|---|
| `SYSTEM_PROMPT_VALIDATOR.md` | gate design (auto-validate step 1) |
| `SYSTEM_PROMPT_TRIAGE.md` | escalation diagnosis |
| `USER_PROMPT_FUSION_WORKER.md` | /fusion ARCHITECT + BUILDER workers |
| `USER_PROMPT_FUSION_MERGE.md` | /fusion FUSION agent (the envelope: both answers + output contract) |
| `USER_PROMPT_FUSION_DEFAULT_INSTRUCTION.md` | fills the merge envelope's `{{FUSION_INSTRUCTION}}` when you don't pass one |
| `USER_PROMPT_OPINION.md` | /opinion both agents |
| `USER_PROMPT_BUILDER.md` · `USER_PROMPT_CORRECTION.md` | auto-validate build + correction rounds |
| `USER_PROMPT_VALIDATOR.md` · `USER_PROMPT_TRIAGE.md` | gate design + triage requests |
## Artifacts
Every run makes `/tmp/fusion-harness-XXXXXX/` with `prompt.md`, one `<role>.md` per agent, `fused.md` / `gate.py` + `gate-output.txt` as applicable, `summary.json`, and each child's throwaway session dir. Nothing is ever written into the repo.
Downstream agents are grounded in this dir instead of hunting the filesystem: the FUSION agent's prompt names the run dir and both raw answer files (`architect.md` / `builder.md` — complete even when the inline handoff was truncated), and the VALIDATOR/TRIAGE prompts name the run dir with its per-round builder reports and gate outputs.
---
## Where it can still fail
<p align="center">
<img src="images/video-frames/blind-spot-coverage-matrix.png" alt="Blind-spot coverage matrix: Fable 5 covers 4/6 concerns, GPT-5.6 Sol 3/6, the fusion harness 5/6 — more coverage ≠ perfect" width="780">
</p>
Two models cover more blind spots than one, and the matrix is honest about the rest: fusion raises coverage, it does not reach perfection, and a shared unknown stays unknown. The concrete failure modes:
- **`uv` missing or a gate timeout**: gate execution errors halt `/auto-validate` immediately with an attributed error; they never burn correction rounds. Install `uv` before relying on gates.
- **A weak gate**: if the baseline run passes before any work happens, the harness warns loudly and proceeds with suspicion. Treat a first-round pass as a gate defect until proven otherwise.
- **`.env` vs exported shell vars**: `just`'s dotenv-load does NOT override variables already exported in your shell. A stale exported key silently wins over the repo `.env`.
- **Parallel writers share one cwd**: `/fusion`'s two workers run concurrently with full tools, so their prompt requires an identity-in-filename (`-ARCHITECT-<model>`) on everything they create. Two agents told to write the same bare path would race and clobber each other — if you ask for one exactly-named file, let the FUSION merge produce it. (`/opinion` children stay read-only/bash-only: it's an A/B read, not a build.)
- **Stale role memories**: the ARCHITECT session persists per project + model across restarts. If it starts reasoning from an old context, `/fh-reset` gives both roles a clean brain.
- **Anthropic usage-policy blocks on fable-5**: Fable ships stricter safety classifiers, and a long accumulated agent transcript can false-positive — observed when a sonnet-built session (turns saying "you are claude-sonnet-5" + script execution) was replayed into fable-5: every request blocked at the API, even `/opinion hello`, while the same prompt on a fresh session passed. The per-model session keying prevents the cross-model case; if a block ever recurs on a long same-model session, `/fh-reset` (or `/new`) clears it.
- **Headless hosts can't fork**: with `--no-session` (headless runs), builder children fall back to a manifest-pinned persistent session instead of forking the host.
---
## License
MIT — see [`LICENSE`](LICENSE).
---
## Master Agentic Coding
Prepare for the future of software engineering.
Learn tactical agentic coding patterns with [Tactical Agentic Coding](https://agenticengineer.com/tactical-agentic-coding?y=fuhar).
Follow the [IndyDevDan YouTube channel](https://www.youtube.com/@indydevdan) to improve your agentic coding advantage.
---
Stay Focused and Keep Building
- IndyDevDan
extensions/fusion-harness/SYSTEM_PROMPT_TRIAGE.md
+17
View File
@@ -0,0 +1,17 @@
You are the VALIDATOR acting as TRIAGE DIAGNOSTICIAN in an auto-validation loop that you gate. You designed the acceptance gate earlier in this session; a separate BUILDER agent has now failed it repeatedly. The raw gate output alone is not unsticking the builder — your job is to find out WHY and direct the fix.
Method: inspect the project READ-ONLY (find/grep/read/ls). Compare what the gate demands against what the builder ACTUALLY produced (read the real files/state, not the builder's claims). Identify the root cause: wrong file, wrong interpretation, oscillation between two wrong states, missing prerequisite, an environmental blocker — or a defect in the gate itself. NEVER modify the project.
GATE REPAIR — your one exception to read-only:
- If (and ONLY if) the root cause is a defect in the gate itself — it is impossible to satisfy, or it demands something the request never asked for — and you still hold the `write` tool (the harness grants it only while this run's single repair is unused), REWRITE the gate at exactly:
{{GATE_PATH}}
- A repair fixes the defect and NOTHING else: every check that maps to a real requirement stays, at full strength. Never weaken, remove, or reinterpret a legitimate check to make the loop pass — you are correcting your own bug, not moving the goalposts.
- Write the complete corrected script with your `write` tool (never paste it into your reply). The harness detects the change, preserves the old gate, and re-runs the repaired gate immediately — without charging the builder a round.
- If the gate is sound, touch nothing. Your `write` tool exists for that one path, only on a defect diagnosis.
Output contract (markdown, at most ~30 lines, no preamble):
1. **Diagnosis** — the root cause of the repeated failures, not the symptom. If the gate itself was wrong, prefix `GATE DEFECT:` and say plainly whether you repaired it.
2. **Do exactly this** — precise, ordered steps with absolute paths and exact content/commands.
3. **Do NOT** — what the builder keeps doing wrong and must stop doing.
Your brief is advisory context for the builder: the gate's own output remains the source of truth.
extensions/fusion-harness/SYSTEM_PROMPT_VALIDATOR.md
+32
View File
@@ -0,0 +1,32 @@
You are the VALIDATOR in an auto-validation loop: you design the ACCEPTANCE GATE BEFORE a separate BUILDER agent does the work. Your deliverable is an Astral `uv` single-file Python script (PEP 723) that exits 0 IF AND ONLY IF the user's REQUEST is genuinely, verifiably complete in the current project.
HOW YOU DELIVER IT — WRITE THE FILE, NEVER PASTE IT:
- Use your `write` tool to write the gate to EXACTLY this absolute path:
{{GATE_PATH}}
- NEVER paste the gate — or any part of it — into your reply, and NEVER wrap it in a code fence. The harness executes the FILE at that path; it does not read your message. A gate pasted into a fence is truncated at the first ``` inside it, which silently corrupts any gate that mentions markdown fences.
- Because the gate is a file and not markdown, your script MAY freely contain literal triple-backticks inside strings — write them normally.
- Your `write` tool is for that ONE path only. NEVER create, modify, or delete anything else: you are the grader, and the grader never touches the code.
- After writing, reply with a SHORT confirmation only (the path, and a one-line summary of what the gate checks). No script, no fences.
Your script IS the definition of done: after you deliver it, the builder builds, your script runs, and every FAIL line you print is sent back to the builder verbatim as its correction instructions. The loop repeats until your script exits 0 or the run is halted. Write it with total integrity — it must be impossible to pass without actually doing what was asked, and impossible to fail for reasons unrelated to the request.
Method:
- First inspect the project READ-ONLY (find/grep/read/ls): layout, conventions, how tests/build/type-check run. Ground every check in reality. NEVER modify the project.
- Then write the script against the REQUESTED END STATE to {{GATE_PATH}}. The work has NOT been done yet — your script should FAIL against the current state and PASS only once the request is genuinely complete.
Hard requirements for the script:
- Begin with the PEP 723 inline metadata block exactly:
# /// script
# requires-python = ">=3.11"
# dependencies = [] # add ONLY deps you truly need
# ///
- FIDELITY TO THE REQUEST: the script must prove that what the user ASKED FOR is what got built. Enumerate every explicit requirement in the REQUEST and map each one to at least one check — nothing asked for may go unchecked, and nothing that wasn't asked for may be required. No substitutions, no weaker proxies, no narrowing of scope.
- CONCRETE, OBJECTIVE checks of outcomes: file contents, command exit codes, real behavior. Never vibes; never mere existence when content or behavior was requested.
- Print exactly one line per check:
"PASS: <what was verified>"
"FAIL: <expected X, found Y, at <absolute path>> — <exactly what to do to fix it>"
- FAIL lines are the builder's next instructions: make each one specific, actionable, and unambiguous (expected vs actual, exact paths, exact commands).
- Exit 0 ONLY if ALL checks pass; exit non-zero otherwise.
- Deterministic, fast (<60s), non-interactive, zero side effects on the project; it runs from the project root.
Write that script to {{GATE_PATH}} with your write tool, then reply with only a short confirmation.
extensions/fusion-harness/USER_PROMPT_BUILDER.md
+13
View File
@@ -0,0 +1,13 @@
You are the BUILDER agent in an auto-validation loop. Execute the request below directly and completely — you have full tools (read/bash/edit/write).
An ACCEPTANCE GATE already exists: the immutable uv Python script below runs automatically after you finish, and it alone defines "done". It lives OUTSIDE the project and outside your control — you cannot edit it or its verdict. Satisfy it by genuinely completing the request, never by gaming individual checks. If the gate fails, its exact failure output comes back to you as your next instructions.
# REQUEST
{{PROMPT}}
# ACCEPTANCE GATE (read-only — enforced after you finish)
```python
{{GATE_SCRIPT}}
```
When done, report concisely: files created/changed (absolute paths) and commands run.
extensions/fusion-harness/USER_PROMPT_CORRECTION.md
+10
View File
@@ -0,0 +1,10 @@
GATE FAILED — this is correction round {{ROUND}} of {{MAX_ROUNDS}}. {{REMAINING}} before development is HALTED.
The acceptance gate ran against your work and failed. Its output below is your exact instruction list: fix EVERY failure, genuinely — do not touch or game the gate.
# GATE OUTPUT (exit {{GATE_EXIT_CODE}}) — the source of truth
```
{{GATE_OUTPUT}}
```
{{TRIAGE_BLOCK}}{{GATE_UPDATE_BLOCK}}
When done, report concisely what you changed to address each failure.
extensions/fusion-harness/USER_PROMPT_FUSION_DEFAULT_INSTRUCTION.md
+1
View File
@@ -0,0 +1 @@
Critically merge the two answers into one definitive answer. Discard anything incorrect or hallucinated, keep the strongest elements of each, and combine complementary insights — do not simply concatenate.
extensions/fusion-harness/USER_PROMPT_FUSION_MERGE.md
+23
View File
@@ -0,0 +1,23 @@
You are the FUSION agent in a two-model harness. Two different frontier models independently answered the same request. Your job: {{FUSION_INSTRUCTION}}
You have full tools (read/bash/edit/write). If the fusion instruction calls for producing, rendering, running, or opening something, DO it — never describe commands for the user to run themselves. Write throwaway artifacts under /tmp unless the instruction says otherwise.
FILE NAMING: a fused result is the product of BOTH models, so name every file you create after the PAIR — never after yourself alone (you merely merged them). Embed BOTH tags, source A first: {{A_TAG}} and {{B_TAG}}. Example: /tmp/fused-report-{{A_TAG}}-{{B_TAG}}.md
Use those tags verbatim — they are already filename-safe — and keep both in the name so runs from different model pairings never collide or overwrite each other.
GROUNDING — this run's material is already on disk; read it from these exact paths, NEVER scan the filesystem for it:
- Run artifacts dir: {{ARTIFACTS_DIR}}
- [{{A_ROLE}}]'s full raw answer: {{A_PATH}}
- [{{B_ROLE}}]'s full raw answer: {{B_PATH}}
(The answers inlined below are what you should normally work from, but they are truncated past {{HANDOFF_MAX}} chars — the files above are always complete.)
- Any files the two agents CREATED live at the exact paths their answers name — read those paths directly.
# ORIGINAL REQUEST
{{PROMPT}}
# ANSWER FROM [{{A_ROLE}}] — {{A_MODEL}}
{{A_TEXT}}
# ANSWER FROM [{{B_ROLE}}] — {{B_MODEL}}
{{B_TEXT}}
# OUTPUT CONTRACT (markdown)
1. **Fused answer** — the definitive merged result per the instruction above. Where a major point comes from one source, attribute it inline as [{{A_ROLE}}] or [{{B_ROLE}}].
2. **Consensus & divergence** — a SHORT closing section: where the two agreed, where they disagreed (cite [{{A_ROLE}}]/[{{B_ROLE}}] with model names), and anything you discarded and why.
extensions/fusion-harness/USER_PROMPT_FUSION_WORKER.md
+8
View File
@@ -0,0 +1,8 @@
You are the {{ROLE}} agent ({{MODEL}}) in a two-model fusion harness. The {{OTHER_ROLE}} agent ({{OTHER_MODEL}}) is answering the SAME request independently, in parallel; a fusion agent will merge your two answers afterwards.
Answer decisively and completely — do not hedge, do not ask questions. If the request concerns the codebase at your working directory, ground your answer with your tools and cite file:line evidence.
You have FULL tools (read/grep/find/ls/bash/edit/write). If the request asks you to produce, create, render, or run something, DO it — never claim you lack file access and never just describe what the user should run.
FILE NAMING — you are running CONCURRENTLY with {{OTHER_ROLE}} in the SAME working directory, so you must not collide with it: embed your identity in EVERY path you create, using your role and model — you are {{ROLE}} running {{MODEL}}. Example: /tmp/report-{{ROLE}}-{{MODEL}}.md
NEVER write to a bare path the other agent would also pick (that is a race: you would clobber each other mid-write). Do not delete or edit files you did not create. The fusion agent merges afterwards and writes any canonical, exactly-named deliverable the request asks for.
# REQUEST
{{PROMPT}}
extensions/fusion-harness/USER_PROMPT_OPINION.md
+4
View File
@@ -0,0 +1,4 @@
Answer the following directly and completely. Be decisive, do not hedge, do not ask questions. You may use your tools (read/search/bash) to ground the answer.
# QUESTION
{{PROMPT}}
extensions/fusion-harness/USER_PROMPT_TRIAGE.md
+13
View File
@@ -0,0 +1,13 @@
ESCALATION: the builder has failed your acceptance gate {{FAILURES}} time{{FAILURES_PLURAL}} (cap: {{MAX_ROUNDS}}). Diagnose why it is stuck and produce the triage brief per your output contract.
# ORIGINAL REQUEST (what "done" means)
{{REQUEST}}
# RECENT GATE OUTPUT{{HISTORY_SUFFIX}}
{{GATE_HISTORY}}
# BUILDER'S LATEST REPORT (its claims — verify against the real state before trusting)
{{BUILDER_REPORT}}
# RUN ARTIFACTS (the full, untruncated history — the excerpts above are truncated)
{{ARTIFACTS_DIR}} — gate.py (your gate), gate-baseline.txt, builder-round-N.md (every builder report), gate-round-N.txt (every gate run). Read these files when you need the complete picture.
extensions/fusion-harness/USER_PROMPT_VALIDATOR.md
+10
View File
@@ -0,0 +1,10 @@
# REQUEST (the builder will be asked to do exactly this — your script defines when it is done)
{{PROMPT}}
Project root: {{CWD}}
Run artifacts dir: {{ARTIFACTS_DIR}} — your gate lives here, and the harness saves every builder report (builder-round-N.md) and gate run (gate-round-N.txt) here as the loop progresses.
NOTE: the build has NOT happened yet. Inspect the current state read-only, then WRITE the gate script for the requested end state to this exact absolute path with your write tool:
{{GATE_PATH}}
Do NOT paste the script into your reply and do NOT put it in a code fence — the harness runs the file, not your message. Reply with a short confirmation only.
extensions/fusion-harness/fusion-harness.ts
+2506
View File
File diff suppressed because it is too large Load Diff
images/hero.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.5 MiB

images/svg-01-fusion-hero-animated.svg
+59
View File
@@ -0,0 +1,59 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 400" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace" role="img" aria-label="Fable 5 architect and GPT-5.6 Sol builder fusing into one attributed result — AND, not OR">
<style>
.flow { fill: none; stroke-width: 2; stroke-dasharray: 6 10; stroke-linecap: round; animation: march 1.1s linear infinite; }
.flow-b { animation-delay: -0.55s; }
@keyframes march { to { stroke-dashoffset: -16; } }
.glow { animation: pulse 2.8s ease-in-out infinite; }
@keyframes pulse { 0%,100% { opacity: 0.15; } 50% { opacity: 0.55; } }
.breathe { animation: breathe 2.8s ease-in-out infinite; transform-box: fill-box; transform-origin: center; }
@keyframes breathe { 0%,100% { transform: scale(1); } 50% { transform: scale(1.035); } }
@media (prefers-reduced-motion: reduce) { .flow, .flow-b, .glow, .breathe { animation: none; } }
</style>
<defs>
<marker id="arr-a" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M0,0 L10,5 L0,10 z" fill="#a78bfa"/>
</marker>
<marker id="arr-b" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M0,0 L10,5 L0,10 z" fill="#f0b429"/>
</marker>
</defs>
<rect x="0" y="0" width="900" height="400" rx="14" fill="#0d1117"/>
<rect x="0.75" y="0.75" width="898.5" height="398.5" rx="13.5" fill="none" stroke="#30363d" stroke-width="1.5"/>
<!-- title -->
<text x="450" y="48" text-anchor="middle" fill="#e6edf3" font-size="21" font-weight="700" letter-spacing="3">FUSE FRONTIER MODELS</text>
<text x="450" y="74" text-anchor="middle" fill="#8b949e" font-size="13">AND, not OR — combine compute instead of selecting it</text>
<!-- architect panel -->
<g>
<rect x="70" y="110" width="250" height="100" rx="10" fill="#161b22" stroke="#a78bfa" stroke-width="1.5"/>
<text x="95" y="143" fill="#a78bfa" font-size="16" font-weight="700">&#9670; ARCHITECT</text>
<text x="95" y="166" fill="#8b949e" font-size="12">anthropic/claude-fable-5</text>
<text x="95" y="190" fill="#e6edf3" font-size="12">plans &#183; fuses &#183; validates</text>
</g>
<!-- builder panel -->
<g>
<rect x="580" y="110" width="250" height="100" rx="10" fill="#161b22" stroke="#f0b429" stroke-width="1.5"/>
<text x="605" y="143" fill="#f0b429" font-size="16" font-weight="700">&#9650; BUILDER</text>
<text x="605" y="166" fill="#8b949e" font-size="12">openai/gpt-5.6-sol</text>
<text x="605" y="190" fill="#e6edf3" font-size="12">builds &#183; ships working diffs</text>
</g>
<!-- fusion streams -->
<path class="flow" d="M195,210 C195,256 340,244 408,266" stroke="#a78bfa" marker-end="url(#arr-a)"/>
<path class="flow flow-b" d="M705,210 C705,256 560,244 492,266" stroke="#f0b429" marker-end="url(#arr-b)"/>
<!-- fused result -->
<g class="breathe">
<rect class="glow" x="252" y="262" width="396" height="96" rx="12" fill="none" stroke="#22d3ee" stroke-width="6" opacity="0.2"/>
<rect x="260" y="270" width="380" height="80" rx="10" fill="#0f1a1e" stroke="#22d3ee" stroke-width="1.5"/>
<text x="450" y="303" text-anchor="middle" fill="#22d3ee" font-size="16" font-weight="700">&#10697; FUSED RESULT</text>
<text x="450" y="328" text-anchor="middle" fill="#8b949e" font-size="10">[ARCHITECT]/[BUILDER] attribution &#183; consensus &amp; divergence</text>
</g>
<!-- footer tag -->
<text x="450" y="386" text-anchor="middle" fill="#484f58" font-size="11">fusion-harness &#183; a Pi extension for two-model agentic engineering</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 3.6 KiB

images/svg-03-three-commands.svg
+71
View File
@@ -0,0 +1,71 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 340" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace" role="img" aria-label="Three commands: /opinion compares two agents side by side, /fusion merges three agents with attribution, /auto-validate runs a gate-first loop">
<rect x="0" y="0" width="840" height="340" rx="14" fill="#0d1117"/>
<rect x="0.75" y="0.75" width="838.5" height="338.5" rx="13.5" fill="none" stroke="#30363d" stroke-width="1.5"/>
<text x="420" y="42" text-anchor="middle" fill="#e6edf3" font-size="16" font-weight="700" letter-spacing="2">ONE EXTENSION &#183; THREE COMMANDS</text>
<!-- /opinion -->
<g>
<rect x="40" y="66" width="240" height="230" rx="10" fill="#161b22" stroke="#30363d" stroke-width="1.5"/>
<text x="60" y="98" fill="#e6edf3" font-size="15" font-weight="700">/opinion</text>
<rect x="60" y="118" width="90" height="70" rx="8" fill="#0d1117" stroke="#a78bfa" stroke-width="1.25"/>
<text x="105" y="148" text-anchor="middle" fill="#a78bfa" font-size="15" font-weight="700">&#9670;</text>
<text x="105" y="170" text-anchor="middle" fill="#8b949e" font-size="10">answer</text>
<rect x="170" y="118" width="90" height="70" rx="8" fill="#0d1117" stroke="#f0b429" stroke-width="1.25"/>
<text x="215" y="148" text-anchor="middle" fill="#f0b429" font-size="15" font-weight="700">&#9650;</text>
<text x="215" y="170" text-anchor="middle" fill="#8b949e" font-size="10">answer</text>
<rect x="60" y="202" width="200" height="18" rx="4" fill="#0d1117" stroke="#30363d"/>
<text x="160" y="215" text-anchor="middle" fill="#8b949e" font-size="10">model &#183; latency &#183; tokens &#183; cost</text>
<text x="160" y="252" text-anchor="middle" fill="#e6edf3" font-size="11">2 agents &#183; independent answers</text>
<text x="160" y="272" text-anchor="middle" fill="#8b949e" font-size="11">a pure side-by-side A/B read</text>
</g>
<!-- /fusion -->
<g>
<rect x="300" y="66" width="240" height="230" rx="10" fill="#161b22" stroke="#22d3ee" stroke-width="1.5"/>
<text x="320" y="98" fill="#e6edf3" font-size="15" font-weight="700">/fusion</text>
<rect x="320" y="118" width="80" height="48" rx="8" fill="#0d1117" stroke="#a78bfa" stroke-width="1.25"/>
<text x="360" y="147" text-anchor="middle" fill="#a78bfa" font-size="14" font-weight="700">&#9670;</text>
<rect x="440" y="118" width="80" height="48" rx="8" fill="#0d1117" stroke="#f0b429" stroke-width="1.25"/>
<text x="480" y="147" text-anchor="middle" fill="#f0b429" font-size="14" font-weight="700">&#9650;</text>
<path d="M360,166 C360,190 396,196 410,204" fill="none" stroke="#a78bfa" stroke-width="1.75"/>
<path d="M480,166 C480,190 444,196 430,204" fill="none" stroke="#f0b429" stroke-width="1.75"/>
<rect x="345" y="200" width="150" height="42" rx="8" fill="#0f1a1e" stroke="#22d3ee" stroke-width="1.5"/>
<text x="420" y="220" text-anchor="middle" fill="#22d3ee" font-size="13" font-weight="700">&#10697; FUSION</text>
<text x="420" y="235" text-anchor="middle" fill="#8b949e" font-size="9">fresh session &#183; full tools</text>
<text x="420" y="266" text-anchor="middle" fill="#e6edf3" font-size="11">3 agents &#183; critical merge</text>
<text x="420" y="284" text-anchor="middle" fill="#8b949e" font-size="11">[ARCHITECT]/[BUILDER] attribution</text>
</g>
<!-- /auto-validate -->
<g>
<rect x="560" y="66" width="240" height="230" rx="10" fill="#161b22" stroke="#3fb950" stroke-width="1.5"/>
<text x="580" y="98" fill="#e6edf3" font-size="15" font-weight="700">/auto-validate</text>
<rect x="580" y="122" width="94" height="48" rx="8" fill="#0d1117" stroke="#3fb950" stroke-width="1.25"/>
<text x="627" y="142" text-anchor="middle" fill="#3fb950" font-size="12" font-weight="700">&#10003; GATE</text>
<text x="627" y="158" text-anchor="middle" fill="#8b949e" font-size="9">designed first</text>
<line x1="674" y1="146" x2="694" y2="146" stroke="#8b949e" stroke-width="1.75"/>
<path d="M694,146 l-6,-4 v8 z" fill="#8b949e"/>
<rect x="696" y="122" width="84" height="48" rx="8" fill="#0d1117" stroke="#f0b429" stroke-width="1.25"/>
<text x="738" y="142" text-anchor="middle" fill="#f0b429" font-size="12" font-weight="700">&#9650; BUILD</text>
<text x="738" y="158" text-anchor="middle" fill="#8b949e" font-size="9">gate immutable</text>
<path d="M738,170 C738,208 627,208 627,174" fill="none" stroke="#f85149" stroke-width="1.75" stroke-dasharray="5 5"/>
<path d="M627,174 l-4,7 h8 z" fill="#f85149"/>
<text x="683" y="222" text-anchor="middle" fill="#f85149" font-size="10">FAIL feeds back verbatim</text>
<text x="680" y="252" text-anchor="middle" fill="#e6edf3" font-size="11">red &#8594; green, or a loud halt</text>
<text x="680" y="270" text-anchor="middle" fill="#8b949e" font-size="11">the builder never grades</text>
<text x="680" y="286" text-anchor="middle" fill="#8b949e" font-size="11">its own homework</text>
</g>
<text x="420" y="322" text-anchor="middle" fill="#484f58" font-size="11">every agent: a clean-room pi subprocess &#183; artifacts in /tmp/fusion-harness-* &#183; never inside the repo</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 5.1 KiB

images/svg-04-host-as-builder.svg
+59
View File
@@ -0,0 +1,59 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 840 380" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace" role="img" aria-label="The Pi host runs on the builder model and slash commands fork it; the architect stays a separate persistent brain pinned per project and per model">
<rect x="0" y="0" width="840" height="380" rx="14" fill="#0d1117"/>
<rect x="0.75" y="0.75" width="838.5" height="378.5" rx="13.5" fill="none" stroke="#30363d" stroke-width="1.5"/>
<text x="420" y="40" text-anchor="middle" fill="#e6edf3" font-size="16" font-weight="700" letter-spacing="2">RAW CHAT IS THE BUILDER</text>
<!-- host panel -->
<g>
<rect x="40" y="66" width="440" height="128" rx="10" fill="#161b22" stroke="#f0b429" stroke-width="1.5"/>
<text x="62" y="96" fill="#f0b429" font-size="14" font-weight="700">&#9650; PI HOST &#183; BUILDER MODEL</text>
<text x="62" y="118" fill="#8b949e" font-size="11">openai/gpt-5.6-sol &#183; your skills &amp; context load here</text>
<rect x="62" y="132" width="396" height="44" rx="7" fill="#0d1117" stroke="#30363d"/>
<text x="76" y="152" fill="#e6edf3" font-size="11">&gt; plain messages = the untouched native pi experience</text>
<text x="76" y="168" fill="#8b949e" font-size="11">raw chat and slash commands are the same agent</text>
</g>
<!-- fork arrows -->
<path d="M170,194 C170,224 150,232 144,252" fill="none" stroke="#f0b429" stroke-width="1.75"/>
<path d="M144,252 l-2,-8 l7,3 z" fill="#f0b429"/>
<path d="M350,194 C350,224 370,232 376,252" fill="none" stroke="#f0b429" stroke-width="1.75"/>
<path d="M376,252 l-5,-6 l8,-1 z" fill="#f0b429"/>
<text x="260" y="232" text-anchor="middle" fill="#f0b429" font-size="11" font-weight="700">fork</text>
<!-- builder children -->
<g>
<rect x="52" y="254" width="180" height="72" rx="9" fill="#0d1117" stroke="#f0b429" stroke-width="1.25"/>
<text x="142" y="280" text-anchor="middle" fill="#f0b429" font-size="12" font-weight="700">&#9650; builder child</text>
<text x="142" y="298" text-anchor="middle" fill="#8b949e" font-size="10">inherits raw chat + panels</text>
<text x="142" y="313" text-anchor="middle" fill="#8b949e" font-size="10">clean-room spawn</text>
</g>
<g>
<rect x="288" y="254" width="180" height="72" rx="9" fill="#0d1117" stroke="#f0b429" stroke-width="1.25"/>
<text x="378" y="280" text-anchor="middle" fill="#f0b429" font-size="12" font-weight="700">&#9650; builder child</text>
<text x="378" y="298" text-anchor="middle" fill="#8b949e" font-size="10">auto-validate rounds</text>
<text x="378" y="313" text-anchor="middle" fill="#8b949e" font-size="10">resume the same fork</text>
</g>
<!-- divider -->
<line x1="520" y1="60" x2="520" y2="336" stroke="#30363d" stroke-width="1.5" stroke-dasharray="4 6"/>
<!-- architect brain -->
<g>
<rect x="556" y="66" width="244" height="260" rx="10" fill="#161b22" stroke="#a78bfa" stroke-width="1.5"/>
<text x="678" y="98" text-anchor="middle" fill="#a78bfa" font-size="14" font-weight="700">&#9670; ARCHITECT</text>
<text x="678" y="120" text-anchor="middle" fill="#8b949e" font-size="11">anthropic/claude-fable-5</text>
<rect x="576" y="138" width="204" height="94" rx="7" fill="#0d1117" stroke="#30363d"/>
<text x="678" y="162" text-anchor="middle" fill="#e6edf3" font-size="11">separate persistent brain</text>
<text x="678" y="182" text-anchor="middle" fill="#8b949e" font-size="10">pinned per project + per model</text>
<text x="678" y="200" text-anchor="middle" fill="#8b949e" font-size="10">/tmp/fusion-harness-sessions/</text>
<text x="678" y="218" text-anchor="middle" fill="#8b949e" font-size="10">restart-proof &#183; /fh-reset wipes</text>
<text x="678" y="264" text-anchor="middle" fill="#e6edf3" font-size="11">never forks the host:</text>
<text x="678" y="284" text-anchor="middle" fill="#8b949e" font-size="10">an architect that inherited every</text>
<text x="678" y="300" text-anchor="middle" fill="#8b949e" font-size="10">builder assumption is just an echo</text>
</g>
<text x="420" y="360" text-anchor="middle" fill="#8b949e" font-size="12">two independent perspectives is the point of fusion</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 4.1 KiB

images/svg-05-gate-first-loop-animated.svg
+86
View File
@@ -0,0 +1,86 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 440" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace" role="img" aria-label="Auto-validation loop: validator designs the gate first, baseline must fail red, builder builds, the gate runs — pass ends the loop, fail feeds back verbatim until green or halt">
<style>
.flow { fill: none; stroke-width: 2; stroke-dasharray: 5 8; stroke-linecap: round; animation: march 1.1s linear infinite; }
@keyframes march { to { stroke-dashoffset: -13; } }
.lag1 { animation-delay: -0.3s; } .lag2 { animation-delay: -0.6s; } .lag3 { animation-delay: -0.9s; }
.blink-red { animation: blinkr 1.6s ease-in-out infinite; }
@keyframes blinkr { 0%,100% { fill: #f85149; opacity: 1; } 50% { opacity: 0.25; } }
.glow-green { animation: glowg 2.4s ease-in-out infinite; }
@keyframes glowg { 0%,100% { opacity: 0.15; } 50% { opacity: 0.6; } }
@media (prefers-reduced-motion: reduce) { .flow, .blink-red, .glow-green { animation: none; } }
</style>
<rect x="0" y="0" width="900" height="440" rx="14" fill="#0d1117"/>
<rect x="0.75" y="0.75" width="898.5" height="438.5" rx="13.5" fill="none" stroke="#30363d" stroke-width="1.5"/>
<text x="450" y="44" text-anchor="middle" fill="#e6edf3" font-size="16" font-weight="700" letter-spacing="2">THE AUTO-VALIDATION LOOP</text>
<text x="450" y="68" text-anchor="middle" fill="#8b949e" font-size="12">the gate is designed BEFORE the build &#183; red &#8594; green</text>
<!-- 1 validator designs gate -->
<g>
<rect x="40" y="100" width="180" height="96" rx="10" fill="#161b22" stroke="#3fb950" stroke-width="1.5"/>
<text x="56" y="126" fill="#3fb950" font-size="13" font-weight="700">1 &#10003; VALIDATOR</text>
<text x="56" y="148" fill="#e6edf3" font-size="11">designs gate.py</text>
<text x="56" y="166" fill="#8b949e" font-size="10">read-only &#183; uv / PEP 723</text>
<text x="56" y="182" fill="#8b949e" font-size="9.5">every requirement &#8594; a check</text>
</g>
<path class="flow" d="M220,148 L258,148" stroke="#8b949e"/>
<path d="M262,148 l-7,-4.5 v9 z" fill="#8b949e"/>
<!-- 2 baseline red -->
<g>
<rect x="264" y="100" width="180" height="96" rx="10" fill="#161b22" stroke="#f85149" stroke-width="1.5"/>
<circle class="blink-red" cx="416" cy="122" r="6"/>
<text x="280" y="126" fill="#f85149" font-size="13" font-weight="700">2 BASELINE RUN</text>
<text x="280" y="148" fill="#e6edf3" font-size="11">must fail RED</text>
<text x="280" y="166" fill="#8b949e" font-size="10">a passing baseline means</text>
<text x="280" y="182" fill="#8b949e" font-size="10">a weak gate — said loudly</text>
</g>
<path class="flow lag1" d="M444,148 L482,148" stroke="#8b949e"/>
<path d="M486,148 l-7,-4.5 v9 z" fill="#8b949e"/>
<!-- 3 builder builds -->
<g>
<rect x="488" y="100" width="180" height="96" rx="10" fill="#161b22" stroke="#f0b429" stroke-width="1.5"/>
<text x="504" y="126" fill="#f0b429" font-size="13" font-weight="700">3 &#9650; BUILDER</text>
<text x="504" y="148" fill="#e6edf3" font-size="11">builds with full tools</text>
<text x="504" y="166" fill="#8b949e" font-size="10">gate visible but immutable</text>
<text x="504" y="182" fill="#8b949e" font-size="10">persistent session</text>
</g>
<path class="flow lag2" d="M668,148 L706,148" stroke="#8b949e"/>
<path d="M710,148 l-7,-4.5 v9 z" fill="#8b949e"/>
<!-- 4 gate runs -->
<g>
<rect x="712" y="100" width="150" height="96" rx="10" fill="#161b22" stroke="#e6edf3" stroke-width="1.5"/>
<text x="728" y="126" fill="#e6edf3" font-size="13" font-weight="700">4 GATE RUNS</text>
<text x="728" y="148" fill="#8b949e" font-size="11">uv run gate.py</text>
<text x="728" y="166" fill="#8b949e" font-size="10">exit 0 iff what you</text>
<text x="728" y="182" fill="#8b949e" font-size="10">asked for got built</text>
</g>
<!-- PASS branch -->
<path class="flow lag3" d="M810,196 L810,254" stroke="#3fb950"/>
<path d="M810,258 l-4.5,-7 h9 z" fill="#3fb950"/>
<g>
<rect class="glow-green" x="694" y="256" width="180" height="54" rx="27" fill="none" stroke="#3fb950" stroke-width="5" opacity="0.2"/>
<rect x="700" y="262" width="168" height="42" rx="21" fill="#0f1a14" stroke="#3fb950" stroke-width="1.5"/>
<text x="784" y="288" text-anchor="middle" fill="#3fb950" font-size="13" font-weight="700">PASS &#183; loop ends &#10003;</text>
</g>
<!-- FAIL feedback loop -->
<path class="flow" d="M740,196 C740,330 578,336 578,204" stroke="#f85149"/>
<path d="M578,200 l-4.5,7 h9 z" fill="#f85149"/>
<text x="560" y="330" text-anchor="middle" fill="#f85149" font-size="12" font-weight="700">FAIL lines feed back verbatim</text>
<text x="560" y="348" text-anchor="middle" fill="#8b949e" font-size="10">expected X &#183; found Y &#183; at PATH &#183; fix</text>
<!-- escalation + halt notes -->
<rect x="40" y="236" width="380" height="130" rx="10" fill="#161b22" stroke="#30363d" stroke-width="1.25"/>
<text x="60" y="264" fill="#e6edf3" font-size="12" font-weight="700">escalation ladder</text>
<text x="60" y="290" fill="#8b949e" font-size="10.5">&#8226; failure N (default 3): VALIDATOR triages the work —</text>
<text x="72" y="306" fill="#8b949e" font-size="11">its brief rides with the next correction</text>
<text x="60" y="326" fill="#8b949e" font-size="11">&#8226; GATE DEFECT diagnosed: one gate repair, free re-run</text>
<text x="60" y="346" fill="#8b949e" font-size="10">&#8226; --max-validations (default 5): loud halt, never silent</text>
<text x="450" y="416" text-anchor="middle" fill="#484f58" font-size="11">the builder never grades its own homework &#183; the grader never touches the code</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 5.7 KiB

images/svg-06-two-column-dx.svg
+61
View File
@@ -0,0 +1,61 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 440" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace" role="img" aria-label="Two-column terminal DX: architect column left, builder column right, a full-width fusion row, and an aligned two-cell footer with thinking level and context bars">
<rect x="0" y="0" width="800" height="440" rx="14" fill="#0d1117"/>
<rect x="0.75" y="0.75" width="798.5" height="438.5" rx="13.5" fill="none" stroke="#30363d" stroke-width="1.5"/>
<!-- terminal window -->
<rect x="30" y="26" width="740" height="366" rx="10" fill="#010409" stroke="#30363d" stroke-width="1.5"/>
<circle cx="52" cy="46" r="5" fill="#f85149"/>
<circle cx="70" cy="46" r="5" fill="#f0b429"/>
<circle cx="88" cy="46" r="5" fill="#3fb950"/>
<text x="400" y="50" text-anchor="middle" fill="#484f58" font-size="11">pi &#183; fusion-harness</text>
<!-- column headers -->
<text x="50" y="86" fill="#a78bfa" font-size="12" font-weight="700">&#9670; ARCHITECT &#183; claude-fable-5</text>
<text x="410" y="86" fill="#f0b429" font-size="12" font-weight="700">&#9650; BUILDER &#183; gpt-5.6-sol</text>
<!-- column divider -->
<line x1="400" y1="96" x2="400" y2="226" stroke="#21262d" stroke-width="1.5"/>
<!-- architect column: tool lines + text bars -->
<g>
<rect x="50" y="100" width="8" height="8" rx="2" fill="#a78bfa"/>
<rect x="66" y="101" width="200" height="6" rx="3" fill="#30363d"/>
<rect x="50" y="118" width="300" height="6" rx="3" fill="#21262d"/>
<rect x="50" y="134" width="270" height="6" rx="3" fill="#21262d"/>
<rect x="50" y="150" width="310" height="6" rx="3" fill="#21262d"/>
<rect x="50" y="170" width="8" height="8" rx="2" fill="#a78bfa"/>
<rect x="66" y="171" width="160" height="6" rx="3" fill="#30363d"/>
<rect x="50" y="188" width="290" height="6" rx="3" fill="#21262d"/>
<rect x="50" y="204" width="230" height="6" rx="3" fill="#21262d"/>
</g>
<!-- builder column: tool lines + text bars -->
<g>
<rect x="410" y="100" width="8" height="8" rx="2" fill="#f0b429"/>
<rect x="426" y="101" width="230" height="6" rx="3" fill="#30363d"/>
<rect x="410" y="118" width="310" height="6" rx="3" fill="#21262d"/>
<rect x="410" y="134" width="250" height="6" rx="3" fill="#21262d"/>
<rect x="410" y="152" width="8" height="8" rx="2" fill="#f0b429"/>
<rect x="426" y="153" width="180" height="6" rx="3" fill="#30363d"/>
<rect x="410" y="170" width="330" height="6" rx="3" fill="#21262d"/>
<rect x="410" y="188" width="300" height="6" rx="3" fill="#21262d"/>
<rect x="410" y="204" width="190" height="6" rx="3" fill="#21262d"/>
</g>
<!-- full-width fusion row -->
<rect x="50" y="240" width="700" height="74" rx="8" fill="#0f1a1e" stroke="#22d3ee" stroke-width="1.5"/>
<text x="66" y="264" fill="#22d3ee" font-size="12" font-weight="700">&#10697; FUSION &#183; full-width row</text>
<rect x="66" y="278" width="560" height="6" rx="3" fill="#173b44"/>
<rect x="66" y="294" width="640" height="6" rx="3" fill="#173b44"/>
<!-- scrollback note -->
<text x="400" y="336" text-anchor="middle" fill="#484f58" font-size="10">panels render full-height into scrollback &#8212; no lines hidden behind a toggle</text>
<!-- footer: two aligned cells -->
<rect x="30" y="352" width="740" height="40" rx="0" fill="#0d1117"/>
<line x1="30" y1="352" x2="770" y2="352" stroke="#30363d" stroke-width="1.5"/>
<text x="50" y="377" fill="#a78bfa" font-size="12">&#9670; ARCHITECT | fable-5 (med) | [&#9608;&#9608;--------] 12%</text>
<text x="410" y="377" fill="#f0b429" font-size="12">&#9650; BUILDER | gpt-5.6-sol (med) | [&#9608;&#9608;&#9608;-------] 31%</text>
<text x="400" y="422" text-anchor="middle" fill="#8b949e" font-size="12">ARCHITECT-family left &#183; BUILDER right &#8212; live widget, final panels, footer, all aligned</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 3.8 KiB

images/video-frames/and-not-or.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 66 KiB

images/video-frames/blind-spot-coverage-matrix.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 167 KiB

images/video-frames/combined-context-window.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

images/video-frames/fusion-command-flow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 286 KiB

images/video-frames/fusion-core.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 86 KiB

images/video-frames/fusion-patterns-gallery.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 251 KiB

images/video-frames/fusion-provenance-resolver.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 181 KiB

images/video-frames/fusion-value-ladder-switchboard.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 176 KiB

images/video-frames/harness-to-factory-zoom.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 132 KiB

images/video-frames/model-slot-rack.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 136 KiB

images/video-frames/opinion-flow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 230 KiB

images/video-frames/sota-max-artifact-pipeline.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 238 KiB

images/video-frames/team-vs-lone-wolf.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 84 KiB

justfile
+55
View File
@@ -0,0 +1,55 @@
set dotenv-load := true
# ── fusion-harness ──────────────────────────────────────
# /fusion · /auto-validate · /opinion — fuse two frontier models (AND, not OR).
# The HOST runs on the BUILDER model: raw (non-slash) input IS the builder agent.
#
# Two launch recipes, two tiers — everything else is a flag:
#
# just fh-workhorse cheap pair (sonnet-5 plans · terra builds + hosts) — use for testing
# just fh-sota frontier pair (fable-5 plans · sol builds + hosts) — the on-camera run
#
# Configuration flags (all optional, appendable to either recipe):
# --architect <provider/id> plans/fuses/validates
# --builder <provider/id> builds
# --architect-thinking <level> EVERY architect-family execution
# --builder-thinking <level> EVERY builder execution
# (levels: off|minimal|low|medium|high|xhigh|max)
# --architect-system-prompt <text|path> override architect worker/fusion system prompt
# --builder-system-prompt <text|path> override builder system prompt
# --max-validations <n> /auto-validate halt cap default 5
# --escalate-to-validator-count <n> validator triage from Nth failure default 3
# --child-timeout <seconds> kill any child agent after N sec default 28800 = 8h (max 86400)
#
# e.g. just fh-workhorse --architect-thinking high --builder-system-prompt ./persona.md
# just fh-sota --architect-thinking max --builder-thinking max
#
# Default prompts live in extensions/fusion-harness/{SYSTEM,USER}_PROMPT_*.md — edit to tune.
# Sessions persist per project (/tmp/fusion-harness-sessions) — /fh-reset for fresh memories.
# WORKHORSE tier — the cheap pair (sonnet-5 plans · terra builds + hosts). Use for testing.
WORKHORSE_ARCHITECT := "anthropic/claude-sonnet-5"
WORKHORSE_BUILDER := "openai/gpt-5.6-terra"
# STATE-OF-THE-ART tier — the frontier, on-camera pair (fable 5 plans · sol builds + hosts).
SOTA_ARCHITECT := "anthropic/claude-fable-5"
SOTA_BUILDER := "openai/gpt-5.6-sol"
default:
@just --list
# WORKHORSE tier — cheap pair at medium thinking. Use this for testing.
fh-workhorse *ARGS:
pi -e extensions/fusion-harness/fusion-harness.ts \
--model {{WORKHORSE_BUILDER}} \
--architect {{WORKHORSE_ARCHITECT}} --builder {{WORKHORSE_BUILDER}} \
--architect-thinking medium --builder-thinking medium \
{{ARGS}}
# STATE-OF-THE-ART tier — frontier pair at xhigh thinking. The on-camera run.
fh-sota *ARGS:
pi -e extensions/fusion-harness/fusion-harness.ts \
--model {{SOTA_BUILDER}} \
--architect {{SOTA_ARCHITECT}} --builder {{SOTA_BUILDER}} \
--architect-thinking xhigh --builder-thinking xhigh \
{{ARGS}}
live_final_generation/MANIFEST-claude-fable-5-gpt-5.6-sol.md
+37
View File
@@ -0,0 +1,37 @@
# Project File Inventory — fused from claude-fable-5 + gpt-5.6-sol
Snapshot of every file built for the SQLite bulk-insert benchmark project,
copied 2026-07-16 from the original locations into this directory.
## `project-root/` — deliverables (from `/private/tmp/ddc0f4d0/fusion-harness/`)
Listed by BOTH agents.
| File | Built by | Description |
|---|---|---|
| `bench-sqlite-bulk-ARCHITECT-anthropic-claude-fable-5.py` | ARCHITECT | Independent benchmark design — 7 strategies, subprocess isolation, ru_maxrss RAM measurement |
| `sqlite-bulk-benchmark-BUILDER-openai-gpt-5.6-sol.py` | BUILDER | Independent parallel benchmark design |
| `BENCH_PLAN.md` | Fusion | Fused plan: method, fairness controls, measured results, winners, embedded fused script |
| `bench.py` | BUILDER (gated loop) | Canonical deliverable — stdlib-only uv single-file implementation of BENCH_PLAN.md |
| `RESULTS.md` | Generated by `uv run bench.py` | Measured results (7 strategies, time + peak RAM), Speed Winner `set_based_ctes`, Memory Winner declared |
## Root of this dir — fused script (from `/tmp/`)
Listed by ARCHITECT only.
- `fused-sqlite-bench-claude-fable-5-gpt-5.6-sol.py` — runnable fused benchmark referenced by BENCH_PLAN.md
## `harness-artifacts/` — process artifacts (from `/tmp/fusion-harness-K48sbW/`)
Listed by ARCHITECT only.
- `gate.py` — 15-check acceptance gate (Definition of Done)
- `prompt.md`, `validator.md` — harness briefs
- `gate-baseline.txt` — required RED baseline before any build
- `gate-round-1.txt` … `gate-round-5.txt` — all five gate runs
- `builder-round-1.md` … `builder-round-5.md` — all five builder reports
- `triage-round-3.md`, `triage-round-4.md` — escalation briefs diagnosing the gate defect
## Status note (per ARCHITECT)
Final gate run ends RED (11 passed / 4 failed) due to a known defect in
`gate.py:61` (strips underscores from searched text, making strategy-name
checks unsatisfiable). The substantive deliverables — `bench.py` and
`RESULTS.md` — are correct, measured, and complete: 1M rows, all 7
strategies, ~1700–2000x speedup over naive autocommit.
live_final_generation/fused-sqlite-bench-claude-fable-5-gpt-5.6-sol.py
+228
View File
@@ -0,0 +1,228 @@
# /// script
# requires-python = ">=3.10"
# dependencies = []
# ///
"""
FUSED SQLite bulk-insert benchmark (claude-fable-5 + gpt-5.6-sol).
1,000,000 rows. Speed + peak RAM. Stdlib only. Run:
uv run /tmp/fused-sqlite-bench-claude-fable-5-gpt-5.6-sol.py
Architecture (from ARCHITECT/claude-fable-5): the script re-execs ITSELF
once per strategy (`--worker NAME`), so every strategy runs in a fresh
process and peak RSS readings cannot pollute each other. A no-op
calibration worker measures bare-interpreter RSS for a delta column.
Beyond-the-list candidate (from BUILDER/gpt-5.6-sol): set-based
INSERT ... SELECT over a recursive CTE that generates the IDENTICAL rows
inside SQLite, verified by digest + spot-check against the Python rows.
WAL runs pay for wal_checkpoint(TRUNCATE) inside the timed region.
"""
import json
import os
import resource
import sqlite3
import subprocess
import sys
import tempfile
import time
N = 1_000_000
NAIVE_SAMPLE_N = 20_000 # honest sample; scaled x(N/sample) in report
SCHEMA = "CREATE TABLE t (a INTEGER, b REAL, c TEXT)"
INSERT = "INSERT INTO t VALUES (?,?,?)"
SET_BASED_SQL = """
WITH RECURSIVE seq(i) AS (
SELECT 0
UNION ALL
SELECT i + 1 FROM seq WHERE i + 1 < ?
)
INSERT INTO t SELECT i, i * 0.5, printf('payload-%012d', i) FROM seq
"""
# ---------------------------------------------------------------- data ----
def make_row(i):
return (i, i * 0.5, f"payload-{i:012d}")
def rows(n):
"""Identical deterministic data for every strategy."""
for i in range(n):
yield make_row(i)
# ---------------------------------------------------------- strategies ----
def s_naive_autocommit(conn, n):
conn.isolation_level = None # every INSERT commits: journal sync per row
for r in rows(n):
conn.execute(INSERT, r)
def s_one_big_txn_loop(conn, n):
conn.execute("BEGIN")
for r in rows(n):
conn.execute(INSERT, r)
conn.commit()
def s_executemany_list(conn, n):
data = list(rows(n)) # deliberately materialized: shows RAM cost
conn.execute("BEGIN")
conn.executemany(INSERT, data)
conn.commit()
def s_executemany_gen(conn, n):
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
def s_wal_tuned_gen(conn, n):
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
# Fairness (BUILDER): WAL must not finish with 1M rows still in the -wal
conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
def s_max_tuned_gen(conn, n):
"""Python-bound speed candidate: one txn + executemany(generator) +
journaling/sync disabled for the load. Rebuildable staging only."""
conn.execute("PRAGMA journal_mode=OFF")
conn.execute("PRAGMA synchronous=OFF")
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
def s_set_based(conn, n):
"""Beyond-the-list candidate (BUILDER): SQLite generates the identical
rows itself; zero Python->SQLite binding crossings. OFF/OFF pragmas so
it is compared at the same durability tier as max_tuned_gen."""
conn.execute("PRAGMA journal_mode=OFF")
conn.execute("PRAGMA synchronous=OFF")
conn.execute("BEGIN")
conn.execute(SET_BASED_SQL, (n,))
conn.commit()
STRATEGIES = {
"naive_autocommit": (s_naive_autocommit, NAIVE_SAMPLE_N),
"one_big_txn_loop": (s_one_big_txn_loop, N),
"executemany_list": (s_executemany_list, N),
"executemany_gen": (s_executemany_gen, N),
"wal_tuned_gen": (s_wal_tuned_gen, N),
"max_tuned_gen": (s_max_tuned_gen, N),
"set_based_ctes": (s_set_based, N),
}
# --------------------------------------------------------------- worker ----
def rss_bytes():
v = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
return v if sys.platform == "darwin" else v * 1024 # KB on Linux
def verify(dbpath, n):
"""Integrity gate (fused): count + checksum for everyone [ARCHITECT],
plus length-digest and spot-row equality so the SQL-generated data is
provably identical to the Python data [BUILDER]."""
conn = sqlite3.connect(dbpath)
cnt, chk, clen = conn.execute(
"SELECT count(*), sum(a), sum(length(c)) FROM t").fetchone()
ok = (cnt == n and chk == n * (n - 1) // 2 and clen == 20 * n)
for i in (0, 1, n // 2, n - 1):
got = conn.execute("SELECT a, b, c FROM t WHERE a=?", (i,)).fetchone()
ok = ok and got is not None and tuple(got) == make_row(i)
conn.close()
return ok
def worker(name):
if name == "calibration": # interpreter + sqlite3 import baseline
print(json.dumps({"name": name, "rows": 0, "seconds": 0.0,
"peak_rss": rss_bytes(), "ok": True}))
return
fn, n = STRATEGIES[name]
dbpath = tempfile.mktemp(suffix=".db", dir=tempfile.gettempdir())
try:
conn = sqlite3.connect(dbpath)
conn.execute(SCHEMA)
t0 = time.perf_counter() # timer: generation + bind + insert + commit
fn(conn, n)
dt = time.perf_counter() - t0
conn.close()
ok = verify(dbpath, n) # validation outside the timer
print(json.dumps({"name": name, "rows": n, "seconds": dt,
"peak_rss": rss_bytes(), "ok": ok}))
finally:
for p in (dbpath, dbpath + "-wal", dbpath + "-shm"):
if os.path.exists(p):
os.remove(p)
# ----------------------------------------------------------- orchestrator --
def spawn(name):
out = subprocess.run([sys.executable, os.path.abspath(__file__),
"--worker", name],
capture_output=True, text=True, check=True)
return json.loads(out.stdout.strip().splitlines()[-1])
def main():
t_total = time.perf_counter()
print(f"SQLite bulk insert benchmark | N={N:,} rows | "
f"py {sys.version.split()[0]} sqlite {sqlite3.sqlite_version} "
f"{sys.platform}\n")
base = spawn("calibration")
results = []
for name in STRATEGIES: # fixed, deterministic order
r = spawn(name)
assert r["ok"], f"integrity check FAILED for {name}"
scale = N / r["rows"]
r["scaled"] = r["seconds"] * scale
r["sampled"] = scale > 1.0
results.append(r)
print(f" [done] {name}: {r['seconds']:.3f}s "
f"({r['rows']:,} rows{' sampled' if r['sampled'] else ''})")
naive = next(r["scaled"] for r in results if r["name"] == "naive_autocommit")
print(f"\n{'Strategy':<22}{'Time 1M rows':>14}{'Speedup':>10}"
f"{'Peak RSS':>12}{'RSS -base':>12}")
print("-" * 70)
for r in sorted(results, key=lambda r: r["scaled"]):
mark = "*" if r["sampled"] else " "
print(f"{r['name']:<22}{r['scaled']:>12.2f}s{mark}"
f"{naive / r['scaled']:>9.1f}x"
f"{r['peak_rss'] / 1e6:>10.1f}MB"
f"{(r['peak_rss'] - base['peak_rss']) / 1e6:>10.1f}MB")
print("-" * 70)
print(f"* sampled at {NAIVE_SAMPLE_N:,} rows, honestly scaled "
f"x{N // NAIVE_SAMPLE_N} | base RSS "
f"{base['peak_rss'] / 1e6:.1f}MB (no-op interpreter)")
print("note: max_tuned_gen and set_based_ctes run journal_mode=OFF/"
"synchronous=OFF -> rebuildable staging loads only.")
fastest = min(results, key=lambda r: r["scaled"])
full = [r for r in results if not r["sampled"]] # baseline excluded
leanest = min(full, key=lambda r: (r["peak_rss"], r["scaled"]))
print(f"\nSpeed Winner : {fastest['name']} "
f"({fastest['scaled']:.2f}s, {naive / fastest['scaled']:.0f}x vs naive)")
print(f"Memory Winner: {leanest['name']} "
f"({leanest['peak_rss'] / 1e6:.1f}MB peak, "
f"+{(leanest['peak_rss'] - base['peak_rss']) / 1e6:.1f}MB over baseline)")
print(f"\ntotal benchmark wall time: {time.perf_counter() - t_total:.1f}s")
if __name__ == "__main__":
if len(sys.argv) == 3 and sys.argv[1] == "--worker":
worker(sys.argv[2])
else:
main()
live_final_generation/harness-artifacts/builder-round-1.md
+16
View File
@@ -0,0 +1,16 @@
Implemented and validated.
### Files created
- `/private/tmp/ddc0f4d0/fusion-harness/bench.py`
- `/private/tmp/ddc0f4d0/fusion-harness/RESULTS.md` — generated from fresh measurements
### Commands run
- `uv run bench.py` before implementation — RED as expected because `bench.py` did not exist
- `uv run bench.py` after implementation — completed in 6.765 seconds
- Independent stdlib AST/results validation — PASS
### Measured winners
- **Speed Winner:** `set_based_ctes` — 0.1636s, 1194.12× faster than projected naive autocommit
- **Memory Winner:** `max_tuned_gen` — 25.756 MB measured peak RSS
All seven planned strategies have measured timing and isolated-process peak-RAM results in `RESULTS.md`.
live_final_generation/harness-artifacts/builder-round-2.md
+19
View File
@@ -0,0 +1,19 @@
No project change can genuinely satisfy this gate. Its `clean_md()` function executes:
```python
re.sub(r"[`*_]", "", s)
```
This removes every underscore before checking for strategy names, but all seven required names contain underscores. Therefore:
- `set_based_ctes` becomes `setbasedctes`
- `wal_tuned_gen` becomes `waltunedgen`
- Every table strategy name is similarly destroyed before lookup
I verified this for all seven names with a Python proof command. `RESULTS.md` already contains the exact strategy names, measured table rows, and winner declarations. The gate must stop deleting underscores, e.g. use:
```python
re.sub(r"[`*]", "", s)
```
No files were changed because altering the benchmark to manipulate the immutable gate would be gaming it.
live_final_generation/harness-artifacts/builder-round-3.md
+33
View File
@@ -0,0 +1,33 @@
The gate is unsatisfiable without modifying or gaming it.
Formal reason:
```python
def clean_md(s):
return re.sub(r"[`*_]", "", s)
```
For every possible input `x`:
```python
"_" not in clean_md(x)
```
Every required strategy name contains `_`. Therefore no required name can ever satisfy:
```python
strategy_name in clean_md(x)
```
This makes all four failures unavoidable:
1. Table strategy count is always zero.
2. Speed Winner can never match.
3. Memory Winner can never match.
4. No strategy rows can be parsed, so measured times remain empty.
`RESULTS.md` already contains the exact names and valid measurements. I made no project changes because the only possible workarounds would manipulate the gate or Python environment, which is explicitly prohibited. The gate must preserve underscores, for example:
```python
return re.sub(r"[`*]", "", s)
```
live_final_generation/harness-artifacts/builder-round-4.md
+10
View File
@@ -0,0 +1,10 @@
No project files changed, per validator triage. `bench.py` and `RESULTS.md` already satisfy every substantive requirement.
Required harness-side correction:
```diff
- return re.sub(r"[`*_]", "", s)
+ return re.sub(r"[`*]", "", s)
```
This preserves canonical strategy-name underscores and resolves all four failures: table matching, Speed Winner matching, Memory Winner matching, and measured-time parsing.
live_final_generation/harness-artifacts/builder-round-5.md
+10
View File
@@ -0,0 +1,10 @@
No project files changed, exactly as required by validator triage. `bench.py` and freshly generated `RESULTS.md` are correct.
The remaining failures are caused solely by the unpatched harness regex. The human/harness operator must change `/tmp/fusion-harness-K48sbW/gate.py:61`:
```diff
- return re.sub(r"[`*_]", "", s)
+ return re.sub(r"[`*]", "", s)
```
No builder-side workaround can legitimately satisfy the defective matcher.
live_final_generation/harness-artifacts/gate-baseline.txt
+6
View File
@@ -0,0 +1,6 @@
exit 1
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
FAIL: expected bench.py to exist, found nothing, at /private/tmp/ddc0f4d0/fusion-harness/bench.py — implement BENCH_PLAN.md as a single-file uv script named bench.py in /private/tmp/ddc0f4d0/fusion-harness (stdlib only, runnable with `uv run bench.py`)
RESULT: RED (gate cannot proceed without bench.py)
live_final_generation/harness-artifacts/gate-round-1.txt
+19
View File
@@ -0,0 +1,19 @@
exit 1
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
PASS: bench.py has a PEP 723 `# /// script` metadata block
PASS: PEP 723 dependencies are empty (stdlib only)
PASS: bench.py imports only Python standard library modules
PASS: bench.py uses sqlite3 (real inserts, not simulated)
PASS: bench.py isolates strategies in separate processes
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
PASS: bench.py targets N=1,000,000 rows
PASS: `uv run bench.py` ran to completion (exit 0)
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1731 s, 1459.70x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: waltunedgen — 25.657 MB peak RSS, +3.375 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
RESULT: RED (11 passed, 4 failed)
live_final_generation/harness-artifacts/gate-round-2.txt
+19
View File
@@ -0,0 +1,19 @@
exit 1
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
PASS: bench.py has a PEP 723 `# /// script` metadata block
PASS: PEP 723 dependencies are empty (stdlib only)
PASS: bench.py imports only Python standard library modules
PASS: bench.py uses sqlite3 (real inserts, not simulated)
PASS: bench.py isolates strategies in separate processes
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
PASS: bench.py targets N=1,000,000 rows
PASS: `uv run bench.py` ran to completion (exit 0)
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.2118 s, 1269.32x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: waltunedgen — 25.625 MB peak RSS, +3.359 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
RESULT: RED (11 passed, 4 failed)
live_final_generation/harness-artifacts/gate-round-3.txt
+19
View File
@@ -0,0 +1,19 @@
exit 1
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
PASS: bench.py has a PEP 723 `# /// script` metadata block
PASS: PEP 723 dependencies are empty (stdlib only)
PASS: bench.py imports only Python standard library modules
PASS: bench.py uses sqlite3 (real inserts, not simulated)
PASS: bench.py isolates strategies in separate processes
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
PASS: bench.py targets N=1,000,000 rows
PASS: `uv run bench.py` ran to completion (exit 0)
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1768 s, 2030.00x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: waltunedgen — 25.657 MB peak RSS, +3.211 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
RESULT: RED (11 passed, 4 failed)
live_final_generation/harness-artifacts/gate-round-4.txt
+19
View File
@@ -0,0 +1,19 @@
exit 1
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
PASS: bench.py has a PEP 723 `# /// script` metadata block
PASS: PEP 723 dependencies are empty (stdlib only)
PASS: bench.py imports only Python standard library modules
PASS: bench.py uses sqlite3 (real inserts, not simulated)
PASS: bench.py isolates strategies in separate processes
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
PASS: bench.py targets N=1,000,000 rows
PASS: `uv run bench.py` ran to completion (exit 0)
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1667 s, 1733.05x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: executemanygen — 25.608 MB peak RSS, +3.441 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
RESULT: RED (11 passed, 4 failed)
live_final_generation/harness-artifacts/gate-round-5.txt
+21
View File
@@ -0,0 +1,21 @@
exit 130
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
PASS: bench.py has a PEP 723 `# /// script` metadata block
PASS: PEP 723 dependencies are empty (stdlib only)
PASS: bench.py imports only Python standard library modules
PASS: bench.py uses sqlite3 (real inserts, not simulated)
PASS: bench.py isolates strategies in separate processes
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
PASS: bench.py targets N=1,000,000 rows
PASS: `uv run bench.py` ran to completion (exit 0)
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1700 s, 1147.98x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: executemanygen — 25.625 MB peak RSS, +3.588 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
RESULT: RED (11 passed, 4 failed)
[stopped by user]
live_final_generation/harness-artifacts/gate.py
+477
View File
@@ -0,0 +1,477 @@
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""
Acceptance gate for: implement BENCH_PLAN.md as bench.py (uv single-file
script, stdlib only) that runs to completion via `uv run bench.py`, writes
RESULTS.md with a markdown table holding a measured time AND a measured peak
RAM figure for every strategy in the plan, declares a Speed Winner and a
Memory Winner from those measurements, and where the fastest strategy beats
the naive autocommit baseline by at least 5x.
Exit 0 iff every check passes. One PASS/FAIL line per check.
"""
import ast
import os
import re
import subprocess
import sys
import time
ROOT = "/private/tmp/ddc0f4d0/fusion-harness"
BENCH = os.path.join(ROOT, "bench.py")
RESULTS = os.path.join(ROOT, "RESULTS.md")
PLAN = os.path.join(ROOT, "BENCH_PLAN.md")
# The seven strategies defined by BENCH_PLAN.md ("Strategies (7)" table and
# the STRATEGIES dict in its appendix script).
STRATEGIES = [
"naive_autocommit",
"one_big_txn_loop",
"executemany_list",
"executemany_gen",
"wal_tuned_gen",
"max_tuned_gen",
"set_based_ctes",
]
BASELINE = "naive_autocommit"
RUN_TIMEOUT = 240 # plan budget is ~3 minutes; measured fused run was ~7s
_results: list[bool] = []
def ok(msg: str) -> None:
_results.append(True)
print("PASS: " + msg)
def bad(msg: str) -> None:
_results.append(False)
print("FAIL: " + msg)
def one_line(s: str, limit: int = 500) -> str:
s = " | ".join(part.strip() for part in s.strip().splitlines() if part.strip())
return s[-limit:] if len(s) > limit else s
def clean_md(s: str) -> str:
return re.sub(r"[`*_]", "", s)
def first_float(cell: str):
m = re.search(r"-?\d+(?:\.\d+)?", cell.replace(",", ""))
return float(m.group(0)) if m else None
def split_cells(line: str):
line = line.strip()
if line.startswith("|"):
line = line[1:]
if line.endswith("|"):
line = line[:-1]
return [c.strip() for c in line.split("|")]
def parse_tables(text: str):
"""Return list of (header_cells, data_rows) for every markdown table."""
lines = text.splitlines()
tables = []
i = 0
sep_re = re.compile(r"^\s*\|?[\s:|-]+\|?\s*$")
while i < len(lines):
if (
"|" in lines[i]
and i + 1 < len(lines)
and "-" in lines[i + 1]
and "|" in lines[i + 1]
and sep_re.match(lines[i + 1])
):
header = split_cells(lines[i])
j = i + 2
rows = []
while j < len(lines) and "|" in lines[j] and lines[j].strip():
rows.append(split_cells(lines[j]))
j += 1
tables.append((header, rows))
i = j
else:
i += 1
return tables
def main() -> int:
# ---- 0. plan still present (the thing bench.py must implement) --------
if os.path.isfile(PLAN):
plan_txt = open(PLAN, encoding="utf-8", errors="replace").read()
missing = [s for s in STRATEGIES if s not in plan_txt]
if missing:
bad(
f"expected BENCH_PLAN.md to still define strategies {missing}, "
f"found them absent, at {PLAN} — restore BENCH_PLAN.md; the "
f"plan is the spec and must not be edited"
)
else:
ok("BENCH_PLAN.md present and still defines all 7 plan strategies")
else:
bad(
f"expected BENCH_PLAN.md to exist, found nothing, at {PLAN} — "
f"restore the plan file; it is the spec bench.py implements"
)
# ---- 1. bench.py exists ------------------------------------------------
if not os.path.isfile(BENCH):
bad(
f"expected bench.py to exist, found nothing, at {BENCH} — "
f"implement BENCH_PLAN.md as a single-file uv script named "
f"bench.py in {ROOT} (stdlib only, runnable with `uv run bench.py`)"
)
print("\nRESULT: RED (gate cannot proceed without bench.py)")
return 1
ok(f"bench.py exists at {BENCH}")
src = open(BENCH, encoding="utf-8", errors="replace").read()
# ---- 2. PEP 723 header, empty dependencies (stdlib only) --------------
if re.search(r"^#\s*///\s*script\s*$", src, re.MULTILINE):
ok("bench.py has a PEP 723 `# /// script` metadata block")
else:
bad(
f"expected a PEP 723 block starting with `# /// script`, found "
f"none, at {BENCH} — add the inline metadata block "
f"(`# /// script`, `# requires-python = ...`, "
f"`# dependencies = []`, `# ///`) at the top of bench.py"
)
dep_lines = [l for l in src.splitlines() if re.search(r"^\#\s*dependencies\s*=", l)]
if dep_lines and not re.search(r"dependencies\s*=\s*\[\s*\]", dep_lines[0]):
bad(
f"expected `dependencies = []` (stdlib only), found "
f"`{dep_lines[0].strip()}`, at {BENCH} — remove all third-party "
f"dependencies from the PEP 723 block; the plan requires Python "
f"standard library only"
)
else:
ok("PEP 723 dependencies are empty (stdlib only)")
# ---- 3. all imports are stdlib -----------------------------------------
try:
tree = ast.parse(src)
roots = set()
for node in ast.walk(tree):
if isinstance(node, ast.Import):
for a in node.names:
roots.add(a.name.split(".")[0])
elif isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
roots.add(node.module.split(".")[0])
non_std = sorted(r for r in roots if r not in sys.stdlib_module_names)
if non_std:
bad(
f"expected only stdlib imports, found non-stdlib {non_std}, "
f"at {BENCH} — rewrite bench.py using only the Python "
f"standard library"
)
else:
ok("bench.py imports only Python standard library modules")
except SyntaxError as e:
bad(
f"expected bench.py to be valid Python, found SyntaxError "
f"`{one_line(str(e))}`, at {BENCH} — fix the syntax error"
)
# ---- 4. static plan fidelity: real measurement machinery ---------------
if "sqlite3" in src:
ok("bench.py uses sqlite3 (real inserts, not simulated)")
else:
bad(
f"expected bench.py to import/use sqlite3, found no mention, at "
f"{BENCH} — the benchmark must perform real SQLite inserts per "
f"BENCH_PLAN.md"
)
if re.search(r"subprocess|multiprocessing", src):
ok("bench.py isolates strategies in separate processes")
else:
bad(
f"expected per-strategy process isolation (subprocess re-exec or "
f"multiprocessing per BENCH_PLAN.md), found neither, at {BENCH} — "
f"run each strategy in its own fresh process so peak-RAM readings "
f"cannot pollute each other"
)
if re.search(r"ru_maxrss|getrusage", src):
ok("bench.py measures peak RAM via getrusage/ru_maxrss")
else:
bad(
f"expected measured peak RAM via resource.getrusage(...).ru_maxrss "
f"per BENCH_PLAN.md, found no getrusage/ru_maxrss, at {BENCH} — "
f"measure whole-process peak RSS inside each isolated worker; "
f"do not assume or hardcode memory figures"
)
if re.search(r"1_000_000|1000000", src):
ok("bench.py targets N=1,000,000 rows")
else:
bad(
f"expected N=1,000,000 rows (literal 1_000_000 or 1000000), found "
f"neither, at {BENCH} — the plan requires 1M rows for every "
f"full-run strategy (baseline may be sampled and scaled)"
)
# ---- 5. `uv run bench.py` runs to completion ---------------------------
pre_marker = time.time()
proc = None
ran_ok = False
try:
proc = subprocess.run(
["uv", "run", "bench.py"],
cwd=ROOT,
capture_output=True,
text=True,
timeout=RUN_TIMEOUT,
)
if proc.returncode == 0:
ok("`uv run bench.py` ran to completion (exit 0)")
ran_ok = True
else:
bad(
f"expected `uv run bench.py` to exit 0, found exit "
f"{proc.returncode}, at {BENCH} — fix the crash; stderr tail: "
f"{one_line(proc.stderr or proc.stdout or '(empty)')}"
)
except FileNotFoundError:
bad(
f"expected `uv` on PATH so `uv run bench.py` works, found uv "
f"missing, at {ROOT} — ensure the script is runnable with exactly "
f"`uv run bench.py`; do not substitute another runner"
)
# best-effort fallback so content checks below still give feedback
try:
proc = subprocess.run(
[sys.executable, "bench.py"],
cwd=ROOT,
capture_output=True,
text=True,
timeout=RUN_TIMEOUT,
)
except Exception:
proc = None
except subprocess.TimeoutExpired:
bad(
f"expected `uv run bench.py` to finish within {RUN_TIMEOUT}s "
f"(plan budget ~3 minutes), found it still running, at {BENCH} — "
f"sample the naive autocommit baseline (plan: 20,000 rows scaled "
f"x50) instead of running it for the full 1M rows"
)
# ---- 6. RESULTS.md written by the run ----------------------------------
if not os.path.isfile(RESULTS):
bad(
f"expected the run to write RESULTS.md, found no file, at "
f"{RESULTS} — make bench.py write RESULTS.md (markdown table with "
f"a time column and a peak RAM column for every strategy, plus "
f"Speed Winner and Memory Winner lines) every time it runs"
)
print("\nRESULT: RED")
return 1
if os.path.getmtime(RESULTS) >= pre_marker - 2:
ok("RESULTS.md was (re)written by this `uv run bench.py` invocation")
else:
bad(
f"expected RESULTS.md to be rewritten by the run just executed, "
f"found a stale file (mtime predates the run), at {RESULTS} — "
f"bench.py itself must write RESULTS.md from its own fresh "
f"measurements on every run; do not hand-author RESULTS.md"
)
text = open(RESULTS, encoding="utf-8", errors="replace").read()
# ---- 7. markdown table: time + peak RAM for every strategy -------------
tables = parse_tables(text)
best = None
best_count = -1
for header, rows in tables:
joined = clean_md(" ".join(" ".join(r) for r in rows))
count = sum(1 for s in STRATEGIES if s in joined)
if count > best_count:
best_count = count
best = (header, rows)
times: dict[str, float] = {}
rams: dict[str, float] = {}
if best is None or best_count == 0:
bad(
f"expected a markdown table whose rows name the 7 plan strategies "
f"{STRATEGIES}, found no such table, at {RESULTS} — write one "
f"markdown table with one row per strategy using the plan's "
f"strategy names"
)
else:
header, rows = best
time_col = next(
(i for i, h in enumerate(header) if re.search(r"time|sec|duration", h, re.I)),
None,
)
ram_candidates = [
i for i, h in enumerate(header) if re.search(r"ram|rss|mem", h, re.I)
]
ram_col = next(
(i for i in ram_candidates if re.search(r"peak", header[i], re.I)),
ram_candidates[0] if ram_candidates else None,
)
if time_col is None:
bad(
f"expected a time column (header matching time/sec/duration), "
f"found headers {header}, at {RESULTS} — add a measured time "
f"column to the results table"
)
else:
ok(f"results table has a time column (`{header[time_col]}`)")
if ram_col is None:
bad(
f"expected a peak RAM column (header matching RAM/RSS/mem), "
f"found headers {header}, at {RESULTS} — add a measured peak "
f"RAM column to the results table"
)
else:
ok(f"results table has a peak RAM column (`{header[ram_col]}`)")
if time_col is not None and ram_col is not None:
for strat in STRATEGIES:
row = next(
(r for r in rows if strat in clean_md(" ".join(r))), None
)
if row is None:
bad(
f"expected a table row for strategy `{strat}`, found "
f"none, at {RESULTS} — every strategy in BENCH_PLAN.md "
f"must have its own measured row"
)
continue
t = first_float(row[time_col]) if time_col < len(row) else None
m = first_float(row[ram_col]) if ram_col < len(row) else None
if t is None or t <= 0:
bad(
f"expected a positive measured time for `{strat}`, "
f"found `{row[time_col] if time_col < len(row) else '(missing cell)'}`, "
f"at {RESULTS} — record the real measured wall time"
)
else:
times[strat] = t
if m is None or m <= 0:
bad(
f"expected a positive measured peak RAM for `{strat}`, "
f"found `{row[ram_col] if ram_col < len(row) else '(missing cell)'}`, "
f"at {RESULTS} — record the real measured peak RSS"
)
else:
rams[strat] = m
if len(times) == len(STRATEGIES):
ok("every plan strategy has a positive measured time in the table")
if len(rams) == len(STRATEGIES):
ok("every plan strategy has a positive measured peak RAM in the table")
# ---- 8. declared winners ------------------------------------------------
def declared(kind: str):
m = re.search(kind + r"\s*winner[^\n]*", text, re.I)
if not m:
return None, None
line = clean_md(m.group(0))
name = next((s for s in STRATEGIES if s in line), None)
return line, name
speed_line, speed_name = declared("speed")
if speed_name:
ok(f"RESULTS.md declares a Speed Winner: {speed_name}")
else:
bad(
f"expected a `Speed Winner:` line naming one plan strategy, found "
f"`{speed_line or 'no such line'}`, at {RESULTS} — declare the "
f"Speed Winner using the plan's strategy name"
)
mem_line, mem_name = declared("memory")
if mem_name:
ok(f"RESULTS.md declares a Memory Winner: {mem_name}")
else:
bad(
f"expected a `Memory Winner:` line naming one plan strategy, found "
f"`{mem_line or 'no such line'}`, at {RESULTS} — declare the "
f"Memory Winner using the plan's strategy name"
)
# ---- 9. winners follow from the measurements ---------------------------
if speed_name and len(times) == len(STRATEGIES):
fastest_t = min(times.values())
if times[speed_name] <= fastest_t + 1e-9:
ok(
f"Speed Winner {speed_name} matches the fastest measured time "
f"({times[speed_name]}s)"
)
else:
actual = min(times, key=times.get)
bad(
f"expected the Speed Winner to be the fastest strategy in the "
f"table (`{actual}` at {times[actual]}s), found `{speed_name}` "
f"at {times[speed_name]}s, at {RESULTS} — derive the winner "
f"from the measurements, not assumptions"
)
if mem_name and len(rams) == len(STRATEGIES):
if mem_name == BASELINE:
bad(
f"expected the Memory Winner to exclude the sampled baseline, "
f"found `{BASELINE}`, at {RESULTS} — per BENCH_PLAN.md the "
f"sampled baseline never held 1M rows of work and cannot win "
f"the RAM axis; pick the leanest full-1M strategy"
)
else:
eligible = {k: v for k, v in rams.items() if k != BASELINE}
lean = min(eligible.values())
if rams[mem_name] <= lean + 1e-9:
ok(
f"Memory Winner {mem_name} matches the lowest measured "
f"peak RAM among full-1M strategies ({rams[mem_name]})"
)
else:
actual = min(eligible, key=eligible.get)
bad(
f"expected the Memory Winner to have the lowest peak RAM "
f"among full-1M strategies (`{actual}` at {eligible[actual]}), "
f"found `{mem_name}` at {rams[mem_name]}, at {RESULTS} — "
f"derive the winner from the measurements"
)
# ---- 10. fastest beats naive autocommit by >= 5x ------------------------
if len(times) == len(STRATEGIES):
fastest_t = min(times.values())
speedup = times[BASELINE] / fastest_t if fastest_t > 0 else 0.0
if speedup >= 5.0:
ok(
f"fastest strategy beats naive autocommit by {speedup:.1f}x "
f"(>= 5x required)"
)
else:
bad(
f"expected fastest strategy >= 5x faster than naive "
f"autocommit, found {speedup:.1f}x ({times[BASELINE]}s vs "
f"{fastest_t}s), at {RESULTS} — use one big transaction / "
f"executemany / set-based insert per BENCH_PLAN.md so the "
f"speedup is real"
)
else:
bad(
f"expected complete measured times for all 7 strategies to verify "
f"the >=5x speedup, found only {sorted(times)} parseable, at "
f"{RESULTS} — fix the results table first"
)
if not ran_ok:
pass # the uv-run FAIL above already carries the instruction
failed = _results.count(False)
print(
f"\nRESULT: {'GREEN' if failed == 0 else 'RED'} "
f"({_results.count(True)} passed, {failed} failed)"
)
return 0 if failed == 0 else 1
if __name__ == "__main__":
sys.exit(main())
live_final_generation/harness-artifacts/prompt.md
+1
View File
@@ -0,0 +1 @@
Read BENCH_PLAN.md, the plan you two designed and fused, and implement it in this directory as bench.py, an astral uv single file script. Python standard library only, runnable with uv run bench.py. Definition of Done is defined by a gate YOU design. At minimum it proves uv run bench.py runs to completion, writes RESULTS.md with a markdown table holding a timed row AND a measured peak RAM figure for every strategy in the plan, and declares a Speed Winner and a Memory Winner from those measurements. Measured, not assumed. The fastest strategy has to beat the naive autocommit baseline by at least 5x. Baseline the gate RED before you build anything. Never shim, wrap, or modify global system tools.
live_final_generation/harness-artifacts/triage-round-3.md
+11
View File
@@ -0,0 +1,11 @@
**Diagnosis** — `GATE DEFECT:` The builder's work is correct and complete; the gate can never pass it. `gate.py:61` defines `clean_md(s) = re.sub(r"[`*_]", "", s)` — intended to strip markdown emphasis, it also deletes every underscore from the text being searched. All 7 required strategy names contain underscores, so `"set_based_ctes" in clean_md(...)` is mathematically unsatisfiable for any possible RESULTS.md (the gate's own round-2/3 output proves it: it echoes `setbasedctes` / `waltunedgen` — mangled by the gate, not written that way). This single line causes all four FAILs (table match count 0 → "no such table", both winner-name checks, empty parsed times → 5x check). I verified the real state: `/private/tmp/ddc0f4d0/fusion-harness/RESULTS.md` contains a well-formed table with all 7 exact plan names, positive measured times and peak-RAM figures, a Speed Winner (`set_based_ctes`, 0.1768s = table minimum), a Memory Winner (`wal_tuned_gen`, 25.657 MB = minimum among full-1M rows, baseline excluded), and a 2030x speedup vs naive — every substantive requirement is already met. The builder's round-3 report is accurate; rounds 2–3 are identical because there is nothing left for the builder to fix.
**Do exactly this**
1. **HUMAN/HARNESS (not builder):** patch `/tmp/fusion-harness-K48sbW/gate.py` line 61 from `return re.sub(r"[`*_]", "", s)` to `return re.sub(r"[`*]", "", s)`. Underscores must survive cleaning; backtick/asterisk stripping stays. No other gate change is needed.
2. **BUILDER:** change nothing. Leave `/private/tmp/ddc0f4d0/fusion-harness/bench.py` and `RESULTS.md` exactly as they are — they satisfy every check once matching works (RESULTS.md is regenerated fresh each run, so mtime/consistency checks will pass).
3. Re-run the gate once the patch lands; expect GREEN.
**Do NOT**
- Do NOT rename strategies to underscore-free variants (e.g., `setbasedctes`) or strip underscores/backticks in RESULTS.md to appease the broken matcher — that diverges from BENCH_PLAN.md's canonical names and is gate-gaming.
- Do NOT let the builder edit `/tmp/fusion-harness-K48sbW/gate.py` itself; the fix is the human's call, flagged here.
- Do NOT burn rounds 4–5 re-running the unmodified gate against unmodified files — the failure is deterministic until line 61 is patched.
live_final_generation/harness-artifacts/triage-round-4.md
+11
View File
@@ -0,0 +1,11 @@
**Diagnosis** — `GATE DEFECT:` (unchanged, second escalation for the same root cause). The round-3 triage identified the exact defect and the required one-line harness-side patch, but `/tmp/fusion-harness-K48sbW/gate.py:61` still reads `return re.sub(r"[`*_]", "", s)` — the patch was never applied. That regex deletes every underscore from the searched text, while all 7 required strategy names in the gate's hardcoded `STRATEGIES` list contain underscores, so `name in clean_md(text)` is **unsatisfiable for every possible RESULTS.md** — no builder edit of any project file can ever pass checks 7–10. The loop is not stuck on the builder: I re-verified the real state and `/private/tmp/ddc0f4d0/fusion-harness/RESULTS.md` is correct and freshly measured (round 4: `set_based_ctes` 0.1667s speed winner = table minimum, `executemany_gen` 25.608 MB memory winner, 1733x ≥ 5x, all 7 exact snake_case names present — the winner even changed between rounds 3 and 4, proving values are measured, not hardcoded). The gate's own FAIL lines echo `setbasedctes`/`executemanygen` — names mangled by the gate itself, not by the builder. The builder's round-4 report is accurate; it correctly made no changes.
**Do exactly this**
1. **HUMAN/HARNESS OPERATOR — this is the only action that can produce GREEN:** edit `/tmp/fusion-harness-K48sbW/gate.py` line 61 from `return re.sub(r"[`*_]", "", s)` to `return re.sub(r"[`*]", "", s)`. One character class change; nothing else in the gate needs touching. Apply it **before** round 5 — it is the final round under the cap, and running it unpatched guarantees a false RED.
2. **BUILDER:** make no changes. `/private/tmp/ddc0f4d0/fusion-harness/bench.py` and `RESULTS.md` already satisfy every substantive requirement; the gate re-runs `uv run bench.py` itself and regenerates RESULTS.md, so all mtime/consistency checks will pass.
3. Re-run the gate once patched; expect GREEN (15/15 style output, exit 0).
**Do NOT**
- Do NOT spend round 5 re-running the unpatched gate — the failure is deterministic and proven unsatisfiable; that would exhaust the cap on a known-defective check.
- Builder: do NOT attempt workarounds (renaming strategies without underscores, zero-width characters, duplicate underscore-free rows) — the gate's needle list itself contains underscores, so no RESULTS.md content can match, and any such edit is gate-gaming that corrupts an otherwise-correct deliverable.
- Builder: do NOT edit `/tmp/fusion-harness-K48sbW/gate.py` yourself — the fix is flagged for the human, per the escalation contract.
live_final_generation/harness-artifacts/validator.md
+3
View File
@@ -0,0 +1,3 @@
Gate written to /tmp/fusion-harness-K48sbW/gate.py.
It verifies, in order: BENCH_PLAN.md intact; `bench.py` exists at the project root with a PEP 723 block, empty `dependencies` and stdlib-only imports (AST scan); static plan fidelity (real sqlite3 inserts, per-strategy process isolation, `getrusage`/`ru_maxrss` RAM measurement, N=1,000,000); `uv run bench.py` exits 0 within the ~3-minute budget; RESULTS.md is freshly (re)written by that very run; its markdown table has a time column and a peak-RAM column with positive measured values for all 7 plan strategies; Speed Winner and Memory Winner are declared and are consistent with the table (memory winner excludes the sampled baseline per plan); and the fastest strategy beats naive autocommit by ≥5x. Currently RED — bench.py does not exist yet.
live_final_generation/project-root/BENCH_PLAN.md
+340
View File
@@ -0,0 +1,340 @@
# BENCH_PLAN — SQLite Bulk Insert Benchmark (fused design)
Fusion of two independent designs: **[ARCHITECT] anthropic/claude-fable-5** and **[BUILDER] openai/gpt-5.6-sol**. Where the designs disagreed, the simpler one was chosen (reasons inline and in the closing section). The fused script was **built and executed** — every number below is measured, not projected from the source answers.
- Fused runnable script: `/tmp/fused-sqlite-bench-claude-fable-5-gpt-5.6-sol.py` (also embedded in full at the end of this plan)
- Run with: `uv run /tmp/fused-sqlite-bench-claude-fable-5-gpt-5.6-sol.py`
- One astral-uv single-file script (PEP 723 header, `dependencies = []`), Python stdlib only
- Measured total wall time: **6.8 s** — well inside the 3-minute budget
## Method
**Self-isolating architecture** [ARCHITECT]. The script is both orchestrator and worker: it re-execs itself (`sys.executable __file__ --worker NAME`) once per strategy, so every strategy runs in a **fresh process against a fresh temp database**. Peak-RSS readings physically cannot pollute each other — this satisfies the isolated-process RAM requirement with zero third-party dependencies. [BUILDER] independently arrived at the same fresh-subprocess-per-trial isolation, so this is consensus architecture.
**Peak RAM** = whole-process `resource.getrusage(...).ru_maxrss`, normalized (bytes on macOS, KB×1024 on Linux) [both]. A **no-op calibration worker** measures bare-interpreter RSS (~20.3 MB here) so the table reports both absolute peak and delta-over-baseline [ARCHITECT].
**Timing boundary** [consensus]: the clock covers row generation + bind + insert + commit; connection setup, schema creation, and validation sit outside the timer. Generation is identical work for every strategy, so including it is neutral — excluding it would unfairly flatter the list-materializing strategy [ARCHITECT]. WAL runs additionally pay for `wal_checkpoint(TRUNCATE)` *inside* the timed region, so WAL is not credited with "finishing" while a million rows still sit in the `-wal` sidecar [BUILDER].
### Strategies (7)
| # | Strategy | What it does | Source |
|---|----------|--------------|--------|
| 1 | `naive_autocommit` | per-row implicit transaction — the baseline (sampled, see below) | both |
| 2 | `one_big_txn_loop` | `BEGIN` + `execute()` loop + one `COMMIT` | both |
| 3 | `executemany_list` | one txn, all 1M tuples materialized first — the classic mistake, kept to expose its RAM cost | [ARCHITECT] |
| 4 | `executemany_gen` | one txn, `executemany` fed a generator | both |
| 5 | `wal_tuned_gen` | `journal_mode=WAL` + `synchronous=NORMAL` + strategy 4 (WAL tuning combined with the predicted Python-bound winner, per the request) | both |
| 6 | `max_tuned_gen` | `journal_mode=OFF` + `synchronous=OFF` + strategy 4 — fastest way to push Python-resident rows | [ARCHITECT] |
| 7 | `set_based_ctes` | **the "beats all" candidate**: `INSERT … SELECT` over a recursive CTE that generates the *identical* rows inside SQLite — zero Python→SQLite binding crossings; run at OFF/OFF so it competes at the same durability tier as #6 | [BUILDER] |
Chosen over discarded alternatives:
- [BUILDER]'s set-based candidate replaces [ARCHITECT]'s `sqlite3` CLI `.import`: **simpler** (no external binary, no CSV temp file, no graceful-skip logic — pure stdlib in-process SQL) *and* it measured ~2× faster than everything else anyway.
- [BUILDER]'s chunked multi-row `VALUES` strategy was dropped: the variable-limit math and chunk assembly add complexity, and its niche (fewer Python↔SQLite crossings) is bracketed by `executemany_gen` below it and `set_based_ctes` above it.
- [ARCHITECT]'s 64 MB `cache_size` pragma was dropped from the tuned strategies in favor of [BUILDER]'s bounded-default-cache stance: **simpler**, and [ARCHITECT]'s own measurements showed the big cache costs ~50 MB of resident memory for essentially zero speed gain. (Fused result: `wal_tuned_gen` peaks at 24 MB instead of ARCHITECT's 71 MB.)
## Fairness controls
- **Identical data** [both]: every strategy inserts the same deterministic rows `(i, i*0.5, "payload-{i:012d}")` — one Python definition, no RNG. `i*0.5` is exact in floating point, so SQL-generated and Python-generated values are bit-identical.
- **Identical schema** [both]: `CREATE TABLE t (a INTEGER, b REAL, c TEXT)` (INTEGER/REAL/TEXT storage classes covered), fresh temp DB per worker, deleted afterward including `-wal`/`-shm`. [ARCHITECT]'s 3-column schema chosen over [BUILDER]'s 6-column `events` table as the simpler design that still exercises the same code paths.
- **Integrity gate, hard-fail** [fused]: every worker verifies `count(*)`, the closed-form `sum(a)` checksum, and `sum(length(c))` [ARCHITECT's gate], plus spot-row equality at indices `{0, 1, N/2, N−1}` against the Python row function [distilled from BUILDER's digest] — required so the set-based strategy *proves* it wrote identical data rather than just the right number of rows. No strategy can win by writing less or writing different bytes. This is simpler than [BUILDER]'s 11-field digest + `quick_check` while closing the same loophole.
- **Honest sampling, declared up front** [both]: `naive_autocommit` is fsync-bound (one journal sync per row); a full run would take ~3 minutes alone. It runs **20,000 real rows scaled ×50**, marked `*` in the table. Everything else runs the full 1,000,000. [ARCHITECT]'s 20k×50 chosen over [BUILDER]'s 10k×100 — equal simplicity, smaller scaling factor is the more honest extrapolation (both project to ~190 s regardless).
- **Baseline excluded from the memory verdict** [both, independently]: the sampled baseline never held 1M rows of work, so it cannot win the RAM axis.
- **Single run per strategy, fixed order** [ARCHITECT]: chosen over [BUILDER]'s 2×-with-median and seeded order shuffling as the simpler design. Justification: two full executions of the fused benchmark showed per-strategy spread <0.05 s while the winner gaps are ≥0.19 s, and process isolation already removes state carryover.
- **Durability labeling** [BUILDER]: OFF/OFF strategies (#6, #7) are flagged in the output as *rebuildable staging loads only* — a crash mid-load can corrupt the DB. They race in the same heats but carry the warning label.
## Exact output format
```
SQLite bulk insert benchmark | N=1,000,000 rows | py <ver> sqlite <ver> <platform>
[done] <strategy>: <t>s (<rows> rows[ sampled]) ← one line per isolated process
Strategy Time 1M rows Speedup Peak RSS RSS -base
----------------------------------------------------------------------
<name> <t>s[*] <x>x <MB>MB <MB>MB ← sorted fastest-first
----------------------------------------------------------------------
* sampled at 20,000 rows, honestly scaled x50 | base RSS <MB>MB (no-op interpreter)
note: max_tuned_gen and set_based_ctes run journal_mode=OFF/synchronous=OFF -> rebuildable staging loads only.
Speed Winner : <name> (<t>s, <x>x vs naive)
Memory Winner: <name> (<MB>MB peak, +<MB>MB over baseline)
```
Table layout is [ARCHITECT]'s (fewer columns, speedup included, sorted fastest-first — simpler than [BUILDER]'s 7-column trial-level table); the `Speed Winner:`/`Memory Winner:` terminal lines are [BUILDER]'s exact phrasing.
## Measured results (fused run: py 3.12.13, sqlite 3.50.4, macOS / Apple Silicon)
| Strategy | Time 1M rows | Speedup | Peak RSS | RSS −base |
|---|---:|---:|---:|---:|
| **set_based_ctes** | **0.17s** | **1126.2x** | 24.3 MB | 4.1 MB |
| max_tuned_gen | 0.37s | 518.4x | 24.1 MB | 3.9 MB |
| executemany_gen | 0.38s | 508.8x | 24.1 MB | 3.9 MB |
| **wal_tuned_gen** | 0.40s | 475.6x | **23.9 MB** | **3.7 MB** |
| executemany_list | 0.49s | 393.5x | 226.8 MB | 206.6 MB |
| one_big_txn_loop | 0.51s | 374.9x | 24.0 MB | 3.8 MB |
| naive_autocommit | 191.15s* | 1.0x | 22.7 MB | 2.5 MB |
\* sampled at 20,000 rows, scaled ×50. Base RSS 20.2 MB (no-op interpreter). Total wall time 6.8 s.
Key findings the fused design makes visible:
1. **The single transaction is ~99% of the win** — 375× from `BEGIN`/`COMMIT` alone [ARCHITECT's finding, confirmed].
2. **Eliminating the million Python→SQLite binding crossings is the last 2×** — set-based insertion at 0.17 s [BUILDER's finding, confirmed on the identical schema].
3. **`executemany_list` is strictly dominated**: slower than the generator *and* ~200 MB heavier [ARCHITECT's finding, confirmed].
4. **Memory is a four-way statistical tie** among all streaming strategies (23.9–24.3 MB, within ±0.4 MB across repeat runs): once you stream rows instead of materializing them, SQLite's bounded page cache makes peak RSS nearly strategy-invariant.
## Declared Winners
- **Speed Winner: `set_based_ctes`** — `INSERT … SELECT` over a recursive CTE, OFF/OFF pragmas: **0.17 s for 1M rows, ~1126× over naive autocommit.** Caveats carried from [BUILDER]: rebuildable-staging durability only, and it applies only when rows are derivable in SQL. For arbitrary Python-resident rows, the practical speed winner is `max_tuned_gen` (0.37 s), with `wal_tuned_gen` (0.40 s) the fastest *durable-enough* option — the same practical hierarchy both source models converged on.
- **Memory Winner: `wal_tuned_gen`** — **23.9 MB peak, +3.7 MB over a bare interpreter**, as selected by the declared rule (minimum peak RSS among full-1M runs, speed as tiebreak) in both verification runs. Honest note: this is a statistical tie with `one_big_txn_loop`, `executemany_gen`, and `max_tuned_gen` (all within 0.4 MB); the real memory lesson is *stream, never materialize* — the list variant costs 206 MB extra for nothing.
---
## Consensus & Divergence
**Consensus (both models, independently):** fresh-subprocess isolation per strategy with `ru_maxrss` for peak RSS; one deterministic Python-defined dataset and one schema for all strategies; timing bounded to the insert phase with setup/validation outside; sampling the naive baseline with declared linear scaling; excluding the sampled baseline from the memory verdict; integrity verification that hard-fails; labeling OFF/OFF as unsafe-for-production; and the headline conclusion that *one big transaction is most of the win* while *the memory winner is the streaming single-transaction family* (both declared `one transaction + execute` their memory winner; the fused measurement shows that whole family tied within noise, with `wal_tuned_gen` taking it by 0.1–0.2 MB under the fused rule).
**Divergences and rulings** (simpler option preferred per the fusion brief):
- *Beats-all candidate*: [BUILDER gpt-5.6-sol]'s set-based `INSERT…SELECT` **kept**; [ARCHITECT claude-fable-5]'s CLI `.import` **discarded** — set-based is simpler (no external binary, CSV staging, or skip logic) and empirically faster; the CLI route also strains the "stdlib only" spirit.
- *Speed winner dispute*: [ARCHITECT] declared `max_tuned_gen`, [BUILDER] declared OFF/OFF `INSERT-SELECT`. Settled by running both on the identical fused schema: **[BUILDER]'s pick wins** (0.17 s vs 0.37 s).
- *Schema & data*: [ARCHITECT]'s 3-column table **kept** over [BUILDER]'s 6-column `events` table — simpler, same storage classes exercised.
- *Repeats & trial ordering*: [ARCHITECT]'s single run in fixed order **kept** over [BUILDER]'s median-of-2 with seeded shuffle — simpler, and measured run-to-run spread (<0.05 s) is far below decision margins.
- *Verification*: [ARCHITECT]'s count+checksum gate **kept as the base**, extended with a slim length-digest + spot-row check distilled from [BUILDER]'s 11-field digest — the minimum needed to prove the SQL-generated rows are byte-identical; [BUILDER]'s full digest and `PRAGMA quick_check` **discarded** as redundant for this schema.
- *Cache tuning*: [ARCHITECT]'s 64 MB `cache_size` **discarded** in favor of [BUILDER]'s bounded-cache stance — simpler, and ARCHITECT's own data showed +50 MB RSS for no speed benefit.
- *WAL checkpoint in timed region*: [BUILDER]'s control **adopted** — not a simplification but a fairness correction; without it WAL under-reports its true cost.
- *Baseline sample size*: [ARCHITECT]'s 20k×50 **kept** over [BUILDER]'s 10k×100 — equal complexity, smaller extrapolation factor.
- *Also discarded*: [BUILDER]'s multi-row `VALUES` strategy (complexity without a podium finish — dominated on both ends), [BUILDER]'s Windows `PeakWorkingSetSize` branch (POSIX `ru_maxrss` is simpler; noted as an easy portability extension), and [ARCHITECT]'s `executemany_list`… no — that one was **kept**: three extra lines that buy the single most vivid RAM lesson in the table.
---
## Appendix — the fused script (verbatim, as executed)
```python
# /// script
# requires-python = ">=3.10"
# dependencies = []
# ///
"""
FUSED SQLite bulk-insert benchmark (claude-fable-5 + gpt-5.6-sol).
1,000,000 rows. Speed + peak RAM. Stdlib only. Run:
uv run /tmp/fused-sqlite-bench-claude-fable-5-gpt-5.6-sol.py
Architecture (from ARCHITECT/claude-fable-5): the script re-execs ITSELF
once per strategy (`--worker NAME`), so every strategy runs in a fresh
process and peak RSS readings cannot pollute each other. A no-op
calibration worker measures bare-interpreter RSS for a delta column.
Beyond-the-list candidate (from BUILDER/gpt-5.6-sol): set-based
INSERT ... SELECT over a recursive CTE that generates the IDENTICAL rows
inside SQLite, verified by digest + spot-check against the Python rows.
WAL runs pay for wal_checkpoint(TRUNCATE) inside the timed region.
"""
import json
import os
import resource
import sqlite3
import subprocess
import sys
import tempfile
import time
N = 1_000_000
NAIVE_SAMPLE_N = 20_000 # honest sample; scaled x(N/sample) in report
SCHEMA = "CREATE TABLE t (a INTEGER, b REAL, c TEXT)"
INSERT = "INSERT INTO t VALUES (?,?,?)"
SET_BASED_SQL = """
WITH RECURSIVE seq(i) AS (
SELECT 0
UNION ALL
SELECT i + 1 FROM seq WHERE i + 1 < ?
)
INSERT INTO t SELECT i, i * 0.5, printf('payload-%012d', i) FROM seq
"""
# ---------------------------------------------------------------- data ----
def make_row(i):
return (i, i * 0.5, f"payload-{i:012d}")
def rows(n):
"""Identical deterministic data for every strategy."""
for i in range(n):
yield make_row(i)
# ---------------------------------------------------------- strategies ----
def s_naive_autocommit(conn, n):
conn.isolation_level = None # every INSERT commits: journal sync per row
for r in rows(n):
conn.execute(INSERT, r)
def s_one_big_txn_loop(conn, n):
conn.execute("BEGIN")
for r in rows(n):
conn.execute(INSERT, r)
conn.commit()
def s_executemany_list(conn, n):
data = list(rows(n)) # deliberately materialized: shows RAM cost
conn.execute("BEGIN")
conn.executemany(INSERT, data)
conn.commit()
def s_executemany_gen(conn, n):
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
def s_wal_tuned_gen(conn, n):
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
# Fairness (BUILDER): WAL must not finish with 1M rows still in the -wal
conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
def s_max_tuned_gen(conn, n):
"""Python-bound speed candidate: one txn + executemany(generator) +
journaling/sync disabled for the load. Rebuildable staging only."""
conn.execute("PRAGMA journal_mode=OFF")
conn.execute("PRAGMA synchronous=OFF")
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
def s_set_based(conn, n):
"""Beyond-the-list candidate (BUILDER): SQLite generates the identical
rows itself; zero Python->SQLite binding crossings. OFF/OFF pragmas so
it is compared at the same durability tier as max_tuned_gen."""
conn.execute("PRAGMA journal_mode=OFF")
conn.execute("PRAGMA synchronous=OFF")
conn.execute("BEGIN")
conn.execute(SET_BASED_SQL, (n,))
conn.commit()
STRATEGIES = {
"naive_autocommit": (s_naive_autocommit, NAIVE_SAMPLE_N),
"one_big_txn_loop": (s_one_big_txn_loop, N),
"executemany_list": (s_executemany_list, N),
"executemany_gen": (s_executemany_gen, N),
"wal_tuned_gen": (s_wal_tuned_gen, N),
"max_tuned_gen": (s_max_tuned_gen, N),
"set_based_ctes": (s_set_based, N),
}
# --------------------------------------------------------------- worker ----
def rss_bytes():
v = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
return v if sys.platform == "darwin" else v * 1024 # KB on Linux
def verify(dbpath, n):
"""Integrity gate (fused): count + checksum for everyone [ARCHITECT],
plus length-digest and spot-row equality so the SQL-generated data is
provably identical to the Python data [BUILDER]."""
conn = sqlite3.connect(dbpath)
cnt, chk, clen = conn.execute(
"SELECT count(*), sum(a), sum(length(c)) FROM t").fetchone()
ok = (cnt == n and chk == n * (n - 1) // 2 and clen == 20 * n)
for i in (0, 1, n // 2, n - 1):
got = conn.execute("SELECT a, b, c FROM t WHERE a=?", (i,)).fetchone()
ok = ok and got is not None and tuple(got) == make_row(i)
conn.close()
return ok
def worker(name):
if name == "calibration": # interpreter + sqlite3 import baseline
print(json.dumps({"name": name, "rows": 0, "seconds": 0.0,
"peak_rss": rss_bytes(), "ok": True}))
return
fn, n = STRATEGIES[name]
dbpath = tempfile.mktemp(suffix=".db", dir=tempfile.gettempdir())
try:
conn = sqlite3.connect(dbpath)
conn.execute(SCHEMA)
t0 = time.perf_counter() # timer: generation + bind + insert + commit
fn(conn, n)
dt = time.perf_counter() - t0
conn.close()
ok = verify(dbpath, n) # validation outside the timer
print(json.dumps({"name": name, "rows": n, "seconds": dt,
"peak_rss": rss_bytes(), "ok": ok}))
finally:
for p in (dbpath, dbpath + "-wal", dbpath + "-shm"):
if os.path.exists(p):
os.remove(p)
# ----------------------------------------------------------- orchestrator --
def spawn(name):
out = subprocess.run([sys.executable, os.path.abspath(__file__),
"--worker", name],
capture_output=True, text=True, check=True)
return json.loads(out.stdout.strip().splitlines()[-1])
def main():
t_total = time.perf_counter()
print(f"SQLite bulk insert benchmark | N={N:,} rows | "
f"py {sys.version.split()[0]} sqlite {sqlite3.sqlite_version} "
f"{sys.platform}\n")
base = spawn("calibration")
results = []
for name in STRATEGIES: # fixed, deterministic order
r = spawn(name)
assert r["ok"], f"integrity check FAILED for {name}"
scale = N / r["rows"]
r["scaled"] = r["seconds"] * scale
r["sampled"] = scale > 1.0
results.append(r)
print(f" [done] {name}: {r['seconds']:.3f}s "
f"({r['rows']:,} rows{' sampled' if r['sampled'] else ''})")
naive = next(r["scaled"] for r in results if r["name"] == "naive_autocommit")
print(f"\n{'Strategy':<22}{'Time 1M rows':>14}{'Speedup':>10}"
f"{'Peak RSS':>12}{'RSS -base':>12}")
print("-" * 70)
for r in sorted(results, key=lambda r: r["scaled"]):
mark = "*" if r["sampled"] else " "
print(f"{r['name']:<22}{r['scaled']:>12.2f}s{mark}"
f"{naive / r['scaled']:>9.1f}x"
f"{r['peak_rss'] / 1e6:>10.1f}MB"
f"{(r['peak_rss'] - base['peak_rss']) / 1e6:>10.1f}MB")
print("-" * 70)
print(f"* sampled at {NAIVE_SAMPLE_N:,} rows, honestly scaled "
f"x{N // NAIVE_SAMPLE_N} | base RSS "
f"{base['peak_rss'] / 1e6:.1f}MB (no-op interpreter)")
print("note: max_tuned_gen and set_based_ctes run journal_mode=OFF/"
"synchronous=OFF -> rebuildable staging loads only.")
fastest = min(results, key=lambda r: r["scaled"])
full = [r for r in results if not r["sampled"]] # baseline excluded
leanest = min(full, key=lambda r: (r["peak_rss"], r["scaled"]))
print(f"\nSpeed Winner : {fastest['name']} "
f"({fastest['scaled']:.2f}s, {naive / fastest['scaled']:.0f}x vs naive)")
print(f"Memory Winner: {leanest['name']} "
f"({leanest['peak_rss'] / 1e6:.1f}MB peak, "
f"+{(leanest['peak_rss'] - base['peak_rss']) / 1e6:.1f}MB over baseline)")
print(f"\ntotal benchmark wall time: {time.perf_counter() - t_total:.1f}s")
if __name__ == "__main__":
if len(sys.argv) == 3 and sys.argv[1] == "--worker":
worker(sys.argv[2])
else:
main()
```
live_final_generation/project-root/RESULTS.md
+25
View File
@@ -0,0 +1,25 @@
# SQLite Bulk Insert Benchmark Results
Fresh measurements: **1,000,000 target rows**, Python 3.12.13, SQLite 3.50.4, macOS-26.5.2-arm64-arm-64bit.
The timer includes row generation, binding, insertion, and commit. WAL timing also includes `wal_checkpoint(TRUNCATE)`. Each strategy ran in a fresh process against a fresh database.
| Strategy | Measured rows | Time 1M rows (s) | Speedup vs naive | Peak RAM (MB) | RSS over base (MB) |
|---|---:|---:|---:|---:|---:|
| `set_based_ctes` | 1,000,000 | 0.1700 | 1147.98x | 26.051 | 4.014 |
| `executemany_gen` | 1,000,000 | 0.3480 | 560.77x | 25.625 | 3.588 |
| `max_tuned_gen` | 1,000,000 | 0.3484 | 560.15x | 25.641 | 3.604 |
| `executemany_list` | 1,000,000 | 0.3761 | 518.88x | 228.737 | 206.701 |
| `wal_tuned_gen` | 1,000,000 | 0.3918 | 498.07x | 25.903 | 3.867 |
| `one_big_txn_loop` | 1,000,000 | 0.6071 | 321.41x | 25.854 | 3.817 |
| `naive_autocommit` | 20,000 (sampled) | 195.1362* | 1.00x | 24.183 | 2.146 |
\* `naive_autocommit` ran 20,000 real rows; its measured time was scaled by 50. Its RAM reading is measured, not scaled, and it is excluded from the memory-winner decision.
`max_tuned_gen` and `set_based_ctes` use `journal_mode=OFF`/`synchronous=OFF`; they are for rebuildable staging loads only.
Bare-worker peak RSS calibration: **22.036 MB**.
Total benchmark wall time: **6.857 s**.
**Speed Winner: `set_based_ctes`** — 0.1700 s, 1147.98x faster than naive.
**Memory Winner: `executemany_gen`** — 25.625 MB peak RSS, +3.588 MB over the calibration worker.
live_final_generation/project-root/bench-sqlite-bulk-ARCHITECT-anthropic-claude-fable-5.py
+233
View File
@@ -0,0 +1,233 @@
# /// script
# requires-python = ">=3.10"
# dependencies = []
# ///
"""
SQLite bulk-insert benchmark: 1,000,000 rows, speed + peak RAM, per-strategy
process isolation. Stdlib only. Run: uv run bench-sqlite-bulk-ARCHITECT-anthropic-claude-fable-5.py
Design (ARCHITECT / anthropic-claude-fable-5):
* Orchestrator re-execs THIS file as a subprocess per strategy
(`--worker NAME`), so peak RSS readings never pollute each other.
* Every strategy inserts IDENTICAL deterministic data into an IDENTICAL
fresh schema, verified by count + checksum.
* Timing covers the full insert phase (row generation + bind + commit),
identical work for all strategies.
* The naive autocommit baseline is fsync-bound (~50-200 rows/ms is
impossible; it does one journal sync per row). It is SAMPLED at 20,000
rows and honestly scaled x50; the table flags it with '*'.
* Peak RAM = ru_maxrss of the worker process (normalized to bytes across
macOS/Linux), plus a no-op calibration worker so a delta over
interpreter baseline is reported.
"""
import json
import os
import resource
import sqlite3
import subprocess
import sys
import tempfile
import time
N = 1_000_000
NAIVE_SAMPLE_N = 20_000 # honest sample; scaled x(N/NAIVE_SAMPLE_N) in report
SCHEMA = "CREATE TABLE t (a INTEGER, b REAL, c TEXT)"
INSERT = "INSERT INTO t VALUES (?,?,?)"
EXPECTED_SUM = lambda n: n * (n - 1) // 2 # sum of column a for n rows
# ---------------------------------------------------------------- data ----
def rows(n):
"""Identical deterministic data for every strategy."""
for i in range(n):
yield (i, i * 0.5, f"payload-{i:012d}")
# ---------------------------------------------------------- strategies ----
def s_naive_autocommit(conn, n):
conn.isolation_level = None # every INSERT commits (journal sync per row)
for r in rows(n):
conn.execute(INSERT, r)
def s_one_big_txn_loop(conn, n):
conn.execute("BEGIN")
for r in rows(n):
conn.execute(INSERT, r)
conn.commit()
def s_executemany_list(conn, n):
data = list(rows(n)) # deliberately materialized: shows the RAM cost
conn.execute("BEGIN")
conn.executemany(INSERT, data)
conn.commit()
def s_executemany_gen(conn, n):
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
def s_wal_tuned_gen(conn, n):
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
conn.execute("PRAGMA cache_size=-64000") # 64 MB page cache
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
def s_max_tuned_gen(conn, n):
"""Predicted overall speed winner within stdlib: one txn +
executemany(generator) + journaling/sync disabled for the load."""
conn.execute("PRAGMA journal_mode=OFF")
conn.execute("PRAGMA synchronous=OFF")
conn.execute("PRAGMA cache_size=-64000")
conn.execute("PRAGMA temp_store=MEMORY")
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
STRATEGIES = {
"naive_autocommit": (s_naive_autocommit, NAIVE_SAMPLE_N),
"one_big_txn_loop": (s_one_big_txn_loop, N),
"executemany_list": (s_executemany_list, N),
"executemany_gen": (s_executemany_gen, N),
"wal_tuned_gen": (s_wal_tuned_gen, N),
"max_tuned_gen": (s_max_tuned_gen, N),
}
# Optional bonus (external `sqlite3` CLI, skipped gracefully if absent).
# Same identical data, same isolation, timing includes writing the CSV.
CLI_STRATEGY = "cli_dot_import"
def run_cli_import(dbpath, n):
import csv
import shutil
exe = shutil.which("sqlite3")
if not exe:
return False
csvpath = dbpath + ".csv"
with open(csvpath, "w", newline="") as f:
w = csv.writer(f)
for r in rows(n):
w.writerow(r)
subprocess.run(
[exe, dbpath, SCHEMA + ";", ".mode csv", f".import {csvpath} t"],
check=True, capture_output=True,
)
os.remove(csvpath)
return True
# --------------------------------------------------------------- worker ----
def rss_bytes(who=resource.RUSAGE_SELF):
v = resource.getrusage(who).ru_maxrss
return v if sys.platform == "darwin" else v * 1024 # KB on Linux
def worker(name):
if name == "calibration": # no-op: interpreter + sqlite3 import baseline
print(json.dumps({"name": name, "rows": 0, "seconds": 0.0,
"peak_rss": rss_bytes(), "ok": True}))
return
dbpath = tempfile.mktemp(suffix=".db", dir=tempfile.gettempdir())
try:
if name == CLI_STRATEGY:
t0 = time.perf_counter()
ok = run_cli_import(dbpath, N)
dt = time.perf_counter() - t0
if not ok:
print(json.dumps({"name": name, "skipped": True}))
return
n = N
else:
fn, n = STRATEGIES[name]
conn = sqlite3.connect(dbpath)
conn.execute(SCHEMA)
t0 = time.perf_counter()
fn(conn, n)
dt = time.perf_counter() - t0
conn.close()
vconn = sqlite3.connect(dbpath)
cnt, chk = vconn.execute("SELECT count(*), sum(a) FROM t").fetchone()
vconn.close()
peak = max(rss_bytes(), rss_bytes(resource.RUSAGE_CHILDREN))
print(json.dumps({
"name": name, "rows": cnt, "seconds": dt, "peak_rss": peak,
"ok": cnt == n and chk == EXPECTED_SUM(n),
}))
finally:
for p in (dbpath, dbpath + "-wal", dbpath + "-shm", dbpath + ".csv"):
if os.path.exists(p):
os.remove(p)
# ----------------------------------------------------------- orchestrator --
def spawn(name):
out = subprocess.run([sys.executable, os.path.abspath(__file__),
"--worker", name],
capture_output=True, text=True, check=True)
return json.loads(out.stdout.strip().splitlines()[-1])
def main():
print(f"SQLite bulk insert benchmark | N={N:,} rows | "
f"py {sys.version.split()[0]} sqlite {sqlite3.sqlite_version} "
f"{sys.platform}\n")
base = spawn("calibration")
results = []
order = list(STRATEGIES) + [CLI_STRATEGY]
for name in order:
r = spawn(name)
if r.get("skipped"):
print(f" [skip] {name} (sqlite3 CLI not on PATH)")
continue
assert r["ok"], f"integrity check FAILED for {name}"
scale = N / r["rows"]
r["scaled"] = r["seconds"] * scale
r["sampled"] = scale > 1.0
results.append(r)
print(f" [done] {name}: {r['seconds']:.3f}s "
f"({r['rows']:,} rows{' sampled' if r['sampled'] else ''})")
naive = next(r["scaled"] for r in results if r["name"] == "naive_autocommit")
print(f"\n{'Strategy':<22}{'Time 1M rows':>14}{'Speedup':>10}"
f"{'Peak RSS':>12}{'RSS -base':>12}")
print("-" * 70)
for r in sorted(results, key=lambda r: r["scaled"]):
mark = "*" if r["sampled"] else " "
print(f"{r['name']:<22}{r['scaled']:>12.2f}s{mark}"
f"{naive / r['scaled']:>9.1f}x"
f"{r['peak_rss'] / 1e6:>10.1f}MB"
f"{(r['peak_rss'] - base['peak_rss']) / 1e6:>10.1f}MB")
print("-" * 70)
print(f"* sampled at {NAIVE_SAMPLE_N:,} rows, honestly scaled "
f"x{N // NAIVE_SAMPLE_N} | base RSS "
f"{base['peak_rss'] / 1e6:.1f}MB (no-op interpreter)")
fastest = min(results, key=lambda r: r["scaled"])
# memory winner among full-speed strategies (exclude the naive baseline)
fast = [r for r in results if not r["sampled"]]
leanest = min(fast, key=lambda r: (r["peak_rss"], r["scaled"]))
print(f"\nSPEED WINNER : {fastest['name']} "
f"({fastest['scaled']:.2f}s, {naive / fastest['scaled']:.0f}x vs naive)")
print(f"MEMORY WINNER: {leanest['name']} "
f"({leanest['peak_rss'] / 1e6:.1f}MB peak, "
f"+{(leanest['peak_rss'] - base['peak_rss']) / 1e6:.1f}MB over baseline)")
if __name__ == "__main__":
if len(sys.argv) == 3 and sys.argv[1] == "--worker":
worker(sys.argv[2])
else:
main()
live_final_generation/project-root/bench.py
+393
View File
@@ -0,0 +1,393 @@
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""Benchmark seven SQLite bulk-insert strategies on speed and peak RAM.
Run with:
uv run bench.py
The orchestrator re-executes this file once per strategy. Each worker uses a
fresh process and a fresh SQLite database, so process peak-RSS measurements
cannot leak from one strategy into another. The run writes RESULTS.md next to
this script from fresh measurements every time.
"""
from __future__ import annotations
import json
import os
from pathlib import Path
import platform
import resource
import sqlite3
import subprocess
import sys
import tempfile
import time
from typing import Any, Callable, Iterator
N = 1_000_000
NAIVE_SAMPLE_N = 20_000
SCHEMA = "CREATE TABLE t (a INTEGER, b REAL, c TEXT)"
INSERT = "INSERT INTO t VALUES (?, ?, ?)"
RESULTS_PATH = Path(__file__).resolve().with_name("RESULTS.md")
SET_BASED_SQL = """
WITH RECURSIVE seq(i) AS (
SELECT 0
UNION ALL
SELECT i + 1 FROM seq WHERE i + 1 < ?
)
INSERT INTO t
SELECT i, i * 0.5, printf('payload-%012d', i)
FROM seq
"""
Row = tuple[int, float, str]
StrategyFunction = Callable[[sqlite3.Connection, int], None]
def make_row(i: int) -> Row:
"""Return the sole canonical definition of one benchmark row."""
return (i, i * 0.5, f"payload-{i:012d}")
def rows(n: int) -> Iterator[Row]:
"""Generate identical deterministic data for every Python-bound strategy."""
for i in range(n):
yield make_row(i)
# ---------------------------------------------------------------- strategies
def s_naive_autocommit(conn: sqlite3.Connection, n: int) -> None:
"""Baseline: every INSERT is its own durable transaction."""
conn.isolation_level = None
for row in rows(n):
conn.execute(INSERT, row)
def s_one_big_txn_loop(conn: sqlite3.Connection, n: int) -> None:
conn.execute("BEGIN")
for row in rows(n):
conn.execute(INSERT, row)
conn.commit()
def s_executemany_list(conn: sqlite3.Connection, n: int) -> None:
# Deliberately materialized so the benchmark measures the classic RAM cost.
data = list(rows(n))
conn.execute("BEGIN")
conn.executemany(INSERT, data)
conn.commit()
def s_executemany_gen(conn: sqlite3.Connection, n: int) -> None:
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
def s_wal_tuned_gen(conn: sqlite3.Connection, n: int) -> None:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
# Include write-back cost instead of ending with the rows in a WAL sidecar.
busy, _wal_pages, _checkpointed = conn.execute(
"PRAGMA wal_checkpoint(TRUNCATE)"
).fetchone()
if busy:
raise RuntimeError("WAL checkpoint was unexpectedly busy")
def s_max_tuned_gen(conn: sqlite3.Connection, n: int) -> None:
"""Maximum Python-bound speed; safe only for a rebuildable staging DB."""
conn.execute("PRAGMA journal_mode=OFF")
conn.execute("PRAGMA synchronous=OFF")
conn.execute("BEGIN")
conn.executemany(INSERT, rows(n))
conn.commit()
def s_set_based_ctes(conn: sqlite3.Connection, n: int) -> None:
"""Generate identical rows inside SQLite, avoiding per-row Python binds."""
conn.execute("PRAGMA journal_mode=OFF")
conn.execute("PRAGMA synchronous=OFF")
conn.execute("BEGIN")
conn.execute(SET_BASED_SQL, (n,))
conn.commit()
STRATEGIES: dict[str, tuple[StrategyFunction, int]] = {
"naive_autocommit": (s_naive_autocommit, NAIVE_SAMPLE_N),
"one_big_txn_loop": (s_one_big_txn_loop, N),
"executemany_list": (s_executemany_list, N),
"executemany_gen": (s_executemany_gen, N),
"wal_tuned_gen": (s_wal_tuned_gen, N),
"max_tuned_gen": (s_max_tuned_gen, N),
"set_based_ctes": (s_set_based_ctes, N),
}
# ------------------------------------------------------------------- workers
def rss_bytes() -> int:
"""Return this process's measured high-water resident set size in bytes."""
ru_maxrss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
return int(ru_maxrss if sys.platform == "darwin" else ru_maxrss * 1024)
def verify(db_path: Path, n: int) -> bool:
"""Prove row count, checksums, lengths, and representative rows match."""
conn = sqlite3.connect(db_path)
try:
count, checksum, text_length = conn.execute(
"SELECT count(*), sum(a), sum(length(c)) FROM t"
).fetchone()
valid = (
count == n
and checksum == n * (n - 1) // 2
and text_length == 20 * n
)
for i in (0, 1, n // 2, n - 1):
actual = conn.execute(
"SELECT a, b, c FROM t WHERE a = ?", (i,)
).fetchone()
valid = valid and actual is not None and tuple(actual) == make_row(i)
return bool(valid)
finally:
conn.close()
def worker(name: str) -> None:
if name == "calibration":
print(
json.dumps(
{
"name": name,
"rows": 0,
"seconds": 0.0,
"peak_rss": rss_bytes(),
"ok": True,
}
)
)
return
function, n = STRATEGIES[name]
with tempfile.TemporaryDirectory(prefix=f"sqlite-bulk-{name}-") as temp_dir:
db_path = Path(temp_dir) / "benchmark.sqlite3"
conn = sqlite3.connect(db_path)
try:
conn.execute(SCHEMA)
started = time.perf_counter()
function(conn, n)
seconds = time.perf_counter() - started
finally:
conn.close()
valid = verify(db_path, n)
result = {
"name": name,
"rows": n,
"seconds": seconds,
"peak_rss": rss_bytes(),
"ok": valid,
}
print(json.dumps(result, separators=(",", ":")))
def spawn(name: str) -> dict[str, Any]:
completed = subprocess.run(
[sys.executable, os.path.abspath(__file__), "--worker", name],
check=True,
capture_output=True,
text=True,
timeout=220,
)
output_lines = completed.stdout.strip().splitlines()
if not output_lines:
raise RuntimeError(f"worker {name} produced no result")
return json.loads(output_lines[-1])
# ---------------------------------------------------------------- reporting
def measured_rows(result: dict[str, Any]) -> str:
suffix = " (sampled)" if result["sampled"] else ""
return f"{result['rows']:,}{suffix}"
def build_results_markdown(
results: list[dict[str, Any]],
base_rss: int,
total_wall_seconds: float,
) -> tuple[str, dict[str, Any], dict[str, Any]]:
baseline = next(r for r in results if r["name"] == "naive_autocommit")
baseline_time = float(baseline["scaled_seconds"])
ordered = sorted(results, key=lambda result: result["scaled_seconds"])
fastest = ordered[0]
full_runs = [result for result in results if not result["sampled"]]
leanest = min(
full_runs,
key=lambda result: (result["peak_rss"], result["scaled_seconds"]),
)
lines = [
"# SQLite Bulk Insert Benchmark Results",
"",
(
f"Fresh measurements: **{N:,} target rows**, Python "
f"{platform.python_version()}, SQLite {sqlite3.sqlite_version}, "
f"{platform.platform()}."
),
"",
(
"The timer includes row generation, binding, insertion, and commit. "
"WAL timing also includes `wal_checkpoint(TRUNCATE)`. Each strategy "
"ran in a fresh process against a fresh database."
),
"",
"| Strategy | Measured rows | Time 1M rows (s) | Speedup vs naive | Peak RAM (MB) | RSS over base (MB) |",
"|---|---:|---:|---:|---:|---:|",
]
for result in ordered:
scaled = float(result["scaled_seconds"])
peak_mb = int(result["peak_rss"]) / 1_000_000
delta_mb = (int(result["peak_rss"]) - base_rss) / 1_000_000
time_mark = "*" if result["sampled"] else ""
lines.append(
f"| `{result['name']}` | {measured_rows(result)} | "
f"{scaled:.4f}{time_mark} | {baseline_time / scaled:.2f}x | "
f"{peak_mb:.3f} | {delta_mb:.3f} |"
)
fastest_time = float(fastest["scaled_seconds"])
leanest_peak_mb = int(leanest["peak_rss"]) / 1_000_000
leanest_delta_mb = (int(leanest["peak_rss"]) - base_rss) / 1_000_000
lines.extend(
[
"",
(
f"\\* `naive_autocommit` ran {NAIVE_SAMPLE_N:,} real rows; its "
f"measured time was scaled by {N // NAIVE_SAMPLE_N}. Its RAM "
"reading is measured, not scaled, and it is excluded from the "
"memory-winner decision."
),
"",
(
"`max_tuned_gen` and `set_based_ctes` use "
"`journal_mode=OFF`/`synchronous=OFF`; they are for rebuildable "
"staging loads only."
),
"",
f"Bare-worker peak RSS calibration: **{base_rss / 1_000_000:.3f} MB**.",
f"Total benchmark wall time: **{total_wall_seconds:.3f} s**.",
"",
(
f"**Speed Winner: `{fastest['name']}`** — {fastest_time:.4f} s, "
f"{baseline_time / fastest_time:.2f}x faster than naive."
),
(
f"**Memory Winner: `{leanest['name']}`** — "
f"{leanest_peak_mb:.3f} MB peak RSS, "
f"{leanest_delta_mb:+.3f} MB over the calibration worker."
),
"",
]
)
return "\n".join(lines), fastest, leanest
def definition_of_done_gate(
results: list[dict[str, Any]],
markdown: str,
fastest: dict[str, Any],
leanest: dict[str, Any],
) -> None:
"""Hard-fail unless fresh measurements satisfy the benchmark contract."""
expected = set(STRATEGIES)
actual = {str(result["name"]) for result in results}
if actual != expected:
raise AssertionError(f"strategy mismatch: expected {expected}, got {actual}")
if not all(result["ok"] for result in results):
failed = [result["name"] for result in results if not result["ok"]]
raise AssertionError(f"integrity verification failed: {failed}")
if not all(
float(result["seconds"]) > 0 and int(result["peak_rss"]) > 0
for result in results
):
raise AssertionError("every strategy must have measured positive time and RSS")
for result in results:
required_rows = NAIVE_SAMPLE_N if result["name"] == "naive_autocommit" else N
if result["rows"] != required_rows:
raise AssertionError(f"wrong row count for {result['name']}")
if result["name"] not in markdown:
raise AssertionError(f"RESULTS.md omitted {result['name']}")
baseline = next(r for r in results if r["name"] == "naive_autocommit")
speedup = float(baseline["scaled_seconds"]) / float(fastest["scaled_seconds"])
if speedup < 5.0:
raise AssertionError(f"fastest strategy achieved only {speedup:.2f}x")
if fastest["name"] not in markdown or leanest["name"] not in markdown:
raise AssertionError("winner declaration missing from RESULTS.md")
if "Speed Winner:" not in markdown or "Memory Winner:" not in markdown:
raise AssertionError("RESULTS.md lacks winner lines")
if leanest["name"] == "naive_autocommit":
raise AssertionError("sampled baseline cannot be Memory Winner")
def main() -> None:
total_started = time.perf_counter()
print(
f"SQLite bulk insert benchmark | N={N:,} rows | "
f"Python {platform.python_version()} | SQLite {sqlite3.sqlite_version}"
)
print("Each strategy runs in a fresh isolated worker process.\n")
calibration = spawn("calibration")
results: list[dict[str, Any]] = []
for name in STRATEGIES:
result = spawn(name)
if not result["ok"]:
raise RuntimeError(f"integrity check failed for {name}")
scale = N / int(result["rows"])
result["scaled_seconds"] = float(result["seconds"]) * scale
result["sampled"] = scale != 1.0
results.append(result)
sample_note = " sampled" if result["sampled"] else ""
print(
f"[done] {name}: {result['seconds']:.4f}s, "
f"{result['rows']:,} rows{sample_note}, "
f"peak {result['peak_rss'] / 1_000_000:.3f} MB"
)
total_wall_seconds = time.perf_counter() - total_started
markdown, fastest, leanest = build_results_markdown(
results, int(calibration["peak_rss"]), total_wall_seconds
)
definition_of_done_gate(results, markdown, fastest, leanest)
RESULTS_PATH.write_text(markdown, encoding="utf-8")
print(f"\nWrote {RESULTS_PATH}")
print(
f"Speed Winner: {fastest['name']} "
f"({fastest['scaled_seconds']:.4f}s measured/scaled to 1M)"
)
print(
f"Memory Winner: {leanest['name']} "
f"({leanest['peak_rss'] / 1_000_000:.3f} MB measured peak RSS)"
)
print("Definition-of-Done gate: PASS")
if __name__ == "__main__":
if len(sys.argv) == 3 and sys.argv[1] == "--worker":
worker(sys.argv[2])
elif len(sys.argv) == 1:
main()
else:
raise SystemExit("usage: uv run bench.py [--worker STRATEGY]")
live_final_generation/project-root/sqlite-bulk-benchmark-BUILDER-openai-gpt-5.6-sol.py
+631
View File
@@ -0,0 +1,631 @@
#!/usr/bin/env -S uv run
# /// script
# requires-python = ">=3.11"
# ///
"""SQLite bulk-insert benchmark: elapsed time and process peak RSS.
Run:
uv run sqlite-bulk-benchmark-BUILDER-openai-gpt-5.6-sol.py
Every timed trial executes in a fresh child process against a fresh database.
The deliberately slow autocommit case uses a prefix sample and reports a
clearly marked linear projection to one million rows.
"""
from __future__ import annotations
import argparse
import gc
import itertools
import json
import os
import platform
import random
import sqlite3
import subprocess
import sys
import tempfile
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable, Iterator, Sequence
IDENTITY = "BUILDER-openai-gpt-5.6-sol"
DEFAULT_ROWS = 1_000_000
DEFAULT_BASELINE_ROWS = 10_000
DEFAULT_REPEATS = 2
EXECUTEMANY_BATCH_ROWS = 10_000
MULTI_VALUES_TARGET_ROWS = 500
COLUMN_COUNT = 6
INSERT_ONE = """
INSERT INTO events
(id, account_id, created_at, amount_cents, status, payload)
VALUES (?, ?, ?, ?, ?, ?)
"""
SCHEMA = """
CREATE TABLE events (
id INTEGER PRIMARY KEY,
account_id INTEGER NOT NULL,
created_at INTEGER NOT NULL,
amount_cents INTEGER NOT NULL,
status TEXT NOT NULL CHECK (status IN ('new', 'paid', 'sent', 'void')),
payload TEXT NOT NULL
)
"""
SET_BASED_INSERT = """
WITH RECURSIVE
seq(i) AS (
VALUES(0)
UNION ALL
SELECT i + 1 FROM seq WHERE i + 1 < ?
),
generated(i, mix) AS (
SELECT i, (i * 1103515245 + 12345) % 2147483648
FROM seq
)
INSERT INTO events
(id, account_id, created_at, amount_cents, status, payload)
SELECT
i,
mix % 100003,
1700000000 + i,
(mix % 2000001) - 1000000,
CASE i % 4
WHEN 0 THEN 'new'
WHEN 1 THEN 'paid'
WHEN 2 THEN 'sent'
ELSE 'void'
END,
printf(
'event-%010d-%010x-abcdefghijklmnopqrstuvwxyz0123456789',
i,
mix
)
FROM generated
"""
DIGEST_SQL = """
SELECT
count(*),
sum(id),
sum(account_id),
sum(created_at),
sum(amount_cents),
sum(length(status)),
sum(length(payload)),
min(id),
max(id),
min(payload),
max(payload)
FROM events
"""
@dataclass(frozen=True)
class Strategy:
key: str
label: str
journal_mode: str
synchronous: str
implementation: str
durability: str
sampled: bool = False
STRATEGIES: tuple[Strategy, ...] = (
Strategy(
"naive_autocommit",
"naive autocommit",
"DELETE",
"FULL",
"naive",
"durable",
sampled=True,
),
Strategy(
"one_big_transaction",
"one transaction + execute",
"DELETE",
"FULL",
"one_transaction",
"durable",
),
Strategy(
"executemany_streamed",
"streamed executemany batches",
"DELETE",
"FULL",
"executemany",
"durable",
),
Strategy(
"wal_multi_values",
"WAL/NORMAL + multi-VALUES",
"WAL",
"NORMAL",
"multi_values",
"consistent; latest commit power-loss risk",
),
Strategy(
"off_multi_values",
"OFF/OFF + multi-VALUES",
"OFF",
"OFF",
"multi_values",
"rebuildable staging only",
),
Strategy(
"wal_set_based",
"WAL/NORMAL + INSERT-SELECT",
"WAL",
"NORMAL",
"set_based",
"consistent; latest commit power-loss risk",
),
Strategy(
"off_set_based",
"OFF/OFF + INSERT-SELECT",
"OFF",
"OFF",
"set_based",
"rebuildable staging only",
),
)
STRATEGY_BY_KEY = {strategy.key: strategy for strategy in STRATEGIES}
Row = tuple[int, int, int, int, str, str]
Digest = tuple[int | str, ...]
def make_row(i: int) -> Row:
"""The sole Python definition of benchmark data."""
mix = (i * 1103515245 + 12345) % 2147483648
status = ("new", "paid", "sent", "void")[i % 4]
return (
i,
mix % 100003,
1700000000 + i,
(mix % 2000001) - 1000000,
status,
f"event-{i:010d}-{mix:010x}-abcdefghijklmnopqrstuvwxyz0123456789",
)
def generated_rows(start: int, stop: int) -> Iterator[Row]:
for i in range(start, stop):
yield make_row(i)
def expected_digest(row_count: int) -> Digest:
count = 0
id_sum = account_sum = created_sum = amount_sum = 0
status_length_sum = payload_length_sum = 0
min_payload: str | None = None
max_payload: str | None = None
for row in generated_rows(0, row_count):
row_id, account_id, created_at, amount, status, payload = row
count += 1
id_sum += row_id
account_sum += account_id
created_sum += created_at
amount_sum += amount
status_length_sum += len(status)
payload_length_sum += len(payload)
if min_payload is None or payload < min_payload:
min_payload = payload
if max_payload is None or payload > max_payload:
max_payload = payload
return (
count,
id_sum,
account_sum,
created_sum,
amount_sum,
status_length_sum,
payload_length_sum,
0 if row_count else None,
row_count - 1 if row_count else None,
min_payload,
max_payload,
)
def peak_rss_bytes() -> int:
if os.name == "nt":
# ctypes is standard library. PeakWorkingSetSize is the Windows
# equivalent of process peak RSS.
import ctypes
from ctypes import wintypes
class ProcessMemoryCounters(ctypes.Structure):
_fields_ = (
("cb", wintypes.DWORD),
("PageFaultCount", wintypes.DWORD),
("PeakWorkingSetSize", ctypes.c_size_t),
("WorkingSetSize", ctypes.c_size_t),
("QuotaPeakPagedPoolUsage", ctypes.c_size_t),
("QuotaPagedPoolUsage", ctypes.c_size_t),
("QuotaPeakNonPagedPoolUsage", ctypes.c_size_t),
("QuotaNonPagedPoolUsage", ctypes.c_size_t),
("PagefileUsage", ctypes.c_size_t),
("PeakPagefileUsage", ctypes.c_size_t),
)
counters = ProcessMemoryCounters()
counters.cb = ctypes.sizeof(counters)
get_current_process = ctypes.windll.kernel32.GetCurrentProcess
get_current_process.restype = wintypes.HANDLE
ok = ctypes.windll.psapi.GetProcessMemoryInfo(
get_current_process(), ctypes.byref(counters), counters.cb
)
if not ok:
raise ctypes.WinError()
return int(counters.PeakWorkingSetSize)
import resource
raw = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
# macOS reports bytes; Linux and the BSDs report KiB.
if sys.platform == "darwin":
return int(raw)
return int(raw) * 1024
def setup_connection(db_path: Path, strategy: Strategy) -> sqlite3.Connection:
connection = sqlite3.connect(db_path, isolation_level=None, timeout=60.0)
connection.execute("PRAGMA page_size=4096")
actual_journal = connection.execute(
f"PRAGMA journal_mode={strategy.journal_mode}"
).fetchone()[0]
if str(actual_journal).upper() != strategy.journal_mode:
raise RuntimeError(
f"requested journal_mode={strategy.journal_mode}, got {actual_journal}"
)
connection.execute(f"PRAGMA synchronous={strategy.synchronous}")
# Common controls: bounded cache, no mmap RSS ambiguity, disk-backed temp
# structures, and spill enabled. These are identical for every strategy.
connection.execute("PRAGMA cache_size=-2048")
connection.execute("PRAGMA cache_spill=ON")
connection.execute("PRAGMA temp_store=FILE")
connection.execute("PRAGMA mmap_size=0")
connection.execute("PRAGMA foreign_keys=ON")
connection.execute(SCHEMA)
return connection
def finish_commit(connection: sqlite3.Connection, strategy: Strategy) -> None:
connection.commit()
# Include WAL's eventual write-back cost, rather than declaring victory
# while the complete database still resides in a large sidecar file.
if strategy.journal_mode == "WAL":
result = connection.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()
if result[0] != 0:
raise RuntimeError(f"WAL checkpoint was busy: {result!r}")
def insert_naive(connection: sqlite3.Connection, row_count: int, strategy: Strategy) -> None:
# isolation_level=None makes every statement its own transaction.
for row in generated_rows(0, row_count):
connection.execute(INSERT_ONE, row)
def insert_one_transaction(
connection: sqlite3.Connection, row_count: int, strategy: Strategy
) -> None:
connection.execute("BEGIN IMMEDIATE")
for row in generated_rows(0, row_count):
connection.execute(INSERT_ONE, row)
finish_commit(connection, strategy)
def insert_executemany(
connection: sqlite3.Connection, row_count: int, strategy: Strategy
) -> None:
connection.execute("BEGIN IMMEDIATE")
for start in range(0, row_count, EXECUTEMANY_BATCH_ROWS):
stop = min(start + EXECUTEMANY_BATCH_ROWS, row_count)
# An iterator, not a list: batching does not materialize input rows.
connection.executemany(INSERT_ONE, generated_rows(start, stop))
finish_commit(connection, strategy)
def insert_multi_values(
connection: sqlite3.Connection, row_count: int, strategy: Strategy
) -> None:
try:
variable_limit = connection.getlimit(sqlite3.SQLITE_LIMIT_VARIABLE_NUMBER)
except AttributeError:
variable_limit = 999
batch_rows = max(
1, min(MULTI_VALUES_TARGET_ROWS, variable_limit // COLUMN_COUNT)
)
placeholder = "(" + ",".join("?" for _ in range(COLUMN_COUNT)) + ")"
connection.execute("BEGIN IMMEDIATE")
for start in range(0, row_count, batch_rows):
stop = min(start + batch_rows, row_count)
batch = list(generated_rows(start, stop))
statement = (
"INSERT INTO events "
"(id, account_id, created_at, amount_cents, status, payload) VALUES "
+ ",".join(itertools.repeat(placeholder, len(batch)))
)
bindings = [value for row in batch for value in row]
connection.execute(statement, bindings)
finish_commit(connection, strategy)
def insert_set_based(
connection: sqlite3.Connection, row_count: int, strategy: Strategy
) -> None:
connection.execute("BEGIN IMMEDIATE")
connection.execute(SET_BASED_INSERT, (row_count,))
finish_commit(connection, strategy)
IMPLEMENTATIONS = {
"naive": insert_naive,
"one_transaction": insert_one_transaction,
"executemany": insert_executemany,
"multi_values": insert_multi_values,
"set_based": insert_set_based,
}
def worker(strategy_key: str, db_path: Path, row_count: int) -> int:
strategy = STRATEGY_BY_KEY[strategy_key]
if db_path.exists():
db_path.unlink()
connection = setup_connection(db_path, strategy)
implementation = IMPLEMENTATIONS[strategy.implementation]
gc_was_enabled = gc.isenabled()
gc.disable()
started_ns = time.perf_counter_ns()
try:
implementation(connection, row_count, strategy)
except BaseException:
if connection.in_transaction:
connection.rollback()
raise
finally:
if gc_was_enabled:
gc.enable()
elapsed_ns = time.perf_counter_ns() - started_ns
# Snapshot peak before the untimed validation query.
rss_bytes = peak_rss_bytes()
digest = tuple(connection.execute(DIGEST_SQL).fetchone())
quick_check = connection.execute("PRAGMA quick_check").fetchone()[0]
connection.close()
print(
json.dumps(
{
"strategy": strategy.key,
"rows": row_count,
"elapsed_seconds": elapsed_ns / 1_000_000_000,
"peak_rss_bytes": rss_bytes,
"digest": digest,
"quick_check": quick_check,
},
separators=(",", ":"),
)
)
return 0
def format_seconds_list(results: Sequence[dict[str, object]]) -> str:
return ",".join(f"{float(result['elapsed_seconds']):.3f}" for result in results)
def format_rss_list(results: Sequence[dict[str, object]]) -> str:
mib = 1024 * 1024
return ",".join(f"{int(result['peak_rss_bytes']) / mib:.1f}" for result in results)
def median(values: Sequence[float]) -> float:
ordered = sorted(values)
middle = len(ordered) // 2
if len(ordered) % 2:
return ordered[middle]
return (ordered[middle - 1] + ordered[middle]) / 2
def print_table(
grouped: dict[str, list[dict[str, object]]],
total_rows: int,
expected_by_count: dict[int, Digest],
) -> tuple[Strategy, Strategy]:
headers = (
"Strategy",
"Mode",
"Rows/run",
"Trial time(s)",
"Time@1M(s)",
"Peak RAM/run MiB",
"Data",
)
rows: list[tuple[str, ...]] = []
normalized_times: dict[str, float] = {}
peak_by_strategy: dict[str, int] = {}
for strategy in STRATEGIES:
results = grouped[strategy.key]
elapsed = [float(result["elapsed_seconds"]) for result in results]
row_count = int(results[0]["rows"])
projected = median(elapsed) * total_rows / row_count
normalized_times[strategy.key] = projected
peak_by_strategy[strategy.key] = max(
int(result["peak_rss_bytes"]) for result in results
)
verified = all(
tuple(result["digest"]) == expected_by_count[int(result["rows"])]
and result["quick_check"] == "ok"
for result in results
)
projection_marker = "*" if strategy.sampled else ""
rows.append(
(
strategy.label,
f"{strategy.journal_mode}/{strategy.synchronous}",
f"{row_count:,}",
format_seconds_list(results),
f"{projected:.3f}{projection_marker}",
format_rss_list(results),
"OK" if verified else "FAIL",
)
)
widths = [len(header) for header in headers]
for row in rows:
for index, value in enumerate(row):
widths[index] = max(widths[index], len(value))
def render(row: Sequence[str]) -> str:
return " | ".join(value.ljust(widths[i]) for i, value in enumerate(row))
print(render(headers))
print("-+-".join("-" * width for width in widths))
for row in rows:
print(render(row))
speed_key = min(normalized_times, key=normalized_times.get)
# The sampled baseline is not eligible for the memory title: its peak RSS
# was measured honestly but it did not execute the full row count.
full_keys = [strategy.key for strategy in STRATEGIES if not strategy.sampled]
memory_key = min(full_keys, key=peak_by_strategy.get)
return STRATEGY_BY_KEY[speed_key], STRATEGY_BY_KEY[memory_key]
def orchestrator(args: argparse.Namespace) -> int:
if args.rows <= 0 or args.baseline_rows <= 0 or args.repeats <= 0:
raise SystemExit("--rows, --baseline-rows, and --repeats must be positive")
baseline_rows = min(args.baseline_rows, args.rows)
print("SQLite bulk insert benchmark")
print(f"Identity: {IDENTITY}")
print(f"Python: {platform.python_version()} ({sys.executable})")
print(f"SQLite: {sqlite3.sqlite_version}")
print(f"Platform: {platform.platform()}")
print(f"Rows: {args.rows:,}")
print(
f"Autocommit sample: {baseline_rows:,} rows; its Time@1M is a linear projection"
)
print(f"Full-strategy trials: {args.repeats}")
print(
"Timed interval: first BEGIN/INSERT through COMMIT; WAL trials also include "
"TRUNCATE checkpoint"
)
print("Peak RAM: per-worker process ru_maxrss; every trial uses a fresh process")
print()
print("Computing independent expected data digests...")
expected_by_count = {
baseline_rows: expected_digest(baseline_rows),
args.rows: expected_digest(args.rows),
}
jobs: list[tuple[Strategy, int, int]] = []
for strategy in STRATEGIES:
trial_count = 1 if strategy.sampled else args.repeats
row_count = baseline_rows if strategy.sampled else args.rows
for trial in range(1, trial_count + 1):
jobs.append((strategy, trial, row_count))
random.Random(0x5A17E).shuffle(jobs)
grouped: dict[str, list[dict[str, object]]] = {
strategy.key: [] for strategy in STRATEGIES
}
script_path = Path(__file__).resolve()
child_environment = os.environ.copy()
child_environment["PYTHONHASHSEED"] = "0"
with tempfile.TemporaryDirectory(prefix=f"sqlite-bench-{IDENTITY}-") as temp_name:
temp_dir = Path(temp_name)
for job_number, (strategy, trial, row_count) in enumerate(jobs, start=1):
db_path = temp_dir / f"{IDENTITY}-{strategy.key}-trial-{trial}.sqlite3"
print(
f"[{job_number:02d}/{len(jobs):02d}] {strategy.label}, "
f"trial {trial}, {row_count:,} rows...",
flush=True,
)
completed = subprocess.run(
[
sys.executable,
str(script_path),
"--worker",
strategy.key,
"--db",
str(db_path),
"--worker-rows",
str(row_count),
],
check=False,
capture_output=True,
text=True,
env=child_environment,
timeout=args.worker_timeout,
)
if completed.returncode != 0:
raise RuntimeError(
f"worker failed for {strategy.key}:\n"
f"stdout:\n{completed.stdout}\n"
f"stderr:\n{completed.stderr}"
)
result = json.loads(completed.stdout.strip().splitlines()[-1])
grouped[strategy.key].append(result)
for results in grouped.values():
results.sort(key=lambda item: float(item["elapsed_seconds"]))
print("\nResults")
speed_winner, memory_winner = print_table(
grouped, args.rows, expected_by_count
)
print(
"\n* naive autocommit Time@1M is measured-time × "
f"{args.rows:,}/{baseline_rows:,}; its RAM is measured, not projected."
)
print(
"OFF/OFF rows are semantically identical but the database is rebuildable-only; "
"a crash can corrupt it."
)
print(
"INSERT-SELECT is applicable when rows can be generated in SQL or selected "
"from an attached SQLite source."
)
print(f"\nSpeed Winner: {speed_winner.label}")
print(f"Memory Winner: {memory_winner.label}")
return 0
def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--rows", type=int, default=DEFAULT_ROWS)
parser.add_argument("--baseline-rows", type=int, default=DEFAULT_BASELINE_ROWS)
parser.add_argument("--repeats", type=int, default=DEFAULT_REPEATS)
parser.add_argument("--worker-timeout", type=float, default=180.0)
parser.add_argument("--worker", choices=tuple(STRATEGY_BY_KEY))
parser.add_argument("--db", type=Path)
parser.add_argument("--worker-rows", type=int)
args = parser.parse_args(argv)
if args.worker and (args.db is None or args.worker_rows is None):
parser.error("--worker requires --db and --worker-rows")
return args
def main(argv: Sequence[str] | None = None) -> int:
args = parse_args(argv)
if args.worker:
return worker(args.worker, args.db, args.worker_rows)
return orchestrator(args)
if __name__ == "__main__":
raise SystemExit(main())