Hey @djjones,
The local-model harness idea from the NC discussions is now a full spec draft, revised this week with Dave W’s direction. Posting it here so you have it ahead of build planning, along with recommendations for keeping the build as small as possible out of the gate. Nothing here is ratified yet — the proposal decision is still yours and Dave W’s to make together.
What it is, in three sentences
A small process-orchestration layer in the BrainDrive TS runtime that runs defined multi-stage processes (interview → spec → plan → journal) as explicit stages with code-enforced sequencing, validation gates, and one retry — while every stage’s behavior (voice, questions, criteria) stays in plain-English, owner-editable text files. It attacks the documented small-model failure mode (can’t hold a long process + owner instructions in one open-ended context) by narrowing each model call to one stage. The pattern is Mellea’s (decompose → externalize control flow → validate → retry), built natively in TypeScript — we steal the pattern, not the Python library.
What changed in this week’s revision
- Backend-agnostic by design. No
if (isLocalModel)fork anywhere. The guarantees (journal always written, artifacts routed correctly, stages sequenced unless the owner overrides) are system guarantees, not a small-model crutch. What varies per backend is a temporary rollout flag (local lane first, planned death) and policy config (retry budgets, validation strictness) — never the semantics. - “No observable change to the default flow” became “no regression.” The GLM-5.2 flag-flip is gated on evidence: same before/after gates, no regression on pass rates, latency, cost, or feel. Regression on the default lane is the release blocker; while the flag is off, the default lane is byte-identical to today.
- Hardware floor: the structured-stage win must land on a model that runs on a 16 GB RAM MacBook or better (~7–12B @ Q4). A win only Spark-class hardware can run doesn’t validate the bet. (Note: this creates a tension with the fine-tune project’s current base-model order — dense Gemma 4 31B doesn’t fit the floor. Flagged in that project’s open questions, not yet resolved.)
Build-plan recommendations (simplicity-first)
Dave W’s directive for the build: as simple as possible out of the gate, start small, and only add complexity once we’ve proven the small version won’t work. Concretely, here’s the ladder I’d propose — each rung has an explicit entry condition, and we don’t climb without evidence:
- Phase A stays scripts, not runtime code. Dave W + agents prototype the harness as plain scripts against Ollama on the Spark. Zero product risk, and the outputs (eval deltas, traces) are the evidence that decides whether Phase B happens at all. No runtime work from you until Phase A shows a gate-pass delta.
- Hardcode the one process. V1 registers exactly one process as an ordered array of stages in code. No process-definition DSL, no registry abstraction, no config format for process shapes. Entry condition for generalizing: a second real process exists and the copy-paste actually hurts.
- Deterministic checks only, at first. Start with “artifact exists, required sections present” + one retry-with-feedback. Skip the model-judged check in the first cut — it’s the most complex, most expensive gate, and we don’t know yet whether the cheap gates alone move pass rates. Entry condition for adding the judge: deterministic gates + retry demonstrably plateau below the target.
- One fixed retry. No adaptive retry policies, no per-stage tuning. The per-backend policy config can literally be constants in V1.
- The state machine stays boring. Plain TS, hand-rolled, no workflow-engine dependency (already a hard constraint in the spec — holding the line on it is the single biggest simplicity lever).
- Traces are append-only JSONL. No dashboard, no metrics surface. We read them with scripts until that hurts.
- Rollout flag = one config read. Not a feature-flag system. A single boolean per backend lane, checked in one place, with a property test that flag-off lanes never enter harness code.
- Prove it at the floor first. Run the eval on the smallest model that fits the 16 GB MacBook floor before touching anything bigger — if the harness only helps large-ish local models, that’s an abort signal worth having early.
Everything already excluded in the spec stays excluded: mixed per-stage routing, multiple processes, Tier-2 config UI, any plugin surface.
The two things worth deciding before any code: the quantitative success/abort bar (what gate-pass delta validates the bet — spec Q4), and the regression bar for the eventual GLM-5.2 flag-flip (spec Q8, yours to sign off since it touches the production flow). If those are defined first, every rung of the ladder above has a cheap, unambiguous “did it work” answer.
Full spec draft below. It lives at projects/active/foundation/local-model-fine-tune/local-harness-spec-draft.md in the Library.
Full spec: Local-Model Process Harness (DRAFT)
Spec: Local-Model Process Harness (DRAFT — FOR DISCUSSION)
Purpose: Builder-facing BrainDrive product/engineering spec. This is for Dave W, Dave J, AI coding agents, QA/harness agents, and future maintainers. It is not an owner-facing BrainDrive goal/spec artifact.
Status: DRAFT. The underlying decision (local-harness-proposal.md) is not ratified — every user story is Open, nothing here is safe to build against yet. This draft exists so the discussion has a concrete shape to react to.
Save to:BrainDrive-Library/projects/active/foundation/local-model-fine-tune/local-harness-spec-draft.md
Generated from: DW ↔ Claude chat session, 2026-07-18 (Mellea discussion → text-first refinement → content-story argument)
Revised: 2026-07-21 (DW direction) — the harness is now backend-agnostic by design: no local-only toggle in the architecture. Rollout is staged (local lane first) behind a temporary flag, and enabling it for sophisticated models is gated on no regression, not treated as a separate future decision. “Default flow must not change in any observable way” is replaced by “default flow must not regress.”
Overview
What We’re Building
A small process-orchestration layer (“harness”) in the BrainDrive TS runtime, backend-agnostic by design, that runs defined multi-stage processes (interview → spec → plan → journal) as explicit stages with code-enforced sequencing, validation gates, and retry — while every stage’s behavior (voice, questions, criteria, what “good” looks like) stays in plain-English, owner-editable text files.
The harness’s core guarantees (journal always written, artifacts routed per the memory architecture, stages sequenced unless the owner says otherwise) are system guarantees, not a small-model crutch — they should hold for every model. There is no if (isLocalModel) fork in the architecture; what varies per backend is rollout (a temporary flag, local lane first) and policy config (retry budgets, validation strictness — e.g. cheaper retry policy on metered credits), never the semantics.
The pattern is Mellea’s (decompose → externalize control flow → validate → retry), built natively in TypeScript. We steal the pattern, not the Python library.
Why It Matters
Small local models fail BrainDrive’s process work not primarily from lack of raw capability but because they can’t reliably hold a long multi-step process plus arbitrary owner instructions in one open-ended context — the exact failure mode the 10-model gauntlet, the Nemotron adherence tests, and D289 (“residual flake is a weights property”) documented. Narrowing each model call to one stage and moving sequence/guarantees into the runtime attacks the failure mode the fine-tune (T-764) alone can’t fully fix; the two compose.
Product payoff: a credible “runs fully locally” mode owners can experience, which is also the teaching mechanism for the local-vs-cloud layers nuance that posts alone haven’t landed. Content payoff: the build itself — gauntlet receipts, failure data, harness deltas, fine-tune runs — is a build-in-public series for local-AI communities (per D3, content pulls this lane’s work forward).
Target Audience And Stakeholders
| Audience | Role / Need |
|---|---|
| BrainDrive owner persona(s) affected | Privacy-motivated Katie variants who opt into local models (Tier-2 decision per progressive disclosure). Default-flow owners: unaffected. |
| Builder / implementation owner | Phase A experiments: Dave W + AI agents on the DGX Spark. Runtime integration: Dave J (only after Phase A evidence). |
| QA / harness audience | Starter-pack testing harness (Katie A/B/C, d23-calibrated gates) judges stage outputs and before/after gate-pass deltas. |
| Operator / maintainer | Dave W (experiments, content series); Dave J (runtime code once integrated). |
Persona Baseline And Surface Starting Positions
N/A in full — this is model-layer/foundation work, not a page. The relevant persona fact: the harness serves owners who have made the Tier-2 choice to run a local model and accept its tradeoffs; it must never surface at Tier 0 (model identity = progressive disclosure).
Success Definition
When this work is complete, the target audience will be able to:
- Run a defined BrainDrive process end-to-end on a local model (Ollama backend) with zero sequencing failures — no skipped journal, no plan-before-spec — across the harness eval set.
- See at least one gauntlet-failing local model (Gemma-class or the T-764 fine-tune) pass structured stages (spec drafting, journal capture) it previously failed, measured on the frozen gauntlet / Katie A/B/C gates.
- Hardware floor (DW, 2026-07-21): achieve criterion 2 with a model that runs on a 16 GB RAM MacBook or better — in practice roughly a 7–12B model at 4-bit quantization, leaving headroom for the OS and BrainDrive itself alongside the weights + KV cache. The harness bet isn’t proven by a model only Spark-class hardware can run; the target owner has a stock MacBook.
- Change stage behavior (tone, questions, criteria) by editing a text file only — verified by a standing owner-edit eval.
- (Content) Publish a receipts-backed build-in-public series from the experiment data.
The work is not successful if:
- The default GLM-5.2 flow regresses — gate-pass rates, latency, cost, or conversational feel. (Observable change is acceptable if neutral or better; regression on any of these is the blocker. While the rollout flag is off for the default lane, its flow is untouched entirely.)
- An owner instruction (“skip the spec”) is overridden by a gate — a don’t-force-don’t-prevent violation.
- A behavior change ends up requiring a code change (bright-line violation — judgment leaked into the harness).
- Local-lane gate-pass rates don’t improve over the un-harnessed baseline (the bet failed; see abort criteria, Open Questions).
Definition of Done (V1)
When V1 is “done done,” it can:
- Execute one registered process (interview → spec → plan → journal) on a local backend: stages sequenced by the runtime, each stage’s behavior loaded from an owner-editable text file, per-stage model calls with narrowed context.
- Enforce system guarantees as post-stage hooks (journal always written; artifacts land where the memory architecture routes them) with deterministic validation (artifact exists, required sections present) + one model-judged check + one retry-with-feedback before surfacing to the owner.
- Honor owner overrides: any stage skippable/reorderable by owner say-so in-conversation; the harness records the override rather than blocking it.
- Produce evidence: per-run trace of stages, gate results, retries, and model used — feeding both the testing harness and the content series.
Explicitly NOT required for done-done (separate buckets — name where each lives instead):
- Harness enabled on the default GLM-5.2 flow → not a separate architectural decision anymore, but a rollout step gated on evidence: after local-lane proof, run the same before/after gates on GLM-5.2; no regression (pass rates, latency, feel, cost) → enable for all models and retire the flag. Regression → the flag stays, and per-backend enablement becomes the fallback posture.
- Per-stage mixed routing (local stage + cloud stage in one run) → V-next of this spec; V1 is single local backend per run.
- The fine-tuned model itself → [[T-764]] (plan.md in this project).
- New owner-facing settings UI for harness config → Tier-2 config surface, [TBD: to be decided during build plan].
- The content posts → marketing runway (idea inbox → series outline).
Product Behavior
User / System Experience
An owner who has opted into a local model starts a process (“help me define my fitness goal”) and experiences the same conversational flow as today — the harness is invisible plumbing. Under the hood, the runtime walks the process stage by stage: each stage is a fresh, narrow model call carrying only that stage’s text instructions + the artifacts it needs. Between stages, the runtime validates the output; a failed check retries once with the failure fed back before anything reaches the owner. Stage artifacts land per the memory architecture. If the owner redirects (“I don’t need a formal spec”), the flow follows the owner; the harness logs the deviation.
For DW/QA, every run leaves a trace: stages run, checks passed/failed, retries, tokens, model — the raw material for both eval and content.
Primary Flows
- Harnessed process run: owner starts a process on a local backend → runtime sequences stages, narrow call per stage → validation + retry per transition → artifacts written by hooks → process completes with zero sequencing failures.
- Owner override: owner says skip/reorder/stop mid-process → harness complies immediately, records the override in the run trace → downstream guarantees adjust (no spec means the plan stage is told there is no spec, not blocked).
- Behavior customization: owner (or DW) edits a stage’s text file → next run reflects the change, no code touched.
- Eval run: testing harness replays the Katie A/B/C scenario set through the harnessed flow → gate-pass delta vs. un-harnessed baseline reported.
Secondary Flows
- Validation exhausted: deterministic check + retry both fail → surface the best attempt to the owner with a plain-language note about what’s weak, never a silent bad artifact and never a dead end.
- Local backend dies mid-process: run trace preserves completed stages; process resumes at the failed stage, not from scratch.
UX / Trust Bar
- The harness must be invisible in the happy path — no wizard feel, no “step 2 of 4” chrome imposed on conversation. [TODO: needs clarification — whether any progress affordance is wanted at Tier 2.]
- Owner overrides must feel like being heard, not like an error path.
- “The file says X but it does Y” must never happen: if code constrains something, the constraint is sequencing/guarantee only, never behavior.
User Stories
All stories Open — the underlying decision is unratified.
US-1: Harnessed process run on a local model - Open
As a local-model owner, I want defined processes to run reliably stage-by-stage so that a small model gives me the same dependable process experience a frontier model does.
Details - source, steps, acceptance criteria
Source: 2026-07-18 chat session; Mellea pattern; gauntlet/adherence evidence.
Acceptance Criteria (Given/When/Then):
Given a local Ollama backend and the interview→spec→plan→journal process
When the owner completes the flow across the harness eval set
Then sequencing failures (skipped journal, missing spec-before-plan) = 0
And each stage's model call contains only that stage's instructions and required artifacts
Status: Open
US-2: Owner override always wins - Open
As an owner, I want to skip or reorder any stage by saying so, so that the system’s guarantees never become constraints on me.
Details - source, steps, acceptance criteria
Source: don’t-force-don’t-prevent; proposal doc Open Question 1.
Acceptance Criteria:
Given a harnessed process run
When the owner says "skip the spec, just help me plan"
Then the spec stage is skipped without argument or blocking
And the run trace records the override
And the plan stage receives "no spec exists" as context rather than failing a gate
Status: Open
US-3: Behavior is text-editable - Open
As an owner (or DW tuning the product), I want to change how a stage behaves by editing its plain-English file, so that the ownership story (“everything is readable and editable”) survives the code layer.
Details - source, steps, acceptance criteria
Source: bright-line test, 2026-07-18 session; starter-pack principles (owner-editable, no “DO NOT MODIFY”).
Acceptance Criteria:
Given a Katie-plausible behavior change (e.g. "make the interview less formal", "add a budget question")
When it is applied as a text-file edit only
Then the next run reflects the change with no code modified
Status: Open
US-4: Validation gates with retry - Open
As the system, I want each stage transition checked (deterministic first, model-judged second) with one retry-with-feedback, so that bad artifacts are caught where they’re cheap instead of surfacing to the owner.
Details - source, steps, acceptance criteria
Source: Mellea pattern; d23 calibration (11 auto-gate criteria).
Acceptance Criteria:
Given a stage output missing a required section
When the transition check runs
Then the deterministic check fails, the failure is fed back, and the stage retries once
And if the retry also fails, the best attempt surfaces to the owner with a plain-language note
Status: Open
US-5: Run traces as evidence - Open
As DW/QA, I want every harnessed run to emit a trace (stages, checks, retries, model, tokens) so that eval deltas and content receipts come from ground truth, not recollection.
Details - source, steps, acceptance criteria
Acceptance Criteria:
Given any harnessed run (success or failure)
When it ends
Then a run trace exists recording stage sequence, gate results, retries, overrides, and model identity
Status: Open
Requirements
Functional Requirements
- Process registry: a process = ordered stages, each with an artifact contract and a pointer to its behavior text file. V1 registers exactly one process (interview → spec → plan → journal).
- Harness code is backend-agnostic — no backend-conditional branch anywhere in harness logic. A temporary rollout flag (per lane, with a planned death once no-regression evidence lands) controls which backends run harnessed; V1 ships with it on for the local lane only.
- Per-backend policy config (retry budget, validation strictness) — config, not code fork; e.g. metered-credit backends may get a tighter retry budget than free local backends.
- Per-stage model calls carry narrowed context: stage instructions + required upstream artifacts only.
- Post-stage hooks perform guaranteed writes (journal, artifact placement per memory architecture).
- Transition validation: deterministic checks (artifact exists, sections present) → model-judged check against stage criteria → max one retry-with-feedback.
- Owner override handling per US-2 (skip/reorder/stop honored and recorded).
- Run-trace emission per US-5.
AI / Model / Tool Behavior
- Stage behavior files are plain English, owner-editable, no protected blocks (starter-pack principles).
- Model-judged checks reuse the d23-calibrated gate criteria where applicable — eval and runtime converge on one definition of “good.” [TBD: which of the 11 criteria are runtime-suitable vs. eval-only.]
- Judgment never lives in harness code: code may check presence/structure deterministically and may invoke a text-defined judge; it may not embed criteria, questions, or tone.
- Fallback: validation exhausted → surface best attempt + plain-language weakness note (never silent failure, never a block).
Data, Memory, And Artifact Contracts
| Data / Artifact | Source | Owner | Read/Write Rules | Retention / Migration |
|---|---|---|---|---|
| Process definition (stages, order, artifact contracts) | Runtime code (V1: hardcoded registry) | Dave J / runtime | Sequencing only; no behavior content | Versioned with the runtime |
| Stage behavior files | Product text files | Owner-editable | Plain English; read fresh each run | Owner’s install |
| Stage artifacts (spec, plan, journal entries) | Stage outputs via hooks | Owner | Placed per memory-architecture.md routing — same homes as the un-harnessed flow |
Owner’s memory, unchanged |
| Run traces | Harness | System/QA | Append-only; never contains owner-content beyond artifact refs [TBD: exact redaction rule] | [TBD: retention window] |
Interface / UX Requirements
- No new Tier-0 UI. Harness state never surfaces in the default chat chrome.
- [TBD: Tier-2 config surface for enabling/inspecting the harness — build plan.]
Observability / Evidence Requirements
- Run trace per US-5 is the acceptance evidence substrate.
- A failing run preserves its trace + all stage attempts (including rejected retries — these are DPO-candidate data for T-764 and content material).
- Eval reports: gate-pass rate harnessed vs. un-harnessed, per stage, per model.
Scope
Work Type
- Prototype — Phase A: DW-side harness experiments on the DGX Spark (outside the product runtime), generating eval + content
- V1 — Phase B: runtime integration behind the local backend, per Definition of Done
- Production hardening → after V1 evidence
Implementation Location / Ownership
- Model/provider layer — rollout flag + per-backend policy config (V1: flag on for local lane)
- BrainDrive core/foundation — process registry + hooks in the TS runtime (Phase B)
- Testing / harness / factory — eval integration with the starter-pack testing harness
- External repo: experiment scaffolding may live in ModelMatch ([[T-761]]) with the fine-tune pipeline [TBD]
Included
- One registered process, one local backend at a time, validation + retry, override handling, run traces, eval integration.
Explicitly Excluded
- Enabling the harness on the default GLM-5.2 flow in V1 — the flag stays off for the default lane until the no-regression gates pass (see Definition of Done). Refactoring the default flow “while we’re in there” stays excluded outright.
- Mixed per-stage routing (local + cloud in one run) → V-next.
- A plugin/extension surface — this is internal process definition, not third-party code. NO plugins (retired Core stays retired).
- Adopting Mellea (Python) as a dependency → pattern only.
- Generalizing to owner-defined custom processes → future; V1 processes are product-defined.
Future Versions / Deepenings
- Per-stage model routing (cheap local model on structured stages, bigger model on interview).
- Additional registered processes as their shapes stabilize.
- Flag retirement — harness on for all models once the GLM-5.2 no-regression gates pass; this is the intended end state, not a maybe.
Invariants And Edge Cases
Properties That Must Always Hold
- Rollout-flag invariance: a lane whose flag is off runs byte-identical to today’s flow; a lane whose flag is on must show no regression vs. its un-harnessed baseline before the flag flips in production.
- Owner-override supremacy: no gate ever blocks an explicit owner instruction.
- Bright line: behavior changes require only text edits; code changes are sequencing/guarantee changes only.
- Model identity stays progressive-disclosure compliant — the harness never surfaces model names at Tier 0.
- No gauntlet contamination: frozen gauntlet eval set never appears in stage instructions, retries, or training exports (extends the project’s no-cheating contract).
- Stage artifacts land in the same memory-architecture homes as un-harnessed runs — no parallel artifact universe.
Edge Cases To Test
- Owner overrides at every stage boundary (skip first stage, skip last, stop mid-retry).
- Stage behavior file missing or owner-edited into contradiction with the artifact contract.
- Local backend timeout/death mid-stage; resume behavior.
- Validation infinite-loop guard: retry cap respected when a model can never pass a gate.
- Very long interview stage exceeding local context window.
- Owner content containing text that looks like gate/override instructions (prompt-injection-shaped input).
Failure Modes
| Scenario | Expected Behavior |
|---|---|
| Stage fails deterministic check twice | Best attempt surfaces + plain-language weakness note; run trace marks the gate failure |
| Local model can’t complete a stage at all | Process pauses resumably; owner told plainly which stage struggled and their options |
| Behavior file edit breaks a stage | Harness reports which file + what the artifact contract expects — in plain English, not a stack trace |
| Backend dies mid-process | Completed stages preserved; resume at failed stage |
| Owner rejects an artifact post-gate | Owner edit/redo wins; gates validate system output, never owner judgment |
Technical Context
Existing System Context
- Current behavior: TS runtime + React web client, chat-only; processes run as text instructions inside one conversational flow; GLM-5.2 default (D61). No local-model support in the shipped product yet ([[T-685]] deferred to V1.1+).
- Known gaps: small-model process/adherence failures per the gauntlet, Nemotron tests, D289.
- Related specs/docs: local-harness-proposal.md, plan.md, data-recipe.md,
braindrive/architecture/memory-architecture.md, starter-pack testing harness project. - Related decisions: D1/D2/D3 (this project), D61 (GLM default), D289 (weights property), D257 (no in-product execution apps).
Integration Points
- Model/provider layer (backend detection), Ollama, the agent loop, memory architecture write routing, starter-pack testing harness (eval), ModelMatch repo (experiment scaffolding, TBD), T-764 fine-tune pipeline (retry archives → DPO candidates).
Hard Constraints
- TypeScript-native — no Python dependency in the product.
- Backend-agnostic architecture; V1 rollout is local lane only (flag), and the flag’s removal is gated on GLM-5.2 no-regression evidence.
- Owner-editability of behavior files is non-negotiable (starter-pack principles).
- Never marketed against the 2-min install proof — local mode is a different claim for a different audience.
Build Workflow Inputs
- Sequencing constraint: Phase A (DW Spark experiments) precedes any DJ runtime work; Phase B starts only after a local model passes structured-stage gates in experiments.
- Phase A can prototype the harness as scripts against Ollama on the Spark — same pattern, zero product risk — and its outputs are content + eval data regardless of Phase B’s fate.
- Candidate implementation inputs: reuse gate criteria/scenario assets from the starter-pack testing harness; keep the state machine boring (no workflow-engine dependency).
Test Strategy
Test Levels Required
- Unit — state machine transitions, deterministic validators, override handling
- Integration — Ollama backend, artifact placement, resume
- Property-based — invariants above (esp. rollout-flag invariance, override supremacy)
- Harness / AI judge — Katie A/B/C scenario set, gate-pass deltas, d23 criteria
- Regression — flag-off lanes byte-identical to today; harnessed GLM-5.2 runs (pre-flag-flip) show no regression on pass rates, latency, cost, or feel vs. the un-harnessed baseline
- Human review — DW judges conversational feel (no wizard-ness) on real runs
Verification Approach
- Automated verification: eval runs comparing harnessed vs. un-harnessed local-model gate-pass rates; owner-edit eval (US-3) as a standing check.
- Harness verification: starter-pack testing harness replays; frozen gauntlet for model-capability claims.
- Human verification: DW reviews run traces + transcripts for trust/feel; DJ reviews runtime integration.
- Production/runtime monitoring: run traces; [TBD: aggregate metrics surface].
Ground-Truth Verification (Mandatory — [[planning-system]] D75)
- Ground truth = run traces + artifacts on disk + live process runs on the Spark/product — not spec text, not config, not anyone’s assurance.
- Name the layer on every claim: experiment scripts → runtime code → merged → live install. “Passes on the Spark” ≠ “passes in the product.”
- Model-capability claims (esp. anything destined for a public post) cite frozen-gauntlet or harness runs with traces attached.
Acceptance Evidence
- Eval report: gate-pass delta table (harnessed vs. baseline, per stage, per model) from harness runs.
- At least one passing model in the eval set fits the 16 GB MacBook floor (quantized footprint verified, not assumed — an actual run on 16 GB-class hardware or a measured memory ceiling, not “should fit”).
- Run traces for the eval set, archived.
- Owner-edit eval passing (text-only behavior change lands).
- Behavioral-diff evidence that the default flow is untouched.
Baseline / Regression Impact
- Existing user flows that must keep working: the entire default GLM-5.2 experience — untouched while its flag is off, and demonstrably non-regressed before the flag flips. Regression on the default lane is the release blocker.
- Global properties affected: progressive disclosure (model identity), memory-architecture write routing, don’t-force-don’t-prevent.
Security, Privacy, And Trust Considerations
Risk Level
- Medium — handles owner input and memory writes; no new network surface (local backend), no auth/payment.
Threat / Trust Assessment
- User input: owner conversation + owner-edited behavior files; validators must treat file content as untrusted (malformed edits fail soft, per Failure Modes).
- Model/tool behavior: retry loops bounded (cap = 1 retry per gate in V1); judge checks can’t mutate artifacts, only accept/reject.
- Data sensitivity: run traces reference owner artifacts — [TODO: needs clarification — trace redaction rule + retention before Phase B].
- Blast radius: V1 rollout scoped to local lane by flag; worst case = degraded local-mode experience, default flow unreachable while its flag is off. Post-flag-flip blast radius widens to all lanes — which is why the flip is evidence-gated.
- Owner trust: owner must be able to see that gates guarantee system promises and never override them; run trace is inspectable on demand.
Required Mitigations
- Property test enforcing flag-off bypass (a lane with the flag off never enters harness code).
- Retry cap enforced structurally.
- Trace redaction rule defined before Phase B.
Explicit Boundaries
Do Not Modify
- The default (cloud-backend) agent loop path.
- Starter-pack behavior files’ plain-English editability (no protected blocks added).
- The frozen gauntlet eval set.
Do Not Introduce
- Mellea or any Python dependency into the product.
- A workflow-engine library — the state machine stays hand-rolled and boring.
- Any plugin/extension surface.
- Behavior content (criteria, questions, tone) in code — the bright line.
Security / Trust Boundaries
- Never commit secrets or credentials.
- Production configs read-only unless explicitly in scope.
- No telemetry/network egress added by the harness — local mode’s privacy expectation is absolute.
Out Of Scope Even If Related
- Refactoring the default flow “while we’re in there.”
- The fine-tune training pipeline (lives in plan.md / ModelMatch).
- Course/content production mechanics (marketing runway).
Open Questions
- Q1 (crux): exact semantics of “guarantee vs. constraint” per override — does anything remain non-skippable? (Proposal: nothing the owner explicitly countermands.)
- Q2: sequencing — when does Phase A start relative to the content runway and current Pulse priorities? Any DJ involvement before Phase A evidence?
- Q3: ratified wording of the refined text-first rule.
- Q4: quantitative success/abort criteria — what gate-pass delta validates the bet; what result kills it. Partially settled 2026-07-21: the hardware floor is a 16 GB MacBook (Success Definition #3) — a win that only lands on larger models doesn’t validate the bet. Still open: the gate-pass delta itself.
- Q5: which of the 11 d23 gate criteria are runtime-suitable vs. eval-only.
- Q6: run-trace redaction + retention.
- Q7: does Phase A scaffolding live in this project or ModelMatch ([[T-761]])?
- Q8 (new 2026-07-21): what exactly counts as “regression” for the GLM-5.2 flag-flip gate — metric set (gate-pass delta, latency budget per process run, cost per run, feel), thresholds for each, and who signs off (DJ owns the production flow)? Also: does “feel” get a structured eval or stay DW-judgment?
Changelog
| Date | Change | Reason | Source | Decision |
|---|---|---|---|---|
| 2026-07-18 | Initial draft — FOR DISCUSSION, decision unratified | Give the DW/DJ discussion a concrete shape | 2026-07-18 chat session | (none — proposal stage) |
| 2026-07-21 | Backend-agnostic revision: local-only toggle replaced by temporary rollout flag + per-backend policy config; “no observable change to default flow” success criterion relaxed to “no regression”; flag retirement named as intended end state; Q8 added (regression bar) | DW: guarantees are system guarantees, not a small-model crutch — harness should apply to all models unless sophisticated models regress | 2026-07-21 chat session (DW + Claude) | (none — still proposal stage; touches DJ’s production flow, needs DJ ratification) |
| 2026-07-21 | Hardware floor added to Success Definition (#3): the structured-stage win must land on a model that runs on a 16 GB RAM MacBook or better (~7–12B @ Q4); acceptance evidence requires a verified-footprint run; Q4 partially settled | DW: the target owner has a stock MacBook, not Spark-class hardware | 2026-07-21 chat session (DW + Claude) | (none — proposal stage) |
Conversation References
| Date | Source | Topics Discussed | Link |
|---|---|---|---|
| 2026-07-18 | Chat session (DW + Claude) | Mellea small-models post; text-first refinement; changes + expected results; steelman against; content-story argument; scoping to local lane | (this session; capture pending) |
| 2026-07-21 | Chat session (DW + Claude) | Toggle vs. universal harness; guarantees-not-crutch framing; regression risks for sophisticated models (latency, fluidity, metered-retry cost); rollout-flag-with-planned-death design; regression bar as the DJ ratification point | (capture pending) |
Approval
- Reviewed by: _______________
- Date: _______________
- Ready for Planning: — blocked on ratifying the proposal decision first
Next: discuss local-harness-proposal.md (agenda item filed). If ratified: record D## in decisions.md, resolve Q1–Q7, then this draft graduates toward an accepted spec and the build workflow (/test-plan before build planning).