Compare commits

..

159 Commits

Author SHA1 Message Date
Drew Ritter 43f9791850 fix: read Codex plugin version from manifest (PRI-2240) 2026-06-16 12:13:43 -07:00
Jesse Vincent 284be5905e Set v6.0.0 release date to 2026-06-16 2026-06-16 10:09:47 -07:00
Jesse Vincent 77879bbb91 Bump evals submodule: unify per-agent bootstrap scenarios
Points evals at superpowers-evals 70a245c, which replaces the seven
per-agent *-superpowers-bootstrap scenarios with one cross-agent
superpowers-bootstrap scenario (adds the QUORUM_CODING_AGENT env var and
the bootstrap-installed dispatcher check verb).
2026-06-16 10:09:47 -07:00
Jesse Vincent c5a965101b Bump version to 6.0.0 2026-06-16 10:09:47 -07:00
Drew Ritter b3ee712d3a Add visual companion Prime Radiant branding 2026-06-16 10:09:47 -07:00
Jesse Vincent 9c61797773 Draft Superpowers 6 release notes 2026-06-16 10:09:47 -07:00
Jesse Vincent b61b55013a E37: pre-flight plan review — surface plan conflicts as one batched question before Task 1 2026-06-16 10:09:47 -07:00
Jesse Vincent be400204b3 Spec: L2b tested — opus structural win, sonnet transmission+attention gap (E35/E36); bump evals to 9919b27 2026-06-16 10:09:47 -07:00
Jesse Vincent 530476fd00 L2b: plan-mandated defects are findings the human adjudicates
Reviewer tripwire (Calibration): a plan-mandated defect IS a finding,
reported as Important and labeled plan-mandated — the plan's authorship
does not grade its own work.

Controller rule (review loop): a plan-mandated finding, or any finding
conflicting with the plan's text, escalates to the human like any plan
contradiction — never dismissed because the plan mandates it.

E35 micro (frozen 0a98 replay, sonnet reviewer, 6v6): without the
tripwire 0/6 reports give the controller anything to escalate on (all
Approved, defect endorsed as spec-required); with it 6/6 report the
defect as a labeled finding.
2026-06-16 10:09:47 -07:00
Jesse Vincent e97faafb5a E27 stack: conditional impl tier + final-review tier pin + narration recipe + terse reviewer contract 2026-06-16 10:09:47 -07:00
Jesse Vincent cfe48c28ac E03: cheapest-tier implementers when plan carries complete code (transcription hypothesis) 2026-06-16 10:09:47 -07:00
Jesse Vincent 8bcefb12cb Strict-cost spec: L2 final — died at gates; explicit escalation holds at sonnet, implicit adjudication does not 2026-06-16 10:09:47 -07:00
Jesse Vincent 8e1262a3ba writing-plans: task right-sizing, Global Constraints header, per-task Interfaces blocks
Claims are fidelity and variance, not dollars (full attribution in the
superpowers-evals experiment log, 2026-06-11 L1 entry):
- Global Constraints header: 0/5 -> 5/5 adoption in micro-tests, exact
  values verbatim; makes constraints mechanically propagatable to briefs
  and reviewers (a version-floor violation class shipped because they
  weren't). The one fix wave in the elicited full runs was a version-floor
  catch this header enabled.
- Per-task Interfaces blocks: 0 -> 100% of tasks, exact signatures,
  within-plan consistent; removes the controller's per-dispatch interface
  re-derivation.
- Task right-sizing: 9.4 -> 8.4 mean tasks at svelte scale (kills
  standalone Types/README micro-tasks); no effect at small scale.
- End-to-end (opus-written plan executed under SDD): guidance plan ran 1
  fix wave vs control's 2-4 (control plan shipped a real Sierpinski bug);
  execution cost equal within noise.
2026-06-16 10:09:47 -07:00
Jesse Vincent de4672b171 Constraints block is the reviewer's attention lens: copy spec verbatim, never improvise process rules
E30 replay: the planted-DRY catch is causally determined by the
controller-composed constraints block (0/6 with process-shaped vs 5/6
with the spec's own wording). E31 micro: this recipe doubles the rate
at which composed blocks carry the spec's cross-component relationship
(6/6 vs 3/6). Affects dev and the redesign equally (E29: both 4/5).
2026-06-16 10:09:47 -07:00
Jesse Vincent 25192df30b Strict-cost spec: L1 final — cost win re-attributed to complete-code plans; guidance owns fidelity/variance 2026-06-16 10:09:47 -07:00
Jesse Vincent f5e8df4252 Strict-cost spec: L2 recon n=2 (sonnet controller $6.68/$8.05, judgment clean, escalation points unstressed) 2026-06-16 10:09:47 -07:00
Jesse Vincent b5b3b5d99c Strict-cost spec: record batch A-E rung verdicts (L1 validated, L2 recon positive, L3 dead) 2026-06-16 10:09:47 -07:00
Jesse Vincent 30bbeefe89 Spec: strict-cost SDD experiment ladder — judgment as co-invariant, plan-side crispness first 2026-06-16 10:09:47 -07:00
Jesse Vincent d3dd1ecc7d Record writing-plans micro-test result: resolved, no change needed 2026-06-16 10:09:47 -07:00
Jesse Vincent b2872a4a66 Spec: record iterations 4-5 (variance honesty, structural fixes, final validated ranges) 2026-06-16 10:09:47 -07:00
Jesse Vincent e9b88d05c8 Adopt audited positive phrasings: evidence rule leads positive; fix-report completeness as checklist 2026-06-16 10:09:47 -07:00
Jesse Vincent 4298eac856 Land eval-tuned combo: file handoffs, progress ledger, final-review package, REQUIRED model lines, reviewer risk budget
Validated 2026-06-10 (all gates pass): go-fractals 54.1-54.7 min / $12.81-14.31
(baseline 64.9 / $16.07); svelte-todo 55.0 min / 19.3M / $14.99 (baseline
79.7 / 27.3M / $20.98); planted-defect pass $2.77. Dispatch-model discipline
3/3 runs after moving model: into the templates as a REQUIRED line.
Full experiment log: evals docs/experiments/2026-06-10-sdd-cost-experiments.md
2026-06-16 10:09:47 -07:00
Jesse Vincent 69a00350ff Spec: positive-instruction redesign — audit results, micro-test method, writing-plans variants 2026-06-16 10:09:47 -07:00
Jesse Vincent d7a8c07fe3 Shared: unique review-package collateral names 2026-06-16 10:09:47 -07:00
Jesse Vincent c30d822efe Add review-package script; close fix-dispatch test gap
scripts/review-package generates the reviewer's input deterministically:
commit list, stat summary, and net diff with -U10 context, written to a
file from an explicit BASE. Live runs showed controllers improvising
'git diff HEAD~1..HEAD', which silently truncates multi-commit tasks,
and svelte's five fix dispatches shipped without re-running any tests —
fix dispatches now explicitly carry the implementer's
re-run-and-report contract.
2026-06-16 10:09:47 -07:00
Jesse Vincent 68c9ddb870 Describe the review design as current state, not as a delta
The skill read as a changelog: 'combined task review,' 'one reviewer,
one reading,' 'one dispatch,' and an example still showing diffs pasted
into prompts. A reader who never saw the two-reviewer design has no
referent for 'combined.' Prose now states the design directly, and the
flowchart/example reflect the diff-file handoff.
2026-06-16 10:09:47 -07:00
Jesse Vincent aa80399355 Spec: record iterations 2-3 results and final frozen-config matrix 2026-06-16 10:09:47 -07:00
Jesse Vincent ee656563c9 Hand reviewers the diff as a file, not a paste
Paste adoption stayed at 0/15 even as a Red Flag — and the controller's
reluctance is locally rational: pasting loads the diff into the (most
expensive) controller context permanently, while a reviewer self-fetch
costs a few cheap turns. The diff-file handoff is cheap for both sides:
the controller redirects git diff to /tmp without reading it, and the
reviewer gets the whole change in one Read call.
2026-06-16 10:09:47 -07:00
Jesse Vincent 3280a32259 Reviewer skepticism covers the implementer's design rationales
Fourth planted-defect failure mode: the implementer's self-report said
'noted mild structural duplication; left unabstracted per YAGNI' and the
reviewer deferred to that framing, rating the duplication no finding at
all. The pre-judging keeps relocating — controller prompt, then reviewer
calibration, now the implementer's report. Rationales are claims; they
never downgrade severity.
2026-06-16 10:09:47 -07:00
Jesse Vincent 84d033e967 Make diff-pasting non-optional for task reviewer dispatch
Adoption was 6/11 reviews on fractals and 0/17 on svelte when phrased
as guidance; reviewers without the diff re-derive it by hand, which is
the single largest remaining reviewer cost. Now a Red Flags Never entry
and a REQUIRED marker on the template placeholder.
2026-06-16 10:09:47 -07:00
Jesse Vincent c73e9a9a3f Close the Minor-severity escape hatch
With merged review, a planted verbatim-duplication defect shipped: the
reviewer rated it Minor (YAGNI) under the strict cannot-be-trusted
definition of Important, and the Minor-rolls-up rule meant no fix was
ever dispatched and the final review never saw the finding. Calibration
now names merge-blocking maintainability damage (verbatim duplication,
swallowed errors, assertion-free tests) as Important, and controllers
must paste accumulated Minor findings into the final review dispatch.
2026-06-16 10:09:46 -07:00
Jesse Vincent 097ed5920f Spec: document cost iterations and the per-task review consolidation 2026-06-16 10:09:46 -07:00
Jesse Vincent e08ad0660a Merge per-task reviews into one task reviewer (iteration 2)
Iteration-1 profiling: implementers and per-dispatch overhead dominate
(429 of 686 subagent turns; controller coordination is half the dollars
and scales with dispatch count), reviewers are individually lean, and
the controller pasted the diff in only 2 of 22 review dispatches when
the guidance was phrased as optional.

Changes: spec-reviewer-prompt.md + code-quality-reviewer-prompt.md
replaced by task-reviewer-prompt.md (one reviewer, one reading of a
pasted diff, two verdicts: spec compliance //⚠️ and task quality);
one fix dispatch can address both kinds of findings; controller now
runs git diff itself and pastes it (imperative, not optional);
implementers run focused tests while iterating and the full suite once
before committing; flowchart, example, Red Flags, tool tables updated.
The broad final whole-branch review is unchanged.
2026-06-16 10:09:46 -07:00
Jesse Vincent 5e03007c85 Cut review-cost drivers: turn-aware models, inline diffs, scoped evidence
Round-2 fractals eval regressed to 70min/32.2M tokens (vs round-1's
42.8min/14.5M) while reaching baseline-parity quality. Per-subagent turn
profiling attributed it to: haiku dispatches taking 2-3x the turns of
sonnet (678 of 1197 subagent turns), reviewers re-fetching diffs by hand
(518 Bash calls), and evidence-rule narration. Changes: turn-count-beats-
token-price model guidance; controllers paste small diffs into reviewer
prompts (reviewers then need few or no tool calls); evidence scoped to
findings and would-be-bare-yes checks; Important defined as cannot-trust-
until-fixed with coverage suggestions Minor; fixes dispatched only for
Critical/Important.
2026-06-16 10:09:46 -07:00
Jesse Vincent d55cdce32c Add phrase-level pre-judging triggers to reviewer prompt rule
Resumed the offending eval controller session and asked it why it
pre-judged despite the rule being in context. Its retrospective: the
motive was avoiding a review loop, the abstract rule was read but not
applied at the moment it governs, and a phrase-level trigger ('do not
flag', 'at most Minor', 'don't treat X as a defect', 'the plan chose')
would have fired where the principle did not.
2026-06-16 10:09:46 -07:00
Jesse Vincent 0974229418 Red Flags: never tell a reviewer what not to flag or pre-rate severity
Second observed instance: with the Constructing Reviewer Prompts rule
already live, a controller still wrote 'do not treat that duplication as
a defect to fix — the plan chose it; you may note it as a Minor
observation at most' into a quality reviewer dispatch, fabricating plan
intent from the plan's example snippet. Promote the rule to the Red
Flags Never list and name the rationalization.
2026-06-16 10:09:46 -07:00
Jesse Vincent 62b1682399 Close three review blind spots found by defect tracing
Live eval deliverables shipped five polish defects; tracing each through
the transcripts showed three mechanisms, each now addressed:
- reviewers answered pointed checklist items with unsupported yes
  (evidence rule: every What-to-Check answer needs file:line evidence)
- no reviewer ever saw the design's global constraints (controllers now
  paste binding constraints into task requirements)
- test output noise was invisible everywhere (pristine-output checks in
  implementer self-review and quality review)
2026-06-16 10:09:46 -07:00
Jesse Vincent b42a232192 Require explicit model on subagent dispatch
In live eval runs, controllers given judgment-based model selection
stopped passing a model at all; the omitted parameter inherits the
session's top-tier model, silently making every subagent maximally
expensive (one run dispatched 26/26 reviewers on the session model).
2026-06-16 10:09:46 -07:00
Jesse Vincent 83354984ed Forbid controllers pre-judging reviewer findings
A live eval run of sdd-quality-reviewer-catches-planted-defect caught the
SDD controller fabricating a plan constraint and instructing the quality
reviewer not to flag the planted DRY violation. The duplication shipped.
Constructing Reviewer Prompts now bans suppression directives alongside
open-ended broadening directives.
2026-06-16 10:09:46 -07:00
Jesse Vincent acb746544d Sync plan: escaped pre() pattern in Task 5 checks block 2026-06-16 10:09:46 -07:00
Jesse Vincent 16eaa8a158 Fix plan doc: correct Task 1 grep expectation; sync Task 5 story block 2026-06-16 10:09:46 -07:00
Jesse Vincent 42653013d9 Sync plan's Task 5 blocks with review fixes 2026-06-16 10:09:46 -07:00
Jesse Vincent 16da215270 SDD controller: reviewer prompt budgets, ⚠️ handling, final-review pointer, model judgment 2026-06-16 10:09:46 -07:00
Jesse Vincent cc6205389c Implementer prompt: re-run covering tests after fixing review findings 2026-06-16 10:09:46 -07:00
Jesse Vincent bf46da2472 Scope spec reviewer's Your Job wording to the diff 2026-06-16 10:09:46 -07:00
Jesse Vincent 342f4e2f21 Spec reviewer: judge from the diff, grounded skepticism, ⚠️ verdict channel 2026-06-16 10:09:46 -07:00
Jesse Vincent 12ed80e8ca Use bare placeholder names in quality reviewer prompt body 2026-06-16 10:09:46 -07:00
Jesse Vincent d1a14e37eb Make per-task quality reviewer prompt self-contained and task-scoped 2026-06-16 10:09:46 -07:00
Jesse Vincent 61e2b82367 Add implementation plan for task-scoped review dispatch 2026-06-16 10:09:46 -07:00
Jesse Vincent 1649580749 Harden review-dispatch spec per adversarial review findings 2026-06-16 10:09:46 -07:00
Jesse Vincent b04645dc37 Add design spec: task-scoped review dispatch for SDD 2026-06-16 10:09:46 -07:00
Drew Ritter bfa21156f2 chore: bump evals submodule to claude transcript-capture fix
Bumps evals 7f8e80c -> db37d5f (superpowers-evals#16): the claude launcher now
sets CLAUDE_CODE_FORCE_SESSION_PERSISTENCE=1 so nested interactive claude
(>=2.1.176) persists its transcript — restoring claude capture (verdicts +
cost/token data) on the latest CLI (2.1.177) with no version pin. Also folds in
the audit_liveness ruff/ty cleanup and the B1 audit-doc correction.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 10:09:46 -07:00
Drew Ritter 3a907d6a0a Fix companion stop metadata and token permissions 2026-06-16 10:09:46 -07:00
Drew Ritter 1c80914052 Harden Windows browser launcher 2026-06-16 10:09:46 -07:00
Drew Ritter 2a8479b21d Fix Windows lifecycle validation 2026-06-16 10:09:46 -07:00
Drew Ritter fd9972a4bd Align visual companion docs with shipped scope 2026-06-16 10:09:46 -07:00
Drew Ritter 69ed41af9e Fix companion test cleanup and argv assertions 2026-06-16 10:09:46 -07:00
Drew Ritter 51323e4c64 Harden companion platform tests 2026-06-16 10:09:46 -07:00
Drew Ritter 3402d4e7d7 Fix companion lifecycle test ownership metadata 2026-06-16 10:09:46 -07:00
Drew Ritter 6bc49f0183 Harden companion stop ownership proof 2026-06-16 10:09:46 -07:00
Drew Ritter 8f2525a803 Isolate companion fallback tokens 2026-06-16 10:09:45 -07:00
Drew Ritter 85914fbcf8 Fix server test fallback cleanup 2026-06-16 10:09:45 -07:00
Drew Ritter 0410679757 Harden root screen containment 2026-06-16 10:09:45 -07:00
Drew Ritter d9ec1196b8 Plan visual companion final hardening fixup 2026-06-16 10:09:45 -07:00
Drew Ritter 92a0a7acc0 Tighten visual companion hardening spec 2026-06-16 10:09:45 -07:00
Drew Ritter ce6be66c87 Document visual companion final hardening fixup 2026-06-16 10:09:45 -07:00
Drew Ritter 69270c9007 Harden companion Windows lifecycle coverage 2026-06-16 10:09:45 -07:00
Drew Ritter b17d54f839 Harden brainstorm companion auth regressions 2026-06-16 10:09:45 -07:00
Drew Ritter 83b5d3a963 Document visual companion auth hardening plan 2026-06-16 10:09:45 -07:00
Jesse Vincent 7fbae0252f fix(brainstorm-server): fix auth-integration bugs from full-branch review
A second adversarial review of the merged branch found that combining the
session-key auth with the feature work created real bugs the (vacuous) tests
missed:

- [Critical] GET /files/ (empty name) resolved to CONTENT_DIR and crashed the
  process with uncaught EISDIR — newly reachable because the query-stripping
  refactor turns /files/?key=... into /files/. Reject non-regular-file names.
- [High] --open opened a KEYLESS url, which the auth gate 403s — the headline
  feature landed on the error page. Open the keyed url.
- [High] Same-port restart regenerated the token (port persisted, token not), so
  the open tab's old cookie 403'd and never reconnected — contradicting the
  documented promise. Persist the token (BRAINSTORM_TOKEN_FILE / .last-token)
  alongside the port.
- [Medium] Token sat in world-readable server-info/server.log (0644 in /tmp).
  umask 077 in start-server.sh + mode 0600 on server-info/.last-token.
- [Medium] touchActivity() ran before the auth check, so unauthenticated requests
  defeated the idle timeout. Count activity only after authorization.
- [Low] COOKIE_NAME embedded the pre-fallback port; derive it from the actual
  bound port (also prevents a cross-server cookie-jar collision on fallback).

Tests added/strengthened (previously passed vacuously): /files/ no-crash; the
auto-open url carries the key and is reachable (200); restart reuses the same key
not just the port; unauthenticated requests don't reset the idle clock.
Full suite green (ws-protocol 32, helper 12, auth 13, server 29, lifecycle 8,
stop-server 4); restart smoke confirms same port+key and old URL -> 200.
2026-06-16 10:09:45 -07:00
Jesse Vincent 01de36703d test(brainstorm-server): thread session key through tests after auth merge
Integrating the per-session-key auth onto the same branch as the dotfile and
lifecycle work: two tests added after the auth commit opened WebSockets without a
key (server.test.js dotfile-reload, lifecycle.test.js idle-shutdown), which the
auth gate now resets. Pass ?key=/BRAINSTORM_TOKEN in both. Full suite green:
ws-protocol 32, helper 12, auth 13, server 28, lifecycle 7, stop-server 4.
2026-06-16 10:09:45 -07:00
Jesse Vincent cb5bb885fd feat(brainstorm-server): gate every endpoint behind a per-session key
The companion server is reachable by any local browser tab (default loopback
bind) and by any host that can route to it (remote --host bind). It served
screens, files, and accepted event-injecting WebSocket connections with no
authentication, so a malicious browser tab or a direct remote client could read
brainstorm content or inject events that the agent reads as the user's input
(prompt injection into a live session).

Generate a per-session secret token, carry it in the served URL as ?key=, and
mirror it into an HttpOnly SameSite=Strict per-port cookie on first load so
same-origin subresources and the WebSocket handshake authenticate automatically.
Every HTTP request and WebSocket upgrade now requires a valid key (query or
cookie, constant-time compared); unauthenticated requests get a friendly 403
explaining they need the full URL. A secret authenticates the client uniformly
across loopback, tunnel, and remote binds and defeats DNS rebinding, which a
Host/Origin allowlist cannot.

Also guard handleMessage against a null JSON payload that crashed the process.

Tests: new auth.test.js (13 cases) covering the key on /, /files/*, and WS plus
cookie bootstrap and the null-payload guard; server.test.js threads the key;
ws-protocol.test.js + auth.test.js wired into npm test.

Closes #1014
Refs #1110, #1553, #1504
2026-06-16 10:09:45 -07:00
Jesse Vincent 09b6b25e08 docs(brainstorm): catalog visual companion issues; choose session-key for security
Records the triage of open issues/PRs touching the brainstorm companion server
and the decision to protect it with a per-session secret key (supersedes the
Host/Origin allowlist approach) so remote-connected users are covered, not just
loopback.
2026-06-16 10:09:45 -07:00
Jesse Vincent 7c805f34d2 fix(brainstorm-server): tie stop-server PID check to the session's port
The node+server.cjs command match (from the adversarial review) still matched any
unrelated node process running a file named server.cjs. When we recorded the
bound port (state/server-info) and lsof is available, additionally require the
PID to be the process actually LISTENING on this session's port — which rules out
a different project's server.cjs / editor task runner that recycled the stale
PID. Falls back to the command match when the port or lsof isn't available.

Test: a 'node server.cjs' process not listening on the recorded port is spared.

Refs #1703
2026-06-16 10:09:45 -07:00
Jesse Vincent fb08947ded fix(brainstorm-server): address adversarial review findings
From a two-reviewer adversarial pass:

- [High] EADDRINUSE fallback clobbered the shared .last-port: onListen wrote the
  bound port unconditionally, so a fallback to a random port overwrote the
  preferred port another live session still owns — stranding that session's open
  tab forever. Now persist only when we bound the preferred port (not on
  fallback). The fallback test now asserts .last-port integrity (teeth-verified).

- [Medium] maybeOpenBrowser ran the URL through a shell (exec + JSON.stringify),
  which does NOT neutralize $(...) in a url-host. Platform launchers now use
  execFile with the URL as an argv element (no shell). The operator-set
  BRAINSTORM_OPEN_CMD path stays shell-based (trusted input).

- [Medium] --open was a silent no-op on native Windows (no win32 branch). Added.

- [Medium] helper.js reconnect/status/tombstone had only substring-grep tests.
  Added behavioral tests driving the state machine against a mocked browser:
  Reconnecting+backoff (500->1000->2000), tombstone after the grace period, and
  reload-on-recovery.

- [Low] status pill showed a false 'Connected' before the socket opened; now
  starts 'Connecting…' until onopen.

Not changed (flagged): stop-server.sh's PID-ownership check still matches any
'node ... server.cjs' (narrow residual — a recycled PID onto an unrelated node
server.cjs); robust fix needs fragile cross-platform process introspection.
2026-06-16 10:09:45 -07:00
Jesse Vincent 5a0f895387 feat(brainstorming): offer the visual companion just-in-time; harden lifecycle guidance
Move the companion consent from an upfront, anticipatory offer to the first
moment a question would genuinely be clearer shown than told. If no visual
question ever arises, it's never offered. On approval the agent starts the
server with --open, so the user's browser opens to the first screen — the pop is
tied to that approval, never unsolicited.

Also hardens visual-companion.md: confirming the server is alive (server-info
present, server-stopped absent) before referring to the URL is now a required
step; restart with the same --project-dir reuses the port so the open tab
reconnects on its own (paused overlay while down); idle default corrected to 4h.

NOTE: SKILL.md is behavior-shaping content — this flow change should be
eval-tested (writing-skills adversarial pressure test) before merge.

Refs #1237, #1037
2026-06-16 10:09:45 -07:00
Jesse Vincent 463dfb7fd4 feat(brainstorm-server): opt-in auto-open of the browser on the first screen
When the user approves the visual companion, open their browser automatically the
first time a screen is actually ready to show — rather than at startup (just the
waiting page) or making them open the URL by hand.

Opt-in and gated on approval: off unless BRAINSTORM_OPEN is set (start-server.sh
--open, which the agent passes only after the user agrees to use the companion).
Even then it fires once, and is skipped if a browser is already connected, on a
non-loopback/remote bind, or when headless. Launcher is the platform default
(open / xdg-open / WSL cmd.exe) or BRAINSTORM_OPEN_CMD; best-effort, never fatal.

lifecycle.test.js: opens once on the first screen when approved; does NOT open
without approval.

Closes #755
Refs #759
2026-06-16 10:09:45 -07:00
Jesse Vincent dd9fcc21ee feat(brainstorm-server): reuse the same port on session restart
When the companion idle-shuts-down and the agent restarts it, a fresh random
port meant the user's open browser tab pointed at a dead URL. Persist the bound
port per project and prefer it on the next start, so the restarted server comes
up on the same port and the open tab's reconnect just works.

- start-server.sh exports BRAINSTORM_PORT_FILE=<project>/.superpowers/brainstorm/
  .last-port for project sessions (not /tmp).
- server.cjs prefers an explicit BRAINSTORM_PORT, else the recorded port, else
  random; writes the actually-bound port back; and on EADDRINUSE (preferred port
  still in use) falls back to a random port once instead of crashing.

lifecycle.test.js: restart reuses the recorded port; a taken preferred port
falls back to a random one without crashing.

Refs #1237
2026-06-16 10:09:45 -07:00
Jesse Vincent 36ac3e1336 feat(brainstorm-companion): resilient reconnect, live status, paused overlay
The injected client reconnected on a fixed 1s timer with no feedback: if the
laptop slept or the server restarted, the page showed 'Connected' over a dead
socket and silently queued events. And when the server stopped, the user got a
bare connection-refused with no explanation.

helper.js now:
- reconnects with exponential backoff (500ms, doubling, capped at 30s; reset on
  open), with an onerror->close handler, nulls the socket on close, and clears a
  pending timer before scheduling another;
- drives the frame status pill Connected/Reconnecting/Disconnected via a
  --status-color custom property (frame-template.html);
- after ~15s disconnected, shows a self-styled 'Companion paused' overlay
  (tombstone) explaining the companion stopped and will reconnect automatically;
- on recovery from a tombstoned outage (e.g. server restarted on the same port)
  reloads to pick up the restarted server's current screen.

The reconnect-backoff is an exported pure function; helper.test.js unit-tests it
(doubling + cap progression) and asserts the status/tombstone/reconnect wiring.
DOM behaviour is verified live.

Refs #856, #1237
2026-06-16 10:09:45 -07:00
Jesse Vincent 56757f6877 feat(brainstorm-server): 4h configurable idle timeout; close WS on shutdown
The companion shut down after only 30 minutes idle — too short for real
brainstorming, where a single question can sit far longer. And shutdown() never
closed upgraded WebSocket sockets, so an open browser connection could keep the
Node process alive after it was supposed to exit.

- Default idle timeout raised to 4 hours, configurable via BRAINSTORM_IDLE_TIMEOUT_MS
  and start-server.sh --idle-timeout-minutes (validated positive integer).
- Reported as idle_timeout_ms in the server-started JSON / server-info.
- shutdown() now destroys all client sockets so the process exits even with an
  open WebSocket.
- Watchdog check interval is configurable (BRAINSTORM_LIFECYCLE_CHECK_MS, default
  60s) so the lifecycle can be tested without minute-long waits.

Adds lifecycle.test.js (configured timeout reported; idle shutdown exits despite
an open WS — teeth-verified; the start-server flag). Wires ws-protocol,
lifecycle, and stop-server suites into npm test.

Closes #1237
Refs #1689
2026-06-16 10:09:45 -07:00
Jesse Vincent 5ddce063df fix(brainstorm-server): verify PID ownership before stopping
stop-server.sh read server.pid and SIGKILL'd that PID with no checks. After a
reboot or PID wraparound the pid file can point at an unrelated, live process —
which we would then kill.

Verify the PID is actually our server (a running 'node ... server.cjs') before
signalling it. If ownership can't be proven, fail closed: remove the stale pid
file and report {status: stale_pid} without killing anything. Real servers still
stop ({status: stopped}); a missing pid file still reports not_running.

Adds stop-server.test.sh covering: an unrelated reused PID is left alone, a real
server is stopped, and a missing pid file.

Refs #1703
2026-06-16 10:09:45 -07:00
Jesse Vincent 2b108b7dc2 fix(brainstorm-server): ignore macOS resource-fork dotfiles
On macOS (and ExFAT/SMB volumes) the OS writes ._<name>.html sidecar files
holding binary resource-fork metadata. These end with .html, so they passed the
content filter and could be picked as the newest screen — serving binary garbage
to the browser instead of the mockup — or fetched via /files/.

Skip dotfiles (leading '.') at all four sites that list or serve content:
getNewestScreen, the /files/ endpoint, the known-files seed, and the fs.watch
handler. Tests cover serving (/ and /files/) and the watch path (a ._ file must
not trigger a reload).

Refs #950
2026-06-16 10:09:45 -07:00
Drew Ritter 565845f251 chore(evals): bump submodule to SUP-333 boundary + plumbing scenarios (7f8e80c)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-16 10:09:45 -07:00
Jesse Vincent 1aa45d20d2 fix(writing-skills): hang backfire mechanism on the separated prohibition-vs-recipe comparison (NEW-4); control comparison stated as trend 2026-06-16 10:09:45 -07:00
Jesse Vincent fdb0f42595 fix(writing-skills): scope empirical claims, honest noise reporting, conditionalize micro-test checklist line
Adversarial review findings 1/3/9: the head-to-head result is now scoped
to its context (dispatch-prompt guidance) with an explicit micro-test-your-
own-case instruction; the nuance-clause result is reported as
consistent->noisy rather than 'measurably dilutes'; the checklist line is
scoped to behavior-shaping guidance and the micro method no longer assumes
raw API access.
2026-06-16 10:09:45 -07:00
Jesse Vincent cbc8273bdd feat(writing-skills): form-selection table + micro-test wording method
RED battery (35 opus authoring samples against the current skill) showed
authors default to prohibition+rationalization-table for composition-
shaping problems (T1: 5/5), where that form measurably backfires
(prohibition 4.4 vs 3.6 no-guidance control vs 3.0 recipe restatement
errors), and design only full-subagent verification with no wording
micro-tests, no mandatory no-guidance control, no manual inspection of
automated matches, no variance signal (T7: 5/5).

Adds: Match the Form to the Failure (failure-type -> form table, nuance/
exemption rules), scope note on Bulletproofing, Micro-Test Wording
subsection, two checklist lines. Deliberately narrow: T3/T4/T5/T6 RED
samples showed Iron Law / elicit-first behavior already strong.
2026-06-16 10:09:45 -07:00
Drew Ritter c8fc00435b chore(evals): bump submodule for Claude Haiku target 2026-06-16 10:09:45 -07:00
Drew Ritter 7813867bbc chore(evals): bump submodule to --scenarios filter (ff3ee83)
Adds `run-all --scenarios` for resuming a scenario subset across the Code
Assist rate-limit windows. Follows the agy rate-limit fix (79f9963).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 10:09:45 -07:00
Drew Ritter 657174abdb chore(evals): bump submodule to antigravity rate-limit fix (79f9963)
Serialize antigravity against the Gemini Code Assist rate limit
(max_concurrency=1), diagnose 429/RESOURCE_EXHAUSTED honestly instead of as
auth, fail-fast on a latched window, and tolerant preflight OK match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 10:09:45 -07:00
Rahul d9d3d99245 fix(brainstorming): cap websocket frame payloads 2026-06-16 10:09:45 -07:00
Drew Ritter 21b44e44d3 Add shell lint script 2026-06-16 10:09:45 -07:00
Drew Ritter 2c2e2bcbd4 Tighten Kimi plugin porting coverage 2026-06-16 10:09:44 -07:00
Drew Ritter c8778664cd docs: restore Kimi direct install command 2026-06-16 10:09:44 -07:00
Drew Ritter e47add1dba docs: simplify Kimi README install steps 2026-06-16 10:09:44 -07:00
Drew Ritter f61300eac8 fix: wire Kimi plugin into release metadata 2026-06-16 10:09:44 -07:00
Drew Ritter e15d4ecd88 fix: align Kimi manifest with supported fields 2026-06-16 10:09:44 -07:00
qer 9f798e4a9e feat: add Kimi Code plugin manifest 2026-06-16 10:09:44 -07:00
Matt Van Horn afbf0fcfac feat(subagent-dev): add TDD RED evidence to implementer report format
Add a conditional TDD Evidence field to the implementer report format so controllers can verify RED and GREEN output when TDD was required.

The field asks for the command run, relevant RED/GREEN output, and the expected RED failure reason rather than raw full logs.

Fixes #994.
2026-06-16 10:09:44 -07:00
Drew Ritter 4548b69c60 docs(windows): trim polyglot hook implementation copy 2026-06-16 10:09:44 -07:00
dev_Hakaze 3d0725756c docs(windows): update polyglot hook docs
Rewrite the Windows polyglot hook documentation to match the current run-hook.cmd dispatcher and update the porting guide cross-reference.\n\nFixes #1653.
2026-06-16 10:09:44 -07:00
nestorluiscamachopaz 48696e6519 fix: foreground mode saves node PID and clears OWNER_PID on Windows/MSYS2
Verified on real Windows Git Bash: lifecycle test passed 12/12, manual start/stop released the port, and no brainstorm node processes remained.
2026-06-16 10:09:44 -07:00
nawfal 4e3707fbbe fix(finishing-a-development-branch): remove gh-specific PR creation instruction
Per obra's guidance on #1609: remove the github-specific instruction rather
than replacing it with a platform-detection table. Agents already know their
forge tooling; the skill only needs to cover the push step.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 10:09:44 -07:00
nawfal 24ae4c8001 fix(finishing-a-development-branch): detect remote platform before creating PR/MR
Replaces hardcoded `gh pr create` in Option 2 with a platform-neutral
note: check `git remote get-url origin` first, then use gh (GitHub),
glab (GitLab), or fall back to the compare URL for unknown platforms.

Adds matching Red Flag entry so agents don't skip the detection step.

Fixes #1609

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 10:09:44 -07:00
Jesse Vincent 36ce0a21e4 feat: add Antigravity CLI (agy) support
Antigravity (Google's `agy` CLI) installs the existing Superpowers plugin
directly:

    agy plugin install https://github.com/obra/superpowers

agy imports the bundled skills and runs the plugin's SessionStart hook, so
using-superpowers bootstraps from the first message — verified on agy 1.0.3:
a fresh session given "Let's make a react todo list" auto-triggers the
brainstorming skill instead of writing code. agy discovers skills natively
and, having no Skill tool, loads them by reading SKILL.md with view_file.

No scaffold, installer, or generated context file is needed. This adds only:

- README.md: an Antigravity install section + Quickstart link
- skills/using-superpowers/SKILL.md: reference to the agy tool mapping
- skills/using-superpowers/references/antigravity-tools.md: action->tool
  mapping for agy (view_file, write_to_file, invoke_subagent, manage_task,
  and skill loading via view_file on SKILL.md)
- tests/antigravity/: structural test for the tool mapping, mirroring
  tests/pi/
2026-06-16 10:09:44 -07:00
Jesse Vincent 3608167e05 docs: add 'Porting Superpowers to a New Harness' guide
An evergreen guide for adding support for a new harness (IDE, CLI, or agent
runner). Teaches the invariants — automatic session-start bootstrap, skill
discovery/invocation, tool mapping, the acceptance test — and points at the
closest reference integration shape (shell-hook, in-process plugin,
instructions-file / declared context file) to copy. Covers discovery, build,
local install, tmux-driven verification, distribution, and PR submission, with a
live reference-integration index and a gotchas appendix.

Two non-negotiable rules: (1) never edit skill bodies; (2) everything ships
through the harness's own install mechanism — never edit the user's config. When
a plugin installer strips undeclared files, declare the bootstrap as a recognized
component (a manifest contextFileName-style context file the installer preserves
and the harness loads every session), generated at install time from the live
SKILL.md + tool mapping. Surfaced-skill-description bootstrap is the softer
fallback.

Hardened against real end-to-end ports (Antigravity CLI): shapes can compose; a
fork doesn't inherit its parent's behavior; a hook system != a usable
session-start event; verify @-includes AND context-file preservation with a
marker; web-search the docs and study existing plugins; reverse-engineer
undocumented harnesses; print/headless modes may hang; workspace-trust gates
stall tmux; declared context files survive plugin install while undeclared files
are stripped; skills-path registration is per-harness.
2026-06-16 10:09:44 -07:00
Jesse Vincent 18726fe0a3 fix(sync-to-codex-plugin): exclude /.pi/ so the pi extension doesn't leak into the Codex plugin
The .pi/ directory holds the pi-harness extension (.pi/extensions/superpowers.ts),
which is tracked (not git-ignored), so the git-ignored-path exclusion helpers
never caught it. It was also missing from the static EXCLUDES list alongside the
other harness dotdirs (.opencode, .cursor-plugin, .claude-plugin), so a sync
would rsync pi's files into the Codex plugin distribution. Add /.pi/ to EXCLUDES.
2026-06-16 10:09:44 -07:00
Jesse Vincent 8ca7d218d0 Revert "Make visual-companion.md script paths skill-rooted, not plugin-rooted"
This reverts commit e9f5188289.
2026-06-16 10:09:44 -07:00
Jesse Vincent c676d3639d Revert "Probe per-user Git Bash and Scoop before falling back to PATH on Windows"
This reverts commit a8f0738e3a.
2026-06-16 10:09:44 -07:00
Jesse Vincent ce86e63eb6 Probe per-user Git Bash and Scoop before falling back to PATH on Windows
Stock Windows 10/11 ships C:\Windows\System32\bash.exe (the WSL
launcher) as the first match for `where bash`. WSL's bash cannot
execute Windows-style script paths, so when Git Bash is installed
outside the two standard system locations -- specifically the
per-user "Only for me" Git for Windows installer
(%LOCALAPPDATA%\Programs\Git) or a Scoop install
(%USERPROFILE%\scoop\apps\git\current\usr\bin) -- run-hook.cmd
silently fails: WSL prints "Windows Subsystem for Linux must be
updated", the script returns 0, and Superpowers' SessionStart
bootstrap is never injected. From the user's perspective skills
auto-trigger inconsistently or not at all, with no surfaced error.

Add explicit probes for both locations between the existing system-
wide Git for Windows checks and the `where bash` fallback. Also add
a comment to the fallback documenting the WSL-launcher trap so future
maintainers understand why the explicit probes must come first.

Verified on a Windows 11 VM (dockur/windows 11, Git Bash 2.x, Node
22):
- System Git present: existing probe still matches (no regression)
- System Git absent, per-user Git present via junction: new probe
  matches, hook produces valid 6422-byte JSON, exit 0
- All Git probes absent: confirmed WSL trap fires
  ("Windows Subsystem for Linux must be updated") and the hook exits 0
  silently, demonstrating the original bug

Existing tests/hooks/test-session-start.sh still passes on macOS (7/7).

Reported by @ytchenak in #1607.

Co-authored-by: ytchenak <ytchenak@users.noreply.github.com>
Closes #1607.
2026-06-16 10:09:44 -07:00
Jesse Vincent d72560e462 Pipe SessionStart hook printf through cat to absorb EPIPE on Windows
On Windows + Git Bash, the SessionStart hook prints a confusing
diagnostic at every startup ("printf: write error: Permission denied")
when Claude Code closes the hook's stdout pipe before the printf has
finished writing. The hook still runs to completion and context still
gets injected, but the diagnostic surfaces every session because
Git Bash's printf reports EPIPE as "Permission denied" (not "Broken
pipe" like Linux) and our `set -euo pipefail` lets that error escape.

Piping each printf through `cat` makes the external cat process the
recipient of any SIGPIPE / EPIPE. cat's failure does not propagate to
the parent bash under pipefail because cat is the last command in the
pipeline and exits cleanly when the pipe stays open long enough to
hold the data. On macOS/Linux the cat passthrough is transparent (no
behavior change, no measurable cost).

Verified:
- Existing tests/hooks/test-session-start.sh: 7/7 pass on macOS
- Manual run on Windows 11 + Git Bash 5.2 + Node 22 produces valid JSON,
  clean stderr, and exit 0
- JSON output is byte-identical to the unpatched hook

Reported by @silvertakana in #1612, attribution preserved in the
Co-authored-by trailer below — this is the same fix shape the original
PR proposed.

Co-authored-by: silvertakana <silvertakana@users.noreply.github.com>
Closes #1612.
2026-06-16 10:09:44 -07:00
Nick Galatis 90e1721817 fix(systematic-debugging): defuse Claude Code ultrathink keyword scanner trigger (#1558)
The "Signals You're Doing It Wrong" bullet in systematic-debugging/SKILL.md
contains the literal token Claude Code's runtime scans for in tool result
bodies. Every Skill-tool invocation of this skill caused the harness to
inject a spurious system-reminder claiming the user requested deeper
reasoning, silently bumping every session into extended thinking.

Replace the bullet's spelling so the contiguous letter sequence the scanner
matches is broken with a hyphen. The signal text remains recognizable to
the agent and the documented action ("Question fundamentals, not just
symptoms") is unchanged.

Fixes obra/superpowers#1283
2026-06-16 10:09:44 -07:00
Jesse Vincent a318a5f621 Make visual-companion.md script paths skill-rooted, not plugin-rooted
Issue #1134: agents reading visual-companion.md see bare commands like
`scripts/start-server.sh`, correctly identify the plugin install
directory, then look for `<plugin>/scripts/start-server.sh` instead of
`<plugin>/skills/brainstorming/scripts/start-server.sh`. The file
doesn't exist at the plugin-rooted path, so the agent concludes the
visual companion isn't available and falls back to text-only
brainstorming.

Multiple independent reproductions in the issue thread, plus one user's
agent self-reported: "I assumed the scripts folder was in the root
directory of the plugin, it didn't realize it could have been talking
about the skill folder itself."

Change all `scripts/<file>` references in visual-companion.md to
`skills/brainstorming/scripts/<file>`. Agents that correctly identify
the plugin root will now join to the right path.

Closes #1134.
2026-06-16 10:09:44 -07:00
Jesse Vincent 95aa3d5007 Align windows-lifecycle test with current brainstorm server layout
The test had drifted behind three server implementation changes and no
longer ran against the actual server:

- Server entrypoint renamed from server.js to server.cjs; the test still
  invoked node on server.js and failed with MODULE_NOT_FOUND.
- Server state moved to a state/ subdirectory (state/server-info,
  state/server.pid); the test still waited on .server-info and wrote
  .server.pid at the session root.
- Owner-PID startup validation now keeps the server running when the
  owner PID is dead at startup: it logs owner-pid-invalid, disables
  owner monitoring, and falls back to the idle timeout. The test still
  expected the server to self-terminate within 60s of a dead-at-startup
  owner.

Update file/path references to match the current server, and rewrite
the dead-at-startup test to assert the current behavior: server
survives, log contains owner-pid-invalid, log does not contain a
spurious "owner process exited" line.

Verified locally: 9 passed, 0 failed, 3 skipped (Windows-only).
2026-06-16 10:09:44 -07:00
Jesse Vincent 3cd2db9f8a Convert curly to square brackets in code-reviewer.md placeholders
Matches the style used by the spec-reviewer-prompt.md and
code-quality-reviewer-prompt.md call sites, which already use square
brackets ([VAR] or [VAR — description]). No semantic change — these
placeholders are filled in by the controller; nothing programmatic
substitutes them.
2026-06-16 10:09:44 -07:00
Jesse Vincent 8ed7c499b3 Scope spec reviewer to task diff and make reviewers read-only
Two problems with the SDD reviewer prompts on dev:

- spec-reviewer-prompt.md never received a git range, so the
  general-purpose subagent had to crawl the entire codebase to find what
  changed. Reporter measured 20-33 minute spec reviews on simple tasks
  (#1538).
- Neither reviewer prompt told the subagent that review is read-only.
  A spec reviewer running `git checkout <parent-sha>` for historical
  comparison silently detached HEAD on the controller's branch, then
  subsequent task commits accumulated on the detached HEAD and were
  effectively orphaned (#1543, reproduced independently in #1543's
  thread).

Add a Git Range to Review section to spec-reviewer-prompt.md that
mirrors the one code-reviewer.md already has, plus a Read-Only Review
section in both reviewer prompt templates stating the principle: do
not mutate the working tree, the index, HEAD, or branch state. Allow
inspecting other revisions via a separate temporary worktree, so the
read-only rule does not block legitimate historical comparison.

Closes #1538.
Closes #1543.
2026-06-16 10:09:44 -07:00
Jesse Vincent b0a0872774 @mhat reported that his claude got confused about 'debugging' being named as a skill in the bootstrap 2026-06-16 10:09:44 -07:00
Drew Ritter 94b5435617 Bump superpowers-evals submodule 2026-06-16 10:09:44 -07:00
Drew Ritter 1e7cd987d3 [codex] support native Codex plugin hooks (#1540)
* docs: specify Codex native hooks parity

* docs: refine Codex hooks spec after review

* docs: record Codex hook contract spike

* docs: plan Codex native hooks implementation

* feat: support Codex native plugin hooks

* test: add Codex native hook drill coverage

* Simplify Codex hook entrypoint
2026-06-16 10:09:44 -07:00
Drew Ritter 295219a6fa Align Pi mapping with action vocabulary 2026-06-16 10:09:44 -07:00
Drew Ritter db0396a7db Bump evals submodule for Pi backend 2026-06-16 10:09:44 -07:00
Jesse Vincent 3406f5d80f chore: keep pi extension under .pi 2026-06-16 10:09:44 -07:00
Jesse Vincent 71ac601627 feat: add pi superpowers package extension 2026-06-16 10:09:44 -07:00
Jesse Vincent 17a0cf12fa docs: plan pi extension and evals work 2026-06-16 10:09:43 -07:00
Drew Ritter f030d6ef88 Tighten cross-platform tool references 2026-06-16 10:09:43 -07:00
Jesse Vincent d7f47d350a Phase E: action-language tool vocabulary
Replace Claude-Code-specific tool names in skill prose, prompt
templates, and OpenCode-facing docs with action-language descriptions
that resolve to each runtime's native tool via the per-platform refs.

Changes by category:

- Prose mentions ("Use TodoWrite to track...", "Use Task tool with
  general-purpose type") → action language ("Track each item as a
  todo", "Dispatch a general-purpose subagent")

- Prompt template headers (6 files): "Task tool (general-purpose):"
  → "Subagent (general-purpose):" — preserves the type information
  without naming Claude Code's specific dispatch tool

- DOT flowchart node labels: "Invoke Skill tool" → "Invoke the
  skill"; "Create TodoWrite todo per item" → "Create a todo per
  item"

- OpenCode INSTALL.md and docs/README.opencode.md: replace the old
  "TodoWrite → todowrite, Task → @mention" mapping (which both
  taught a vocabulary skills no longer use AND was wrong about
  @mention being a real OpenCode syntax) with an action-language
  mapping verified against the installed OpenCode CLI's tool
  inventory.

The platform-tools refs landed in Phase B already document each
runtime's resolution; skills now speak in the actions those refs
map. Tool names that genuinely belong only in the per-platform
dispatch section ("In Claude Code: Use the `Skill` tool") and the
Claude-Code-specific Bash run_in_background flag note in
visual-companion remain — those are intentional carve-outs.
2026-06-16 10:09:43 -07:00
Jesse Vincent 6ec8686477 Phase D: cross-runtime tweaks (visual-companion, executing-plans, test)
Misc platform/runtime statements and adjacencies that don't fit the
prose, config-ref, README-ordering, or tool-vocabulary buckets:

- visual-companion frame template: rename CSS/HTML id #claude-content
  → #frame-content. The id is purely styling — nothing external
  references it. The brainstorm-server test that asserted the old
  string is updated in lockstep.

- visual-companion launch instructions: add a Copilot CLI section
  alongside Claude Code, Codex, and Gemini CLI; combine the Claude
  Code (macOS / Linux) and (Windows) sections so heading style
  matches the other (non-OS-qualified) platforms.

- visual-companion: "Use Write tool" → "Use your file-creation tool"
  for the cat/heredoc warning. The prohibition is what's load-
  bearing, not the tool name.

- executing-plans/SKILL.md: list all subagent-capable runtimes
  (Claude Code, Codex CLI, Codex App, Copilot CLI, Gemini CLI) and
  point at the per-platform tool refs as the source of truth.

- executing-plans/SKILL.md: relative path "using-superpowers/
  references/" → "../using-superpowers/references/" to resolve
  correctly from the executing-plans/ directory.

No bundled spec doc here — Phase D was scope-extension work that
took place across rounds, with no standalone spec authored.
2026-06-16 10:09:43 -07:00
Jesse Vincent 1681f58a3f Phase C: alphabetize README platform listings + spec
Quickstart link list and the per-harness install sub-sections both
reorder to strict alphabetical:

  Claude Code, Codex App, Codex CLI, Cursor, Factory Droid,
  Gemini CLI, GitHub Copilot CLI, OpenCode

Three blocks moved (Codex App swaps with Codex CLI; Cursor moves up
two slots; GitHub Copilot CLI moves up one). Claude Code stays first
by alphabetical chance.

Each install sub-section's content is byte-identical pre/post —
only the positions change. Quickstart anchors verified against the
new heading order.
2026-06-16 10:09:43 -07:00
Jesse Vincent 6b9f1b214a Phase B: config-file refs + per-platform tool refs + spec
Two structural changes:

1. Generalize CLAUDE.md-specific guidance:
   - "Project-specific conventions (put in CLAUDE.md)" → "(put in
     your instructions file)" in writing-skills/SKILL.md
   - "(explicit CLAUDE.md violation)" → "(explicit instruction-file
     violation)" in receiving-code-review/SKILL.md
   - The instruction-priority list in using-superpowers/SKILL.md
     stays inclusive (CLAUDE.md, GEMINI.md, AGENTS.md) — that's
     load-bearing, not a substitution opportunity.

2. Per-platform tool reference files at skills/using-superpowers/
   references/{claude-code,codex,copilot,gemini}-tools.md. Each ref
   documents:
   - The runtime's preferred instructions file (CLAUDE.md, AGENTS.md,
     GEMINI.md, etc.) and how it loads
   - The runtime's personal-skills directory + cross-runtime
     ~/.agents/skills/ path where applicable
   - Action-language → tool-name mapping table

Tool names and table content reflect the source-verified state from
direct inspection of openai/codex, google-gemini/gemini-cli,
sst/opencode, and the installed @github/copilot package. Filenames
and behaviors are sourced from each runtime's official docs.

Files in this commit also pick up later-phase changes that
accumulated on the same files (using-superpowers/SKILL.md "How to
Access Skills" overhaul, action-language flowchart, refs' final
table content). The bundled spec records original scope.
2026-06-16 10:09:43 -07:00
Jesse Vincent f0e5117fa6 Phase A: agent-neutral prose + CSO → SDO + spec
Replace generic third-person "Claude" with "agents" / "your agent"
forms across active skill prose, the README intro, and the vendored
anthropic-best-practices.md reference. Carve-outs preserved:
historical attribution paths, the "Variant C: Claude.AI Emphatic
Style" example label, model identifiers (Haiku/Sonnet/Opus), and the
"In Claude Code:" per-platform skill-dispatch list.

Coined-term rename: "Claude Search Optimization (CSO)" → "Skill
Discovery Optimization (SDO)" in writing-skills/SKILL.md.

Files in this commit also pick up later-phase changes that
accumulated on the same files (dispatching-parallel-agents code-
example transformation, writing-skills numbering and path fixes).
The bundled spec at docs/superpowers/specs/ records the original
scope and the carve-outs.

README.md gets only its prose change here; the alphabetization
lands in Phase C's commit.
2026-06-16 10:09:43 -07:00
Drew Ritter 741c232768 Move eval harness to submodule (#1541) 2026-06-16 10:09:43 -07:00
Drew Ritter 9ea7e2b6cb fix(tdd): link testing anti-patterns reference (#1532)
Fixes #1529.
2026-06-16 10:09:43 -07:00
Drew Ritter 0fad59e91f [codex] replace Circle K signal with generic review guidance (#1531)
* Remove Circle K signal from review skill

* Add generic review hesitation guidance

* Use Jesse wording for review hesitation guidance
2026-06-16 10:09:43 -07:00
Drew Ritter d00f4ad442 fix: remove global worktree path fallback (#1476) 2026-06-16 10:09:43 -07:00
Drew Ritter ce95985094 fix(using-git-worktrees): repair skipped Step 2 numbering (#1522) 2026-06-16 10:09:43 -07:00
fuleinist 98e39bd9e4 fix: remove stale Cursor plugin refs 2026-06-16 10:09:43 -07:00
Stable Genius fb1dfe9a16 fix(writing-skills): use markdown link for testing methodology reference 2026-06-16 10:09:43 -07:00
Drew Ritter bc2558c3f9 evals: use pre-commit hooks 2026-06-16 10:09:43 -07:00
Drew Ritter 9efbb7dd0d evals: add Gemini 2.5 Flash backend 2026-06-16 10:09:43 -07:00
Drew Ritter f7705f208e evals: drop drill source marker 2026-06-16 10:09:43 -07:00
Drew Ritter 74cddb5575 evals: remove unreleased wave scenarios 2026-06-16 10:09:43 -07:00
Jesse Vincent a325106502 Address adversarial review findings
- evals/README.md, evals/CLAUDE.md: fix uv install command from
  'uv sync --dev' to 'uv sync --extra dev'. Drill's pyproject.toml
  uses [project.optional-dependencies], so --dev is a no-op for
  pytest/ruff/ty; --extra dev is the correct invocation.
- tests/claude-code/run-skill-tests.sh: drop test-requesting-code-review.sh
  from integration_tests array (file deleted earlier in this branch).
- tests/claude-code/README.md: replace test-requesting-code-review.sh
  section with test-worktree-native-preference.sh (the worktree test
  is kept; the code-review test was lifted into drill).
- docs/testing.md, CLAUDE.md: remove "Copilot CLI" from the harness
  list. evals/backends/ has claude*, codex, gemini configs but no
  copilot.yaml, so the claim was unsupported.

Adversarial review credit: reviewer #2 found four legitimate issues
(uv-sync, run-skill-tests stale ref, README stale ref via #1, and
Copilot CLI fabrication); reviewer #1 found two distinct issues
(run-skill-tests + tests/claude-code/README.md). Reviewer #2 wins
this round.
2026-06-16 10:09:43 -07:00
Jesse Vincent 0e7b967e69 docs: introduce evals/ as the canonical skill-behavior eval harness
- docs/testing.md split into Plugin tests + Skill behavior evals.
  Plugin tests section enumerates the bash tests that survive
  (kept by drill-coverage analysis or as describe-skill tests).
- CLAUDE.md adds Eval harness section pointing at evals/.
- README.md Contributing section mentions evals/ alongside tests/.
- .gitignore adds evals/{results,.venv,.env} as belt-and-suspenders
  (evals/.gitignore covers these locally; root-level entries help
  tooling that does not recurse into nested ignore files).
2026-06-16 10:09:43 -07:00
Jesse Vincent 342ccf61d1 docs: annotate dated artifacts referencing lifted bash tests
- RELEASE-NOTES.md: note that test-requesting-code-review.sh and
  test-document-review-system.sh were lifted into drill scenarios
  on 2026-05-06; references are preserved as dated artifacts.
- docs/superpowers/plans/2026-03-23-codex-app-compatibility.md:
  note that tests/skill-triggering/ was lifted into drill scenarios
  on 2026-05-06; the run-all.sh reference is a dated artifact.

Subagent second-pass scrub confirmed no other active references in
the tree (excluding evals/ and the spec/plan for this work itself).
2026-06-16 10:09:43 -07:00
Jesse Vincent 315ef09ebc tests: annotate three kept bash tests with drill coverage notes
- test-worktree-native-preference.sh: drill covers PRESSURE phase only;
  RED + GREEN baselines have no drill counterpart and are kept so
  the RED-GREEN-REFACTOR validation remains rerunnable end-to-end.
- test-subagent-driven-development-integration.sh: drill covers the
  YAGNI subset (forbidden exports + reviewer-as-gate). Bash adds
  >=3 commits, >=2 subagent dispatches, TodoWrite usage, test file
  existence check, and token-budget telemetry. Kept until drill
  scenario covers those or they are retired.
- test-subagent-driven-development.sh: tests agent's ability to
  *describe* SDD (string matches against expected keywords). Drill
  scenarios test behavior, not description-recall. Kept by design.

Subagent verification recorded in commit messages of subsequent
deletions; gap analyses driving these annotations are also in the
verification subagent reports for the gating sweep.
2026-06-16 10:09:43 -07:00
Jesse Vincent 12ef68d55e tests: remove test-requesting-code-review.sh (covered by drill code-review-catches-planted-bugs)
Subagent verification: every bash assertion (skill invocation,
subagent dispatch, SQL injection flagged, credential handling
flagged, no merge approval) maps to drill verify checks. Drill is
stricter: bundles severity (Critical/Important) into the same
criteria as the finding itself (bash split severity into a separate
test). Setup parity covered (src/db.js with string concat + identity
hash, two commits).

The drill scenario header explicitly says it is the
"cross-harness, semantically-judged replacement for the bash test."
2026-06-16 10:09:43 -07:00
Jesse Vincent ea8aad8764 tests: remove test-document-review-system.sh (covered by drill spec-reviewer-catches-planted-flaws)
Subagent verification: every bash assertion (TODO in Requirements
section flagged, "specified later" deferral flagged, Issues section
present, did-not-approve verdict) maps to drill verify.criteria
entries. Setup parity covered by setup.assertions (test-feature-design.md
exists with TODO + 'specified later' content). Drill is stricter:
asserts tool-called Agent (subagent dispatch) which the bash test
did not check.
2026-06-16 10:09:43 -07:00
Jesse Vincent 1f0ad3817d tests: remove subagent-driven-dev fixtures (covered by drill sdd-go-fractals + sdd-svelte-todo)
The bash test had ZERO output assertions — it just ran claude -p
and printed token usage. Drill's scenarios are strictly more
rigorous:

go-fractals: skill-called SDD + tool-called Agent + go test ./...
passes + cmd/fractals/main.go exists + >=4 commits + LLM criteria
verifying real SDD workflow.

svelte-todo: skill-called SDD + tool-called Agent + npm test passes
+ playwright e2e passes + package.json + svelte.config.js or
vite.config.ts + >=4 commits + LLM criteria.

design.md and plan.md are byte-identical between bash fixtures and
drill fixtures (evals/fixtures/sdd-{go-fractals,svelte-todo}/).
Drill's setup helper (scaffold_sdd_*) forces git init -b main
(stricter than bash's reliance on init.defaultBranch). The
.claude/settings.local.json from bash scaffold.sh is unnecessary
for drill since permissions are managed via backend YAML.

Subagent verification: SAFE TO DELETE for both.
2026-06-16 10:09:43 -07:00
Jesse Vincent 7fd1ac7bfc tests: remove run-claude-describes-sdd.sh (covered by drill mid-conversation-skill-invocation)
Subagent verification: every bash assertion (Skill tool invoked +
specific skill name 'subagent-driven-development' loaded after the
agent describes it conversationally in turn 1) maps to the drill
scenario's skill-called assertion + criteria paragraph requiring
the skill to fire in direct response to the second user message.
Drill additionally asserts tool-called Agent (subagent dispatch)
which is stricter than the bash test.

Other runners in tests/explicit-skill-requests/ (haiku, multiturn,
extended-multiturn) and their prompt files are preserved — they
have no drill coverage and exercise different behaviors.
2026-06-16 10:09:43 -07:00
Jesse Vincent 8611a4ea97 tests: remove skill-triggering bash prompts (covered by drill triggering-* scenarios)
Subagent verification confirmed each prompt's intent matches its
corresponding drill scenario's turns[].intent verbatim, and each
scenario has both a deterministic skill-called assertion and a
semantic LLM criterion confirming the matching skill was loaded
(actually a stronger check than the bash test, which only confirms
the skill fires anywhere in the stream).

All 6 prompts deleted. The runner had no remaining prompts to drive,
so run-test.sh and run-all.sh deleted as well.
2026-06-16 10:09:43 -07:00
Jesse Vincent 09046c046b evals: drop SUPERPOWERS_ROOT setup step from README/CLAUDE
The cli.py helper now defaults the env var. Mention as override only.
2026-06-16 10:09:43 -07:00
Jesse Vincent 671ec3769d evals: drop SUPERPOWERS_ROOT from codex/gemini required_env
These backends only read SUPERPOWERS_ROOT via engine.py/setup.py's
os.environ access, which the new cli.py default helper supplies
automatically. claude*.yaml keep SUPERPOWERS_ROOT in required_env
because they interpolate ${SUPERPOWERS_ROOT} into --plugin-dir args.
2026-06-16 10:09:42 -07:00
Jesse Vincent 03cc20d3b5 evals: default SUPERPOWERS_ROOT to parent of evals/ if unset
Adds _set_superpowers_root_default() to drill/cli.py, called at
module import after load_dotenv(). PROJECT_ROOT resolves to evals/
post-lift; its parent is the superpowers repo root, which is the
correct value for SUPERPOWERS_ROOT.

Existing env values are respected as overrides via os.environ.setdefault.

Tests:
- helper sets default when var is unset
- helper does not override when var is already set
2026-06-16 10:09:42 -07:00
Jesse Vincent 6bc6f2279d Lift drill into evals/ at 013fcb8b7dbefd6d3fa4653493e5d2ec8e7f985b
rsync of obra/drill@013fcb8b7d into superpowers/evals/, excluding
.git/, .venv/, results/, .env/, __pycache__/, *.egg-info/,
.private-journal/.

The drill repo is unaffected by this commit; archival is a separate
manual step after this PR merges.

Source SHA recorded at evals/.drill-source-sha for divergence
detection.
2026-06-16 10:09:42 -07:00
Jesse Vincent 1a42ead98f Plan: lift drill into superpowers as evals/
15-task implementation plan derived from the design spec at
docs/superpowers/specs/2026-05-06-lift-drill-into-evals-design.md.

Each task is bite-sized (2-5 min steps) with exact commands, exact
file paths, and exact code where required. Subagent verification
gates per the spec are written out as concrete prompt templates.

Self-review:
- Spec coverage: every spec section maps to a task
- Placeholder scan: no TBD/TODO/placeholder/fill-in-later language
- Type consistency: helper named _set_superpowers_root_default
  consistently; drill SHA recorded in evals/.drill-source-sha
  consistently
2026-06-16 10:09:01 -07:00
Jesse Vincent 09d2c1d39c Spec: address adversarial review findings
Two parallel reviewers raised legitimate issues against the lift-drill-
into-evals spec. Updates:

- Coverage map for tests/explicit-skill-requests/ corrected: 6 run-*.sh
  scripts + prompts, not "2 scenarios cover all". Several scripts
  (Haiku, multi-turn, please-use-brainstorming, use-systematic-debugging)
  have no drill counterpart and stay.
- tests/claude-code/test-subagent-driven-development.sh marked as
  meta/documentation test (asks agent to describe SDD); no drill
  scenario covers description tests; defaults to keep.
- Path-defaults section now shows verified evidence: PROJECT_ROOT
  resolves to evals/ post-move; only claude*.yaml substitute
  ${SUPERPOWERS_ROOT} in args (codex/gemini use it via os.environ
  in pre-run hooks); helper invocation order specified (after
  load_dotenv, before click definitions).
- Step 2 copy uses explicit rsync excludes (.git, .venv, results,
  .env, __pycache__, *.egg-info, .private-journal); checksum-level
  verification rather than file-count.
- Drill SHA recorded at copy time in commit message and
  evals/.drill-source-sha for divergence detection.
- evals/tests/ pytest suite added to verification protocol.
- Reference scrub list expanded: RELEASE-NOTES.md,
  docs/superpowers/plans/, .codex-plugin/ (corrected from .codex/),
  lefthook.yml. Excluded dirs called out (node_modules/, .venv/,
  evals/).
- Historical plan docs / RELEASE-NOTES handling: annotate, don't
  rewrite.
- evals/lefthook.yml move documented (drill ships its own;
  contributors run cd evals && lefthook run pre-commit manually).
- PR description checklist includes archival action item for
  obra/drill post-merge.

False finding rejected: svelte-todo fixture is complete on disk
(design.md + plan.md + scaffold.sh present); reviewer #1 #3 dropped.
2026-06-16 10:09:01 -07:00
Jesse Vincent bce1267adb Spec: lift drill into superpowers as evals/
Records scope, branching, architecture, deletion gate, verification
protocol, path/config edits, migration ordering, and post-implementation
verification. Frames CI integration, scenario co-location, and Python
package rename as deferred work.

Per-file deletion of bash tests under superpowers/tests/ is gated by a
subagent that compares each bash assertion to its drill scenario's
verify block. Default keeps the bash test if any assertion is unmatched.

Branching: independent off dev (f/evals-lift), not stacked on
f/cross-platform.
2026-06-16 10:09:01 -07:00
robotsnh 718cb1d78c docs: turned the dash in "- Jesse" into an escape sequence (#1474)
Replaced the bullet point next to "Jesse" in the sponsorship section of the `README` into a dash. This is needed so the `README` renders properly on markdown viewers.
2026-06-16 10:09:01 -07:00
Jesse Vincent 8cf3900614 Job posting 2026-06-15 12:18:22 -07:00
Jesse Vincent 6fd4507659 Require contributors to disclose authoring environment and target dev
Add a mandatory self-identification disclosure (model, harness, harness
version, all installed plugins) to the PR template and all three issue
templates, and document the requirement in the contributor guidelines.
We weigh contributions differently depending on what produced them:
content reasoned from documentation is held to a different bar than work
grounded in a real session.

Also state explicitly, in both CLAUDE.md and the PR template, that all
PRs must target the dev branch rather than main.
2026-05-29 13:05:25 -07:00
215 changed files with 10736 additions and 13193 deletions
+1 -1
View File
@@ -9,7 +9,7 @@
{
"name": "superpowers",
"description": "Core skills library for Claude Code: TDD, debugging, collaboration patterns, and proven techniques",
"version": "5.1.0",
"version": "6.0.0",
"source": "./",
"author": {
"name": "Jesse Vincent",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "superpowers",
"description": "Core skills library for Claude Code: TDD, debugging, collaboration patterns, and proven techniques",
"version": "5.1.0",
"version": "6.0.0",
"author": {
"name": "Jesse Vincent",
"email": "jesse@fsck.com"
+2 -1
View File
@@ -1,6 +1,6 @@
{
"name": "superpowers",
"version": "5.1.0",
"version": "6.0.0",
"description": "An agentic skills framework & software development methodology that works: planning, TDD, debugging, and collaboration workflows.",
"author": {
"name": "Jesse Vincent",
@@ -21,6 +21,7 @@
"workflow"
],
"skills": "./skills/",
"hooks": "./hooks/hooks-codex.json",
"interface": {
"displayName": "Superpowers",
"shortDescription": "Planning, TDD, debugging, and delivery workflows for coding agents",
+1 -1
View File
@@ -2,7 +2,7 @@
"name": "superpowers",
"displayName": "Superpowers",
"description": "Core skills library: TDD, debugging, collaboration patterns, and proven techniques",
"version": "5.1.0",
"version": "6.0.0",
"author": {
"name": "Jesse Vincent",
"email": "jesse@fsck.com"
+5 -2
View File
@@ -12,14 +12,17 @@ add a comment or reaction to the existing one instead.
- [ ] I searched existing issues and this is not a duplicate
## Environment
## Environment (required)
<!-- Required. We assume an agent filed this report — tell us which one and
where it ran. We weigh reports by what produced them. -->
| Field | Value |
|-------|-------|
| Superpowers version | |
| Harness (Claude Code, Cursor, etc.) | |
| Harness version | |
| Model | |
| Your model + version | |
| All plugins installed | |
| OS + shell | |
## Is this a Superpowers issue or a platform issue?
+14 -1
View File
@@ -30,5 +30,18 @@ progress, and some were intentionally declined.
of project? If this is specific to your domain, workflow, or a
third-party tool, it may belong as its own plugin instead. -->
## Environment (required)
<!-- Required. We assume an agent wrote this request — tell us which one and
where it ran. We weigh proposals reasoned from documentation differently
than ones grounded in a real session where the problem actually came up. -->
| Field | Value |
|-------|-------|
| Superpowers version | |
| Harness (Claude Code, Cursor, etc.) | |
| Harness version | |
| Your model + version | |
| All plugins installed | |
## Context
<!-- Optional: version info, harness, model, workflow where you hit this. -->
<!-- Optional: the workflow where you hit this, links, transcripts. -->
@@ -21,3 +21,14 @@ requested or discussed.
## Have you tried manual installation?
<!-- Many tools work with Superpowers through manual setup even without
official support. Did you try? What happened? -->
## Environment (required)
<!-- Required. We assume an agent wrote this request — tell us which one and
where it ran. -->
| Field | Value |
|-------|-------|
| Harness you currently use (Claude Code, Cursor, etc.) | |
| Harness version | |
| Your model + version | |
| All plugins installed | |
+17
View File
@@ -4,6 +4,23 @@ sections blank, contain multiple unrelated changes, or show no evidence
of human involvement will be closed without review.
-->
> **This PR MUST target the `dev` branch, not `main`.** `main` is the
> released branch; active work lands on `dev` first. PRs opened against
> `main` will be asked to retarget `dev` before review.
## Who is submitting this PR? (required)
<!-- Required. PRs that omit this will be closed. We assume an agent wrote
this PR — tell us which one and where it ran. We weigh contributions by
what produced them: content reasoned from documentation is held to a
different bar than work grounded in a real session. -->
| Field | Value |
|-------|-------|
| Your model + version | |
| Harness + version | |
| All plugins installed | |
| Human partner who reviewed this diff | |
## What problem are you trying to solve?
<!-- Describe the specific problem you encountered. If this was a session
issue, include: what you were doing, what went wrong, the model's
+1
View File
@@ -1,6 +1,7 @@
.worktrees/
.private-journal/
.claude/
.superpowers/
.DS_Store
node_modules/
inspo
+3
View File
@@ -0,0 +1,3 @@
[submodule "evals"]
path = evals
url = git@github.com:prime-radiant-inc/superpowers-evals.git
+38
View File
@@ -0,0 +1,38 @@
{
"name": "superpowers",
"version": "6.0.0",
"description": "An agentic skills framework and software development methodology.",
"author": {
"name": "Jesse Vincent",
"email": "jesse@fsck.com"
},
"homepage": "https://github.com/obra/superpowers",
"license": "MIT",
"keywords": [
"brainstorming",
"subagent-driven-development",
"skills",
"planning",
"tdd",
"debugging",
"code-review",
"workflow"
],
"skills": "./skills/",
"sessionStart": {
"skill": "using-superpowers"
},
"skillInstructions": "Kimi Code tool mapping for Superpowers skills:\n\n- When a Superpowers skill says to ask the user, ask clarifying questions, ask one question at a time, present multiple-choice options, use the terminal for a question, or wait for the user's choice, call Kimi Code's `AskUserQuestion` tool. Do not render those choices as plain assistant text unless `AskUserQuestion` is unavailable or the session is in auto permission mode.\n- For `AskUserQuestion`, provide 1 question with 2-4 concrete options when possible. Put the recommended option first and suffix its label with `(Recommended)`.\n- When a Superpowers skill refers to `TodoWrite`, use Kimi Code's `TodoList` tool.\n- When a Superpowers skill says `Task tool (general-purpose)` or asks you to dispatch an implementer/reviewer subagent, use Kimi Code's `Agent` tool with a Kimi subagent type. Do not pass `general-purpose` as `subagent_type`.\n- For implementation, code review, spec review, quality review, and filled Superpowers subagent prompt templates, call `Agent` with `subagent_type: \"coder\"`, paste the fully filled prompt into `prompt`, and provide a short `description`.\n- For read-only codebase exploration that would take several searches, use `Agent` with `subagent_type: \"explore\"`.\n- For read-only planning or architecture design, use `Agent` with `subagent_type: \"plan\"`.\n- Keep dependent Superpowers subagent steps sequential. Use multiple `Agent` calls, or `run_in_background: true` only when the work is independent and background agents are available.\n- When a Superpowers skill refers to the `Skill` tool, use Kimi Code's native `Skill` tool.\n- Use Kimi Code's `Read`, `Write`, `Edit`, `Bash`, `Grep`, `Glob`, `FetchURL`, `WebSearch`, and MCP tools by their actual exposed names.\n- When a skill asks to search file contents, use `Grep`; when it asks to find files by path or pattern, use `Glob`; when it asks to fetch a URL, use `FetchURL`; when it asks to search the web, use `WebSearch`.",
"interface": {
"displayName": "Superpowers",
"shortDescription": "Planning, TDD, debugging, and delivery workflows for coding agents",
"longDescription": "Use Superpowers to guide agent work through brainstorming, implementation planning, test-driven development, systematic debugging, parallel execution, code review, and finish-the-branch workflows.",
"developerName": "Jesse Vincent",
"capabilities": [
"Interactive",
"Read",
"Write"
],
"websiteURL": "https://github.com/obra/superpowers"
}
}
+11 -6
View File
@@ -45,7 +45,7 @@ Use OpenCode's native `skill` tool:
```
use skill tool to list skills
use skill tool to load superpowers/brainstorming
use skill tool to load brainstorming
```
## Updating
@@ -98,11 +98,16 @@ Then use the installed package path in `opencode.json`:
### Tool mapping
When skills reference Claude Code tools:
- `TodoWrite``todowrite`
- `Task` with subagents → `@mention` syntax
- `Skill` tool → OpenCode's native `skill` tool
- File operations → your native tools
Skills speak in actions ("create a todo", "dispatch a subagent", "read a file"). On OpenCode these resolve to:
- "Create a todo" / "mark complete in todo list" → `todowrite`
- `Subagent (general-purpose):` template → `task` tool with `subagent_type: "general"` (or `"explore"` for codebase exploration)
- "Invoke a skill" → OpenCode's native `skill` tool
- "Read a file" → `read`
- "Create a file" / "edit a file" / "delete a file" → `apply_patch`
- "Run a shell command" → `bash`
- "Search file contents" / "find files by name" → `grep`, `glob`
- "Fetch a URL" → `webfetch`
## Getting Help
+10 -6
View File
@@ -1,7 +1,7 @@
/**
* Superpowers plugin for OpenCode.ai
*
* Injects superpowers bootstrap context via system prompt transform.
* Injects superpowers bootstrap context via message transform.
* Auto-registers skills directory via config hook (no symlinks needed).
*/
@@ -74,11 +74,15 @@ export const SuperpowersPlugin = async ({ client, directory }) => {
const { content } = extractAndStripFrontmatter(fullContent);
const toolMapping = `**Tool Mapping for OpenCode:**
When skills reference tools you don't have, substitute OpenCode equivalents:
- \`TodoWrite\`\`todowrite\`
- \`Task\` tool with subagents → Use OpenCode's subagent system (@mention)
- \`Skill\` tool → OpenCode's native \`skill\` tool
- \`Read\`, \`Write\`, \`Edit\`, \`Bash\` → Your native tools
When skills request actions, substitute OpenCode equivalents:
- Create or update todos\`todowrite\`
- \`Subagent (general-purpose):\`\`task\` with \`subagent_type: "general"\`
- Invoke a skill → OpenCode's native \`skill\` tool
- Read files → \`read\`
- Create, edit, or delete files → \`apply_patch\`
- Run shell commands → \`bash\`
- Search files → \`grep\`, \`glob\`
- Fetch a URL → \`webfetch\`
Use OpenCode's native \`skill\` tool to list and load skills.`;
+121
View File
@@ -0,0 +1,121 @@
import { readFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
const EXTREMELY_IMPORTANT_MARKER = "<EXTREMELY_IMPORTANT>";
const BOOTSTRAP_MARKER = "superpowers:using-superpowers bootstrap for pi";
const extensionDir = dirname(fileURLToPath(import.meta.url));
const packageRoot = resolve(extensionDir, "../..");
const skillsDir = resolve(packageRoot, "skills");
const bootstrapSkillPath = resolve(skillsDir, "using-superpowers", "SKILL.md");
let cachedBootstrap: string | null | undefined;
export default function superpowersPiExtension(pi: ExtensionAPI) {
let injectBootstrap = true;
pi.on("resources_discover", async () => ({
skillPaths: [skillsDir],
}));
pi.on("session_start", async () => {
injectBootstrap = true;
});
pi.on("session_compact", async () => {
injectBootstrap = true;
});
pi.on("agent_end", async () => {
injectBootstrap = false;
});
pi.on("context", async (event) => {
if (!injectBootstrap) return;
if (event.messages.some(messageContainsBootstrap)) return;
const bootstrap = getBootstrapContent();
if (!bootstrap) return;
const bootstrapMessage = {
role: "user" as const,
content: [{ type: "text" as const, text: bootstrap }],
timestamp: Date.now(),
};
const insertAt = firstNonCompactionSummaryIndex(event.messages);
return {
messages: [
...event.messages.slice(0, insertAt),
bootstrapMessage,
...event.messages.slice(insertAt),
],
};
});
}
function getBootstrapContent(): string | null {
if (cachedBootstrap !== undefined) return cachedBootstrap;
try {
const skillContent = readFileSync(bootstrapSkillPath, "utf8");
const body = stripFrontmatter(skillContent);
cachedBootstrap = `${EXTREMELY_IMPORTANT_MARKER}
${BOOTSTRAP_MARKER}
You have superpowers.
The using-superpowers skill content is included below and is already loaded for this Pi session. Follow it now. Do not try to load using-superpowers again.
${body}
${piToolMapping()}
</EXTREMELY_IMPORTANT>`;
return cachedBootstrap;
} catch {
cachedBootstrap = null;
return null;
}
}
function stripFrontmatter(content: string): string {
const match = content.match(/^---\n[\s\S]*?\n---\n([\s\S]*)$/);
return (match ? match[1] : content).trim();
}
function piToolMapping(): string {
return `## Pi tool mapping
Pi has native skills but does not expose Claude Code's \`Skill\` tool. When a Superpowers instruction says to invoke a skill, use Pi's native skill system instead: load the relevant \`SKILL.md\` with \`read\` when the skill applies, or let a human invoke \`/skill:name\` explicitly.
Pi's built-in coding tools are lowercase: \`read\`, \`write\`, \`edit\`, \`bash\`, plus optional \`grep\`, \`find\`, and \`ls\`. Use those for the corresponding actions: read a file, create or edit files, run shell commands, search file contents, find files by name, and list directories.
Pi does not ship a standard subagent tool. If a subagent tool such as \`subagent\` from \`pi-subagents\` is available, use it for Superpowers subagent workflows. If no subagent tool is available, do the work in this session or explain the missing capability instead of inventing \`Task\` calls.
Pi does not ship a standard task-list tool. If an installed todo/task tool is available, use it. Otherwise track work in plan files or a repo-local \`TODO.md\` when task tracking is needed. Treat older \`TodoWrite\` references as this task-tracking action.`;
}
function messageContainsBootstrap(message: unknown): boolean {
const content = (message as { content?: unknown }).content;
if (typeof content === "string") return content.includes(BOOTSTRAP_MARKER);
if (!Array.isArray(content)) return false;
return content.some((part) => {
return (
part &&
typeof part === "object" &&
(part as { type?: unknown }).type === "text" &&
typeof (part as { text?: unknown }).text === "string" &&
(part as { text: string }).text.includes(BOOTSTRAP_MARKER)
);
});
}
function firstNonCompactionSummaryIndex(messages: unknown[]): number {
let index = 0;
while ((messages[index] as { role?: unknown } | undefined)?.role === "compactionSummary") {
index += 1;
}
return index;
}
+1
View File
@@ -4,6 +4,7 @@
{ "path": ".claude-plugin/plugin.json", "field": "version" },
{ "path": ".cursor-plugin/plugin.json", "field": "version" },
{ "path": ".codex-plugin/plugin.json", "field": "version" },
{ "path": ".kimi-plugin/plugin.json", "field": "version" },
{ "path": ".claude-plugin/marketplace.json", "field": "plugins.0.version" },
{ "path": "gemini-extension.json", "field": "version" }
],
+7 -2
View File
@@ -14,7 +14,8 @@ Before you open a PR against this repo, you MUST:
2. **Search for existing PRs** — open AND closed — that address the same problem. If duplicates exist, STOP and tell your human partner. Do not open another duplicate.
3. **Verify this is a real problem.** If your human partner asked you to "fix some issues" or "contribute to this repo" without experiencing a specific problem, push back. Ask them what broke, what failed, what the user experience was.
4. **Confirm the change belongs in core.** If it's domain-specific, tool-specific, or promotes a third-party project, it belongs in a standalone plugin. Tell your human partner.
5. **Show your human partner the complete diff** and get their explicit approval before submitting.
5. **Identify yourself.** Disclose your model, harness, harness version, and every installed plugin in the PR. Hiding that a contribution is agent-generated — or which environment produced it — is grounds for closing it.
6. **Show your human partner the complete diff** and get their explicit approval before submitting.
If any of these checks fail, do not open the PR. Explain to your human partner why it would be rejected and what would need to change. They will thank you for saving them the embarrassment.
@@ -26,6 +27,10 @@ If any of these checks fail, do not open the PR. Explain to your human partner w
**PRs that show no evidence of human involvement will be closed.** A human must review the complete proposed diff before submission.
**Submitters MUST identify themselves.** Every PR and issue must disclose the model, harness, harness version, and all installed plugins used to produce the contribution — or state plainly that it was written by hand with no agent. This is not optional. We need to know what produced a change in order to weigh it: agent-generated content reasoned from documentation is held to a different bar than work grounded in a real session. Contributions that hide their authoring environment will be closed.
**All PRs MUST target the `dev` branch, not `main`.** `main` is the released branch; active work lands on `dev` first. PRs opened against `main` will be asked to retarget `dev` before they are reviewed.
## What We Will Not Accept
### Third-party dependencies
@@ -96,7 +101,7 @@ Skills are not prose — they are code that shapes agent behavior. If you modify
## Eval harness
Skill-behavior evals live at `evals/` see `evals/README.md`. Drill (the harness) drives real tmux sessions of Claude Code / Codex / Gemini CLI and judges skill compliance with an LLM verifier. Plugin-infrastructure tests still live at `tests/`.
Skill-behavior evals live in the `evals/` submodule — after cloning, run `git submodule update --init evals`, then see `evals/README.md`. Drill (the harness) drives real tmux sessions of Claude Code / Codex / Gemini CLI and judges skill compliance with an LLM verifier. Plugin-infrastructure tests still live at `tests/`.
## Understand the Project Before Contributing
+91 -39
View File
@@ -2,9 +2,16 @@
Superpowers is a complete software development methodology for your coding agents, built on top of a set of composable skills and some initial instructions that make sure your agent uses them.
## We're Hiring!
We're hiring someone to help out full time with Superpowers community and code work.
You can read about the job at https://primeradiant.com/jobs/superpowers-community-engineer/
If this sounds like someone you know, definitely send them our way.
## Quickstart
Give your agent Superpowers: [Claude Code](#claude-code), [Codex CLI](#codex-cli), [Codex App](#codex-app), [Factory Droid](#factory-droid), [Gemini CLI](#gemini-cli), [OpenCode](#opencode), [Cursor](#cursor), [GitHub Copilot CLI](#github-copilot-cli).
Give your agent Superpowers: [Claude Code](#claude-code), [Antigravity](#antigravity), [Codex App](#codex-app), [Codex CLI](#codex-cli), [Cursor](#cursor), [Factory Droid](#factory-droid), [Gemini CLI](#gemini-cli), [GitHub Copilot CLI](#github-copilot-cli), [Kimi Code](#kimi-code), [OpenCode](#opencode), [Pi](#pi).
## How it works
@@ -14,19 +21,13 @@ Once it's teased a spec out of the conversation, it shows it to you in chunks sh
After you've signed off on the design, your agent puts together an implementation plan that's clear enough for an enthusiastic junior engineer with poor taste, no judgement, no project context, and an aversion to testing to follow. It emphasizes true red/green TDD, YAGNI (You Aren't Gonna Need It), and DRY.
Next up, once you say "go", it launches a *subagent-driven-development* process, having agents work through each engineering task, inspecting and reviewing their work, and continuing forward. It's not uncommon for Claude to be able to work autonomously for a couple hours at a time without deviating from the plan you put together.
Next up, once you say "go", it launches a *subagent-driven-development* process, having agents work through each engineering task, inspecting and reviewing their work, and continuing forward. It's not uncommon for your agent to work autonomously for a couple hours at a time without deviating from the plan you put together.
There's a bunch more to it, but that's the core of the system. And because the skills trigger automatically, you don't need to do anything special. Your coding agent just has Superpowers.
## Commercial Services
## Sponsorship
If Superpowers has helped you do stuff that makes money and you are so inclined, I'd greatly appreciate it if you'd consider [sponsoring my opensource work](https://github.com/sponsors/obra).
Thanks!
\- Jesse
If you're using Superpowers in enterprise and could benefit from commercial support, additional tooling, or managed spending, please don't hesitate to drop us a line at sales@primeradiant.com.
## Installation
@@ -60,6 +61,25 @@ The Superpowers marketplace provides Superpowers and some other related plugins
/plugin install superpowers@superpowers-marketplace
```
### Antigravity
Install Superpowers as a plugin from this repository:
```bash
agy plugin install https://github.com/obra/superpowers
```
Antigravity runs the plugin's session-start hook, so Superpowers is active from
the first message. Reinstall with the same command to update.
### Codex App
Superpowers is available via the [official Codex plugin marketplace](https://github.com/openai/plugins).
- In the Codex app, click on Plugins in the sidebar.
- You should see `Superpowers` in the Coding section.
- Click the `+` next to Superpowers and follow the prompts.
### Codex CLI
Superpowers is available via the [official Codex plugin marketplace](https://github.com/openai/plugins).
@@ -78,13 +98,15 @@ Superpowers is available via the [official Codex plugin marketplace](https://git
- Select `Install Plugin`.
### Codex App
### Cursor
Superpowers is available via the [official Codex plugin marketplace](https://github.com/openai/plugins).
- In Cursor Agent chat, install from marketplace:
- In the Codex app, click on Plugins in the sidebar.
- You should see `Superpowers` in the Coding section.
- Click the `+` next to Superpowers and follow the prompts.
```text
/add-plugin superpowers
```
- Or search for "superpowers" in the plugin marketplace.
### Factory Droid
@@ -114,29 +136,6 @@ Superpowers is available via the [official Codex plugin marketplace](https://git
gemini extensions update superpowers
```
### OpenCode
OpenCode uses its own plugin install; install Superpowers separately even if you
already use it in another harness.
- Tell OpenCode:
```
Fetch and follow instructions from https://raw.githubusercontent.com/obra/superpowers/refs/heads/main/.opencode/INSTALL.md
```
- Detailed docs: [docs/README.opencode.md](docs/README.opencode.md)
### Cursor
- In Cursor Agent chat, install from marketplace:
```text
/add-plugin superpowers
```
- Or search for "superpowers" in the plugin marketplace.
### GitHub Copilot CLI
- Register the marketplace:
@@ -151,6 +150,55 @@ already use it in another harness.
copilot plugin install superpowers@superpowers-marketplace
```
### Kimi Code
Superpowers is available in Kimi Code's plugin marketplace.
- Open Kimi Code's plugin manager:
```text
/plugins
```
- Go to `Marketplace` > `Superpowers` and install it.
- Or install directly from this repository:
```text
/plugins install https://github.com/obra/superpowers
```
- Detailed docs: [docs/README.kimi.md](docs/README.kimi.md)
### OpenCode
OpenCode uses its own plugin install; install Superpowers separately even if you
already use it in another harness.
- Tell OpenCode:
```
Fetch and follow instructions from https://raw.githubusercontent.com/obra/superpowers/refs/heads/main/.opencode/INSTALL.md
```
- Detailed docs: [docs/README.opencode.md](docs/README.opencode.md)
### Pi
Install Superpowers as a Pi package from this repository:
```bash
pi install git:github.com/obra/superpowers
```
For local development, run Pi with this checkout loaded as a temporary package:
```bash
pi -e /path/to/superpowers
```
The Pi package loads the Superpowers skills and a small extension that injects the `using-superpowers` bootstrap at session startup and again after compaction. Pi has native skills, so no compatibility `Skill` tool is required. Subagent and task-list tools remain optional Pi companion packages.
## The Basic Workflow
1. **brainstorming** - Activates before writing code. Refines rough ideas through questions, explores alternatives, presents design in sections for validation. Saves design document.
@@ -214,7 +262,7 @@ The general contribution process for Superpowers is below. Keep in mind that we
4. Follow the `writing-skills` skill for creating and testing new and modified skills
5. Submit a PR, being sure to fill in the pull request template.
Skill-behavior tests use the eval harness at `evals/`. See `evals/README.md` for setup. Plugin-infrastructure tests live at `tests/` and run via the relevant `run-*.sh` or `npm test`.
Skill-behavior tests use the eval harness submodule at `evals/`. After cloning this repo, run `git submodule update --init evals`, then see `evals/README.md` for setup. Plugin-infrastructure tests live at `tests/` and run via the relevant `run-*.sh` or `npm test`.
See `skills/writing-skills/SKILL.md` for the complete guide.
@@ -226,6 +274,10 @@ Superpowers updates are somewhat coding-agent dependent, but are often automatic
MIT License - see LICENSE file for details
## Visual companion telemetry
Because skills and plugins don't provide any feedback to creators, we have no idea how many of you are using Superpowers. By default, the Prime Radiant logo on brainstorming's optional visual companion feature is loaded from our website. It includes the version of Superpowers in use. It does not include any details about your project, prompt, or coding agent. We don't see your clicks or anything about what you're building. This helps us have a rough idea of how many folks are using Superpowers and which version of Superpowers they're using. It's 100% optional. To disable this, set the environment variable `SUPERPOWERS_DISABLE_TELEMETRY` to any true value. Superpowers also honors Claude Code's `DISABLE_TELEMETRY` and `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` opt-outs.
## Community
Superpowers is built by [Jesse Vincent](https://blog.fsck.com) and the rest of the folks at [Prime Radiant](https://primeradiant.com).
+98
View File
@@ -1,5 +1,103 @@
# Superpowers Release Notes
## v6.0.0 (2026-06-16)
Superpowers 6.0 is a big release. The headline is a rewrite of how `subagent-driven-development` reviews each task — cheaper, stricter, and harder to game.
While these numbers won't hold on every harness and for every workload, in our evals, Claude Code and Codex produce similar high-quality results roughly twice as fast and while spending almost 50% fewer tokens.
It also adds three new harnesses (Kimi Code, Pi, and Antigravity), gives the brainstorming visual companion a better security model, and rewrites a number of skills' tool calls to be significantly more vendor-neutral.
### Visible Changes
- **The two per-task reviewer prompts became one.** `spec-reviewer-prompt.md` and `code-quality-reviewer-prompt.md` are gone, replaced by a single `task-reviewer-prompt.md`. If you dispatch the old files directly, switch to the new one.
- **The legacy global worktree directory is gone.** `using-git-worktrees` and `finishing-a-development-branch` no longer use `~/.config/superpowers/worktrees/`. Worktrees now land in the project — an existing `.worktrees/` or `worktrees/` if you have one, otherwise a fresh `.worktrees/` — unless you say otherwise.
### New Harness Support
Superpowers now runs on three more harnesses. Each ships its own bootstrap, a tool-mapping reference, and tests, and each gets its own install section in the README.
- **Kimi Code** — a plugin manifest, install docs, and manifest tests; install from Kimi's marketplace or straight from the repo. (initial manifest by @qer)
- **Pi** — a session-start extension that registers the skills and injects the `using-superpowers` bootstrap. Pi has native skills, so it needs no compatibility shim.
- **Antigravity (`agy`)** — installs the plugin directly and bootstraps from the first message; verified end-to-end against the standard "make a react todo list" acceptance test.
### Subagent-Driven Development
A long run of cost-and-quality experiments on real projects reshaped how the controller reviews each task. The old flow ran two reviewers per task and leaned on the controller's judgment for model choice and severity, and both turned out to be expensive and easy to game. The new flow runs one reviewer per task, hands work off as files instead of pasted text, and takes several judgment calls away from the controller.
- **One reviewer per task, two verdicts.** A single `task-reviewer-prompt.md` reads the task's diff once and returns both a spec-compliance verdict and a quality verdict, so one fix pass clears both. A new "can't verify from the diff" verdict flags requirements that live in untouched code, for the controller to check itself. (#1538, #1543)
- **One broad review at the end.** The run finishes with a single whole-branch review on the most capable model, instead of re-reviewing everything task by task.
- **Plans get a pre-flight read.** Before the first task, the controller checks the plan for internal conflicts — and for anything the plan asks for that a reviewer would flag as a defect — and raises it all at once, rather than stumbling into it mid-run.
- **Diffs and task text move as files.** A pasted diff parks itself permanently in the most expensive context, and a reviewer without one rebuilds it by hand — the single biggest reviewer cost. Two new scripts, `task-brief` and `review-package`, write the task text and the review diff to files for the subagent to read.
- **Every dispatch states its model.** Left to choose, controllers stopped naming a model at all — and an unnamed model quietly inherits the session's most expensive one, so one run put all 26 of its reviewers on the top tier. The templates now require a model, with guidance that reaches for cheaper tiers when the work allows.
- **The controller can't tell a reviewer what to ignore.** Real runs caught controllers coaching reviewers to skip a finding or call it "Minor at most," and the flaw shipped. Suppressing findings and pre-rating severity are now banned outright, and a defect the plan itself mandates gets reported for you to decide on rather than waved through.
- **Reviewers are read-only and skeptical of rationales.** Review no longer touches the working tree or branch — a reviewer running `git checkout` had been orphaning later commits — and an implementer's "I left this unabstracted on purpose" no longer talks a reviewer out of a real finding.
- **Stronger evidence and reporting.** Reviewers back each answer with a file and line, the implementer's report moves to a file and carries red/green evidence when TDD applies, and a progress ledger lets a controller that loses its context resume instead of redoing finished work. (#994)
### Writing Plans
Plans now carry the structure the controller and reviewers used to re-derive on every dispatch.
- **A Global Constraints block** lists the rules that bind every task — version floors, dependency limits, naming and copy, exact values — copied in verbatim, so they actually reach the implementers and reviewers downstream.
- **A per-task Interfaces block** names exactly what each task consumes and produces, so an implementer who sees only its own task still knows its neighbors' contracts.
- **Right-sizing guidance** keeps a task at the size that earns its own test cycle and a reviewer's pass, folding setup, config, and docs into the task that needs them. In testing, a plan written this way needed one round of fixes where the control needed two to four — and the control shipped a real bug.
### Brainstorming Visual Companion
The visual companion is a small web server the agent opens alongside the conversation. It had no authentication at all, so on a shared or remote machine anyone who could reach the port could read your brainstorm — or inject events the agent treats as your input. This release gives it a real security model and makes it survive restarts and dropped connections.
- **A per-session key now guards everything.** The agent's URL carries a one-time key, the browser tucks it into a tab-scoped cookie, and every request and WebSocket connection has to present it. This closes the door to stray local tabs and routable remote hosts alike, including the DNS-rebinding case an origin allowlist can't catch. (Closes #1014)
- **The file server stays in its sandbox.** It refuses symlinks, dotfiles, and any path that climbs out of the content directory, ignores macOS resource-fork files, and sends the usual no-store and deny-framing headers. Files that hold the session key are written owner-only.
- **The companion is offered only when it helps.** The skill raises it the first time a question would read better shown than told, as its own message, and lets a decline stand. Accepting opens your browser to the first screen. (Closes #755)
- **It survives restarts and flaky connections.** Given a project directory, the server keeps the same port and key across restarts, so an open tab simply reconnects. The page reconnects on its own, shows a live status pill, and raises a "paused" overlay while the server is down.
- **Longer idle life, safer shutdown.** The idle timeout went from 30 minutes to 4 hours, and `stop-server.sh` now confirms it owns the right process before signaling, so it never kills an unrelated `node` after a reboot. (#1703)
- **Windows launch hardening** — consolidated shell detection, and Windows now relies on the idle timeout for shutdown, since Node can't track POSIX process ownership across MSYS2.
### Existing Harness Updates
- **Codex** now bootstraps through its own SessionStart hook rather than shared wiring, and the Codex App gained an install section and fuller tool docs (web search, `AGENTS.md`, personal skills). (#1540)
- **OpenCode** got an action-based tool mapping across its plugin, install doc, and README, plus a bootstrap-caching test.
- **Cursor**'s manifest dropped its `agents` and `commands` entries, since those directories no longer exist.
### One Set of Skills, Every Harness
The skills used to speak Claude Code's dialect — "use the Task tool," "put it in CLAUDE.md." This release rewrites that vocabulary in terms of what you're actually doing ("dispatch a subagent," "your instructions file") and adds a per-harness reference that maps each action to the right tool, checked against each runtime. Prose that named "Claude" now says "your agent."
- **A tool reference per harness** at `skills/using-superpowers/references/`, covering Claude Code, Codex, Copilot, Gemini, Pi, and Antigravity.
- **`finishing-a-development-branch` went forge-neutral** — it no longer hardcodes `gh pr create`, so agents push with whatever forge tooling they have. (#1609)
- **One rename:** "Claude Search Optimization" is now "Skill Discovery Optimization," since the technique isn't Claude-specific.
### Writing Skills
Two additions for skill authors.
- **Match the Form to the Failure** — a short table for picking the right kind of guidance. A flat "don't do X" works for discipline slips but backfires when the problem is the *shape* of an output, where a worked example does better. The table, and a tighter scope on the existing rationalization section, steer authors to the form that actually helps.
- **Micro-Test Wording** — a cheap way to check a phrasing before committing to it: sample it a handful of times against a no-guidance control and read every result by hand, treating run-to-run variance as a warning sign.
### Testing
Skill-behavior testing moved out of `tests/` into a new `evals/` submodule built on "drill," which runs real Claude Code, Codex, and Gemini sessions and judges them with an LLM. Several in-tree bash suites retired once a stricter drill scenario covered them; the few with no equivalent stayed. From here on, `tests/` holds plugin-code tests and `evals/` holds skill-behavior tests, and `docs/testing.md` explains the split. New backends reach Antigravity, Pi, and more models, and new shell-lint and pre-commit checks guard the harness. (#1541)
### Bug Fixes
- **systematic-debugging no longer forces every session into extended thinking.** One bullet held the exact keyword Claude Code scans for, quietly tripping the switch on every session that loaded the skill. A hyphen breaks the keyword; the text still reads. (#1283, by @Nick Galatis)
- **The Windows SessionStart hook stopped printing a write error every session** — each `printf` now routes through `cat` to absorb the broken pipe, and the output is otherwise unchanged. (#1612, reported by @silvertakana)
- **Windows foreground mode** tracks the right process and clears its owner PID on MSYS2. (by @nestorluiscamachopaz)
- **The `using-superpowers` bootstrap** no longer lists "debugging" as a skill that doesn't exist. (reported by @mhat)
- **The TDD skill** links the testing anti-patterns reference. (#1532, #1529; link fix #1474 by @Stable Genius)
- **`using-git-worktrees`** fixes its step numbering and drops stale Cursor references. (#1522, and by @fuleinist)
- **The Codex review skill** swaps a private in-joke for plain guidance. (#1531)
### Documentation & Contributor Guidelines
- **A guide to porting Superpowers to a new harness** (`docs/porting-to-a-new-harness.md`) lays out the three pieces every integration needs and the one rule that makes or breaks it: load the bootstrap at session start.
- **Every PR and issue now discloses how it was made** — model, harness, version, and installed plugins, or a note that it was written by hand. We weigh a contribution differently depending on what produced it. PRs also target `dev`, not `main`. The PR template, all three issue templates, and a new platform-support template carry this.
### Contributors
Thanks to @mattvanhorn, @nawfal, @Nick Galatis, @silvertakana, @nestorluiscamachopaz, @qer, @mhat, @Stable Genius, @fuleinist, @dev_Hakaze, @robotsnh, Rahul, and @arittr.
## v5.1.0 (2026-04-30)
### Removals
+88
View File
@@ -0,0 +1,88 @@
# Superpowers for Kimi Code
Complete guide for using Superpowers with [Kimi Code](https://github.com/MoonshotAI/kimi-code).
## Installation
Superpowers is available in Kimi Code's plugin marketplace.
Open the plugin manager:
```text
/plugins
```
Go to `Marketplace` > `Superpowers` and install it.
You can also install from this repository:
```text
/plugins install https://github.com/obra/superpowers
```
For unreleased validation against `dev`, pin the branch explicitly:
```text
/plugins install https://github.com/obra/superpowers/tree/dev
```
Kimi Code applies plugin changes to new sessions. After installing, updating, enabling, disabling, or reloading a plugin, start a fresh session with `/new`.
## How It Works
The Kimi plugin manifest lives at `.kimi-plugin/plugin.json`.
The manifest does three things:
1. Points Kimi Code at the existing `skills/` directory.
2. Loads `using-superpowers` at session start through `sessionStart.skill`.
3. Provides Kimi-specific tool mapping through `skillInstructions`.
Kimi Code reads Superpowers skills from this repository. There are no copied skills, symlinks, hooks, or extra runtime dependencies.
## Tool Mapping
Skills describe actions instead of hard-coding one runtime's tool names. On Kimi Code these resolve to:
- "Ask the user" / "ask clarifying questions" -> `AskUserQuestion`
- "Create a todo" / "mark complete in todo list" -> `TodoList`
- "Dispatch a subagent" -> `Agent`
- "Invoke a skill" -> Kimi Code's native `Skill` tool
- "Read a file" / "write a file" / "edit a file" -> `Read`, `Write`, `Edit`
- "Run a shell command" -> `Bash`
- "Search file contents" -> `Grep`
- "Find files by path or pattern" -> `Glob`
- "Fetch a URL" -> `FetchURL`
- "Search the web" -> `WebSearch`
## Updating
Use Kimi Code's plugin manager:
```text
/plugins
```
Select Superpowers and update it from there. Start a fresh session with `/new` after updating.
## Troubleshooting
### Plugin not loading
1. Run `/plugins info superpowers` and check diagnostics.
2. Make sure the plugin is enabled.
3. Start a fresh session with `/new` after install or update.
### Direct GitHub install used an old release
Kimi Code installs the latest GitHub release for a bare repository URL when one exists. To test unreleased changes before the next Superpowers release, install the branch explicitly:
```text
/plugins install https://github.com/obra/superpowers/tree/dev
```
### Skills not triggering
1. Confirm `/plugins info superpowers` shows the plugin enabled.
2. Start a fresh session with `/new`.
3. Try the acceptance prompt: `Let's make a react todo list`. A working install should load `brainstorming` before writing code.
+14 -8
View File
@@ -50,7 +50,7 @@ use skill tool to list skills
### Loading a Skill
```
use skill tool to load superpowers/brainstorming
use skill tool to load brainstorming
```
### Personal Skills
@@ -99,17 +99,23 @@ To pin a specific version, use a branch or tag:
The plugin does two things:
1. **Injects bootstrap context** via the `experimental.chat.system.transform` hook, adding superpowers awareness to every conversation.
1. **Injects bootstrap context** via the `experimental.chat.messages.transform` hook, adding superpowers awareness to every conversation.
2. **Registers the skills directory** via the `config` hook, so OpenCode discovers all superpowers skills without symlinks or manual config.
### Tool Mapping
Skills written for Claude Code are automatically adapted for OpenCode:
Skills speak in actions rather than naming any one runtime's tools. On OpenCode these resolve to:
- `TodoWrite``todowrite`
- `Task` with subagents → OpenCode's `@mention` system
- `Skill` tool → OpenCode's native `skill` tool
- File operations → Native OpenCode tools
- "Create a todo" / "mark complete in todo list"`todowrite`
- `Subagent (general-purpose):` template → OpenCode's `task` tool with `subagent_type: "general"` (or `"explore"` for codebase exploration)
- "Invoke a skill" → OpenCode's native `skill` tool
- "Read a file" → `read`
- "Create a file" / "edit a file" / "delete a file" → `apply_patch`
- "Run a shell command" → `bash`
- "Search file contents" / "find files by name" → `grep`, `glob`
- "Fetch a URL" → `webfetch`
(Verified against the installed OpenCode CLI's tool inventory.)
## Troubleshooting
@@ -147,7 +153,7 @@ Then use the installed package path in `opencode.json`:
### Bootstrap not appearing
1. Check OpenCode version supports `experimental.chat.system.transform` hook
1. Check OpenCode version supports `experimental.chat.messages.transform` hook
2. Restart OpenCode after config changes
## Getting Help
+826
View File
@@ -0,0 +1,826 @@
# Porting Superpowers to a New Harness
This guide explains how to add support for a new harness — an IDE, CLI, or
agent runner that isn't Claude Code — so that Superpowers skills auto-trigger
there the same way they do natively.
It is written in two layers. **Part 13** explain how the system works and how
to tell whether a harness can be supported at all; read these before you touch
anything. **Part 48** are a prescriptive procedure for an agent (supervised by
a human partner) to execute the port end to end, through distribution. An
appendix indexes the current reference integrations so you can copy the closest
one.
The integration mechanism differs across harnesses, and it will keep changing.
This guide deliberately teaches the **invariants** — the things that must be
true no matter the mechanism — and points you at a live reference implementation
to copy. When this guide and the code disagree, the code wins; fix the guide.
## Before you start
Adding a harness is the highest-stakes contribution type in this repo. Before
writing anything:
- Read `CLAUDE.md` and `.github/PULL_REQUEST_TEMPLATE.md` in full — the
contributor rules and the new-harness PR requirements are not optional.
- Search open **and closed** PRs for a prior attempt at this harness. If one
exists, understand why it stalled before starting your own.
---
## Part 1 — How Superpowers works across harnesses
Superpowers is the same content everywhere. What changes per harness is the thin
layer that delivers that content to the model and translates its instructions
into the harness's native tools. Three components:
1. **Skills (harness-agnostic).** Everything in `skills/` is the source of
truth, shared verbatim by every harness. Skills are written to describe
*actions* — "invoke a skill", "read a file", "dispatch a subagent", "create a
todo" — and never name a specific tool. This is what lets one skill body run
on Claude Code, Codex, Gemini, pi, and the rest without edits.
2. **Tool mapping (per-harness).** Each harness needs the action vocabulary
translated into its real tool names. That translation lives in
`skills/using-superpowers/references/<harness>-tools.md` and/or inline in the
harness's bootstrap injector (see Part 5). It says, e.g., "*dispatch a
subagent* → call `task` with `subagent_type`."
3. **Bootstrap (per-harness).** At the start of every session, the full
`skills/using-superpowers/SKILL.md` is injected into the model's context,
wrapped in `<EXTREMELY_IMPORTANT>` tags, with the tool mapping appended. That
injected skill is what teaches the model that skills exist and that it must
check for a relevant skill before acting. **The bootstrap is the entire
integration.** Without it, the skill files are inert — present on disk, never
invoked.
### Two rules that make this work
**1. Skills name actions, not tools.** Do **not** edit skill bodies to fit your
harness. Porting adds a tool-mapping reference and a bootstrap injector; it
never reaches into `skills/*/SKILL.md` to swap tool names. (The project's
contributor guidelines treat skill content as carefully-tuned behavior-shaping
code; rewording it for "compliance" is rejected on sight.)
**2. Everything ships through the harness's own install mechanism. Never edit the
user's files.** The bootstrap, the skills, and the tool mapping all get delivered
*as part of what the harness installs* — a plugin, an extension, a marketplace
entry, an extension-bundled context file. A port **must not** reach into a user's
global or personal config (`~/.gemini/config/AGENTS.md`, `settings.json`,
`trustedFolders.json`, a hand-edited `~/.bashrc`, etc.) to inject anything. The
harness owns what it loads; your install artifact is the only thing you get to
write. If the install mechanism genuinely can't carry the bootstrap, that is a
limitation to surface (Part 6) — never a license to hand-edit the user's config.
(Shape C is *not* an exception: Gemini's context file is fine because it ships
*inside the installed extension* and is declared by the manifest's
`contextFileName` — the harness loads the extension's own file, not a file you
edited in the user's home.)
---
## Part 2 — Can this harness be supported?
A harness can support Superpowers only if it can do all of the following. Check
these before writing code — if the first one fails, stop.
### Hard requirement: automatic session-start injection
The harness must let you inject text into the model's context **at the start of
every session, with no per-session opt-in by your human partner.** This is the
one non-negotiable capability. It can take any form:
- a **hook/event system** that runs a shell command at session start and reads
its stdout (Claude Code, Codex, Cursor, Copilot CLI), or
- an **in-process plugin/extension** with a session-start or message lifecycle
callback that can mutate the message array (OpenCode, pi), or
- an **instructions-file** convention where the harness loads a context file that
*your installed extension ships and declares* (e.g. Gemini's `contextFileName`
pointing at the extension's own `GEMINI.md`) — not a file you edit in the user's
home.
If the only way to get Superpowers in front of the model is for your human
partner to opt in each session (paste a prompt, run a command, enable a mode),
the harness
**cannot** be properly supported. The acceptance test in Part 3 will fail, and
the PR will be closed. This is the single most common reason a "port" isn't a
real port.
### The rest of the capability checklist
| Capability | Why it's needed | If absent |
|---|---|---|
| **Skill discovery + invocation** | The model must be able to load a skill's full content on demand | If there's no native skill tool, the sanctioned fallback is to `read` the relevant `SKILL.md` directly — see Part 5. A harness with neither a skill tool nor file-read cannot work. |
| **File read / write / edit** | Nearly every skill manipulates files | Essential. No workaround. |
| **Run shell commands** | TDD, verification, git workflows | Essential. |
| **Subagent / task dispatch** | `dispatching-parallel-agents`, `subagent-driven-development` | Degradable: if unavailable, those specific skills tell the model to do the work inline or report the missing capability — *never* to invent a `Task` call. Some harnesses gate this behind a config flag (e.g. Codex needs multi-agent enabled). |
| **Todo / task tracking** | Progress tracking in several skills | Degradable: fall back to a plan file or `TODO.md`. |
| **Web fetch / search** | A few skills | Degradable. |
| **Shell or polyglot script execution (Windows)** | Only for the shell-hook shape, only if you want Windows support | See Part 7. In-process-plugin harnesses sidestep this entirely. |
"Degradable" means: the skill already has fallback wording for the missing
tool. Your job in the tool mapping is to point at the real tool when it exists
and reuse that fallback wording when it doesn't.
### You may not need a new directory at all
Some "new harnesses" are really existing integrations under a different
installer. Factory's Droid, for example, consumes the Claude Code plugin via its
own `plugin install` command and needs no new files here. Before building,
check whether the harness can simply load an existing manifest. A port that adds
nothing to this repo but a paragraph in the README is a perfectly good outcome.
---
## Part 3 — Definition of done
A port is finished when **all** of these are true:
1. The `using-superpowers` bootstrap loads at session start, every session, with
no per-session opt-in.
2. A tool mapping exists for the harness (in
`references/<harness>-tools.md`, inline in the bootstrap, or both — per Part 5).
3. Skills can actually be invoked — natively, or via the documented
read-`SKILL.md` fallback — and the model follows them.
4. **The acceptance test passes.** In a clean session, the user message:
> Let's make a react todo list
auto-triggers the `brainstorming` skill *before any code is written*. Capture
the full transcript — the PR requires it.
5. Tests cover the integration (Part 5) and pass.
6. A real user can install it through the harness's own mechanism (not by
hand-copying files), and the version is tracked in `.version-bump.json` where
applicable (Part 6). Note that some installers rewrite or strip the manifest on
install (one drops it to just `{"name": …}`), so "the *installed* files report
the repo version" is not always achievable — track the version at the source
manifest and don't treat a rewritten installed manifest as a failure.
A quick smoke check before the full acceptance test: start a session and ask the
model to describe its superpowers. If the bootstrap injected, it knows it has
them. (OpenCode's install doc uses `opencode run --print-logs "hello" 2>&1 |
grep -i superpowers` for the same goal via a different mechanism — log-grep
rather than asking the model; the `2>&1` matters because logs go to stderr. Find
your harness's equivalent.)
---
## Part 4 — Choose your integration shape
There are three structural shapes, distinguished by *how you get the bootstrap
in front of the model*. Pick the one that matches what your harness exposes,
then copy that reference implementation. The shape determines almost everything
in Part 5 — the steps below branch on it.
### How to tell which shape you have
Before routing, learn the harness's *actual* mechanism — and don't assume it's
well documented or that it behaves like whatever harness it forked from.
**Find the surface:**
- **Search the web for the harness's docs** (extension / plugin / hook / skill /
MCP / "context file" / "rules file"). Vendor tools change fast; search rather
than trust training knowledge.
- **Find and read an existing third-party extension/plugin for the harness.** A
real working example beats docs — it shows the manifest shape, the install
command, and which components the harness actually loads.
- Check what the harness loads at startup: a settings file? an extensions
directory? a per-project or global instructions file (`AGENTS.md`, `<NAME>.md`)?
**If it's underdocumented, reverse-engineer it empirically** (a real porter has
had to do every one of these):
- `strings` the binary / grep the install tree for hook event names, config
paths, and the instructions file it reads.
- **Ask the running model to enumerate its own tool names** — e.g. "list the
exact machine names of every tool you can call." This is the authoritative way
to get tool names without inventing them (see Step 4).
- Prove every assumption with a **unique-marker test**: inject a nonsense token
through the mechanism you think works, start a fresh session, and confirm the
token actually reached the model.
**A fork does not inherit its parent's behavior.** A harness derived from another
(e.g. a Gemini-derived CLI) may expose the parent's manifest fields and
`@`-include syntax and *still not honor them the same way*. Verify with a marker;
never assume the parent's recipe transfers.
Then route to a shape:
- Shell command at session start whose stdout is read → **Shape A**.
- Plugin/extension module with lifecycle callbacks you run code in → **Shape B**.
- Only ever an always-on instructions file, no hook and no code plugin →
**Shape C**.
**Shapes compose — they are not mutually exclusive.** The *skill-discovery*
mechanism and the *bootstrap* mechanism need not be the same shape — but **both
must still ride the install mechanism** (rule 2). Decide the two questions
separately: *where do skills get discovered?* and *how does the bootstrap reach
the model every session?* A harness might install skills via a plugin yet need
the bootstrap delivered another install-shipped way (an extension-declared
context file, or — see below — by the harness surfacing the installed
`using-superpowers` skill's own description at session start). If more than one
install-mechanism surface injects automatically, prefer the most reliable. What
you may **not** do is bridge a gap by editing the user's global config.
### Shape A — Shell-hook
The harness has a hook system that runs a shell command at session start and
reads JSON from its stdout. The configured command runs `run-hook.cmd`, a
polyglot wrapper that just locates bash and dispatches the named script; the
script (`hooks/session-start`, or a harness-specific variant like
`hooks/session-start-codex`) is what reads `using-superpowers/SKILL.md` and
prints a JSON object whose **field name and nesting differ per harness**.
- Reference: `hooks/session-start` (and `hooks/session-start-codex`),
`hooks/run-hook.cmd`, and the per-harness hook config `hooks/hooks.json`
(Claude Code), `hooks/hooks-codex.json` (Codex), `hooks/hooks-cursor.json`
(Cursor).
- Manifests: `.codex-plugin/plugin.json`, `.cursor-plugin/plugin.json` point the
harness at `./skills/` and the right `hooks-*.json`. (Claude Code's
`.claude-plugin/plugin.json` sets neither field — it auto-discovers `skills/`
and `hooks/hooks.json` by convention.)
> **A hook *system* is not a session-start *event*.** A harness can have a
> `hooks.json` mechanism — and even contain the literal string `SessionStart` in
> its binary — while having no hook event that fires at session start and can
> inject context. (One real harness only exposed pre/post-tool and stop events;
> the `SessionStart` strings were telemetry.) Confirm the *specific event* you
> need exists and can write to the model's context before committing to Shape A.
> If it can't, the bootstrap belongs in an instructions file (Shape C) instead.
### Shape B — In-process plugin / extension
The harness loads a JS/TS module that exposes lifecycle callbacks. You register
the skills directory through the harness's API and inject the bootstrap by
mutating the message array in code.
- Reference: `.opencode/plugins/superpowers.js` (JavaScript) and
`.pi/extensions/superpowers.ts` (TypeScript). pi is the closest reference for
any harness that has **no native skill tool**.
### Shape C — Instructions-file
The harness has neither a shell hook nor a code plugin — its session-start
surface is a context file that *your installed extension ships and the manifest
declares* (e.g. Gemini's `contextFileName` → the extension's own `GEMINI.md`).
You can't run code or mutate messages; the extension's context file points at the
bootstrap. There is no injector to assemble a string or strip frontmatter — the
harness loads the referenced content as-is. **This works only because the file is
part of the installed extension** — never substitute "edit the user's global
`GEMINI.md`/`AGENTS.md`" for shipping your own (rule 2).
- Reference: `gemini-extension.json` (manifest, with `contextFileName`),
`GEMINI.md` (two `@`-includes — the bootstrap skill and the tool-mapping
reference), `skills/using-superpowers/references/gemini-tools.md`.
- Note: `@`-include is a Gemini feature. If your harness loads an instructions
file but has no include syntax, you must inline the bootstrap content into the
file instead.
- **Don't trust that an `@`-include is actually expanded — prove it.** A
Gemini-*derived* harness can accept `@./path` syntax yet treat it as a *hint
the model may choose to read* (it emits a file-read tool call) rather than a
guaranteed inline expansion. That's the difference between the bootstrap being
reliably present every session and the model maybe-reading it. Run a
unique-marker test: if the marker isn't in context *without* a tool call,
**inline the content** rather than `@`-include it.
### Routing table
| If the harness… | Use shape | Copy from |
|---|---|---|
| runs a shell command at session start and reads its stdout | A (shell-hook) | Codex (`hooks/session-start-codex` + `hooks/hooks-codex.json` + `.codex-plugin/`) |
| is a JS/TS plugin host with session/message lifecycle callbacks | B (in-process) | OpenCode (`.opencode/`) — or pi (`.pi/`) if it has no native skill tool |
| ships an extension-declared context file it always loads | C (instructions-file) | Gemini (`gemini-extension.json` + `GEMINI.md` + `references/gemini-tools.md`) |
| has a plugin install command and a manifest `contextFileName` (or equivalent) the installer keeps | C via the plugin installer | Antigravity (`.antigravity-plugin/``agy plugin install` ships a generated context file; verify the installer preserves it — Part 6) |
Most real harnesses fit one row cleanly; the last is the hybrid case (rule 2 still
holds — the bootstrap rides the install mechanism, never a user-config edit).
---
## Part 5 — The porting procedure
### Step 1 — Study the closest reference implementation
Open the files named in Part 4 for your shape and read them end to end. The
patterns below are summaries; the code is the spec.
### Step 2 — Create the manifest / entry point
Create whatever the harness uses to recognize the plugin. Match the existing
ones in spirit:
- **Shape A:** a `*-plugin/plugin.json` (see `.codex-plugin/plugin.json`) with
`name`, `version`, `description`, author/license/keywords, `"skills":
"./skills/"`, and `"hooks": "./hooks/hooks-<harness>.json"`. Plus the
`hooks-<harness>.json` itself, registering a session-start hook whose command
invokes `run-hook.cmd`.
- **Shape B:** the module the harness loads (e.g. `.<harness>/plugins/*.js`) plus
whatever package metadata it needs to be discovered. The committed package
metadata is the **repo-root `package.json`**: `main` points at the OpenCode
plugin, the `pi` field (`pi.extensions`, `pi.skills`) plus the `pi-package`
keyword declare the pi extension. Per-harness local manifests and lockfiles are
kept out of git — `.opencode/.gitignore` excludes `node_modules`,
`package.json`, and lockfiles. Do the same for your harness's *local* install
artifacts so they don't pollute the repo — but never gitignore the repo-root
`package.json`, which is the tracked source of truth.
- **Build/dependency check.** Decide how the harness loads your module:
does it run the source directly (pi's `.ts` is referenced as-is from
`package.json`; OpenCode ships plain `.js`), or does it need a transpile/build
step? Superpowers is zero-runtime-dependency. pi's `import type
{ ExtensionAPI }` works specifically because the harness runs the `.ts`
directly, supplies that type at load, and the repo never type-checks the file
in CI — the import isn't even declared as a dependency. If *your* harness
actually type-checks or bundles the plugin, that breaks: an undeclared type
import fails, and the PR rules only carve out *runtime* deps for new
harnesses, not dev/type packages. If you hit this, confirm the approach with
the maintainer rather than quietly adding a dependency. Keep any build output
out of git and document the command.
- **Shape C (instructions-file):** a small manifest (see `gemini-extension.json`:
`name`, `description`, `version`, `contextFileName`) plus the context file
itself (`GEMINI.md` is just two `@`-includes: the bootstrap skill and the
tool-mapping reference). The Gemini manifest has no `skills` field — Gemini
auto-discovers the `skills/` directory bundled in the installed extension. If
your harness has a native skill tool but no manifest field to register the
directory, you must find its discovery convention (read its extension docs),
then verify empirically: after wiring, ask the model to list its available
skills — if the bundled skills don't appear, discovery isn't working yet.
### Step 3 — Wire the bootstrap injection
This is the heart of the port. The shared goal: at session start, get the
`using-superpowers` skill content (wrapped in `<EXTREMELY_IMPORTANT>` tags) plus
the harness's tool mapping in front of the model, with a note that the skill is
already active so the model doesn't try to load it again. *How* you do that —
and what you assemble vs. what the harness loads raw — depends entirely on your
shape. Do **not** apply one shape's recipe to another.
**Shape A — a script reads `SKILL.md` and prints the harness's JSON.** The
dispatched script (`hooks/session-start`) `cat`s the whole `SKILL.md` (frontmatter
included — that's fine; it's emitted verbatim), wraps it with the "You have
superpowers… for all other skills use the Skill tool" preamble, escapes it, and
prints the harness's JSON shape. The tool mapping for Shape A does **not** go
inline here — it lives in `references/<harness>-tools.md` (Step 4). Get the JSON
output shape exactly right. `hooks/session-start`
detects the harness from environment variables and prints *one of three* shapes:
- Cursor (`CURSOR_PLUGIN_ROOT` set): `{ "additional_context": "…" }`
- Claude Code (`CLAUDE_PLUGIN_ROOT` set, `COPILOT_CLI` unset):
`{ "hookSpecificOutput": { "hookEventName": "SessionStart", "additionalContext": "…" } }`
- Copilot CLI / SDK standard (else): `{ "additionalContext": "…" }`
This is a trap. Emitting the wrong field, or an extra one, means the bootstrap
either never injects or injects twice (Claude Code reads both
`additional_context` and `hookSpecificOutput` without de-duplicating, so emitting
both double-injects). Find the
exact field, nesting, and event-matcher values your harness expects. Then
decide: add a fourth branch to `hooks/session-start`, or — if the harness needs
a different bootstrap message or env contract — add a dedicated
`hooks/session-start-<harness>` script, the way Codex did. If you add a branch
and your harness *also* sets an env var an earlier branch keys on (some harnesses
set `CLAUDE_PLUGIN_ROOT` too), order your branch before the one that would
otherwise shadow it. Match the harness's
own event-matcher strings (Claude Code uses `startup|clear|compact`, Codex
`startup|resume|clear`, Cursor `sessionStart`); wrong matchers mean the hook
silently never fires.
The **hook-config schema itself varies per harness** — don't assume the
Claude/Codex shape is universal. Compare `hooks/hooks.json`,
`hooks/hooks-codex.json`, and `hooks/hooks-cursor.json`: Cursor's uses
`"version": 1`, a lowercase `sessionStart` key, a relative
`./hooks/run-hook.cmd` command, and omits the `matcher`/`type`/`async` fields the
others use. Match your `hooks-<harness>.json` to whichever existing file is
closest, not to a single canonical template.
The hook **command string references a harness-provided plugin-root variable**,
and its name differs per harness: `hooks.json` uses `${CLAUDE_PLUGIN_ROOT}`,
`hooks-codex.json` uses `${PLUGIN_ROOT}`, Cursor uses a relative path. Use
whatever your harness exports. (The `session-start` script re-derives the root
itself via `dirname`, so the script body doesn't depend on this — but the
command in the manifest does.)
**Discovering the harness's contract.** The three facts above — env var, JSON
field/nesting, matcher strings — are the harness's contract, not Superpowers',
so you have to source them. Read the harness's hook docs, or find out
empirically: register a throwaway session-start hook that dumps its environment
and emits a marker, then observe which env var identifies the harness and
whether/how the harness ingests your stdout. Pin these down before writing the
real branch.
**Shape B — assemble the string in code, then inject as a user message.** Here
you build the bootstrap yourself: read `SKILL.md`, strip its YAML frontmatter,
and assemble `<EXTREMELY_IMPORTANT>` + a short preamble that the skill is already
loaded and must not be re-invoked + the stripped body + the inline tool mapping +
`</EXTREMELY_IMPORTANT>`. One subtlety the references disagree on: OpenCode's
preamble says "do NOT use the skill tool…" (assumes a `skill` tool exists), while
pi's just says "do not try to load using-superpowers again." If your harness has
no skill tool, use pi's wording, not OpenCode's.
Inject the result as a **user-role message, not a system message** — system
messages bloat tokens when repeated every turn (#750) and multiple system
messages break some models (#894). Three things you must replicate:
- **Dedup guard.** The lifecycle callback can fire repeatedly (OpenCode's
transform runs on *every* agent step; pi's `context` fires per turn). Before
injecting, check whether a bootstrap marker is already present and skip if so.
(The references pick different markers — pi a custom string, OpenCode the
`EXTREMELY_IMPORTANT` tag; matching the tag is more robust since it needs no
harness-specific constant.) Cache the bootstrap content at module level so
you're not re-reading and re-parsing `SKILL.md` on every call (#1202).
- **Compaction.** If the harness compacts/summarizes history, re-inject
afterward. pi sets an `injectBootstrap` flag on `session_start` and
`session_compact`, clears it on `agent_end`, and inserts the message *after*
any leading compaction-summary messages. OpenCode relies on its per-step
re-injection plus the dedup guard.
- **Message-object shape is per-harness — discover yours, don't copy a literal.**
The two references use *incompatible* shapes: pi builds
`{ role, content: [{ type, text }], timestamp }`; OpenCode manipulates
`message.info.role` and `message.parts[]`. Find your harness's message shape
from its API; copying a reference's object literal verbatim will fail silently.
**Shape C — point your extension's context file at the bootstrap; assemble
nothing.** There is no injector, so you do *not* strip frontmatter or build a
wrapped string. The context file your extension ships (declared by the manifest —
*not* the user's own global file) pulls in two things: the `using-superpowers`
skill and the harness's tool-mapping reference. `GEMINI.md`
does this with two `@`-includes (`@./skills/using-superpowers/SKILL.md` and
`@./skills/using-superpowers/references/<harness>-tools.md`); the harness loads
them raw, frontmatter and all, and `SKILL.md` already carries its own
`<EXTREMELY-IMPORTANT>` block internally. If your harness has no include syntax,
inline the content into the instructions file instead. Gemini ships **no**
"already loaded, don't re-invoke" preamble — for an `@`-include harness the
content is the active instruction set, not a skill the model would re-load. If
you find your harness does try to re-invoke, add that note as a literal line in
the instructions file (you have no code to add it any other way).
### Step 4 — Write the tool mapping
Translate the action vocabulary into the harness's real tools. Cover every one
of these actions (omit only what genuinely doesn't apply):
- read a file
- create / edit / delete a file (one `apply_patch`-style tool, or separate
write/edit?)
- run a shell command
- search file contents / find files by name (grep, glob)
- fetch a URL / web search
- **dispatch a subagent**, including how to pass the agent type — and any config
flag needed to enable it
- **create / update todos** (treat older `TodoWrite` references as this action)
- **invoke a skill** — see Step 5
**Get the real tool names from the harness; never invent them.** If the docs
don't list them, the authoritative source is the harness itself: in a live
session, ask the model to "list the exact machine names of every tool you can
call, one per line" and use what it reports.
**How the harness finds the `skills/` directory is itself per-harness** — confirm
it, don't assume. Possibilities: a manifest `skills` path field (Codex's
`"skills": "./skills/"`); a *co-located* `skills/` the harness auto-scans (where a
path field is **ignored** — one real harness only scanned a `skills/` sitting next
to `plugin.json`); an API/registration call (OpenCode, pi); or you stage an
install dir that pairs the manifest with a **symlink to the repo's `skills/`** and
point the installer at the staging dir (verify the installer *dereferences* the
symlink and copies the real files — confirm with `agy plugin validate`/`install`
or the equivalent before relying on it). A `skills` path field is *not* portable.
Where the mapping lives depends on shape:
- **Shape A:** put it in `skills/using-superpowers/references/<harness>-tools.md`.
The agent reaches it from the bootstrap — `SKILL.md`'s "Platform Adaptation"
section links the per-harness references files. (Shape A harnesses have no
instructions file; the mapping is *not* inlined into the hook output.)
- **Shape B:** the mapping is typically inlined into the bootstrap string you
inject (see the `toolMapping` constant in `superpowers.js`). pi keeps it in
*both* places — `piToolMapping()` inline **and** `references/pi-tools.md`. If
you maintain it in two places, update both, or the port is half-done.
- **Shape C:** put it in `references/<harness>-tools.md` and pull it into the
always-loaded instructions file (e.g. `GEMINI.md` `@`-includes
`gemini-tools.md`).
You may also add a one-line pointer to your harness in `SKILL.md`'s "Platform
Adaptation" section so an agent reading the bootstrap knows where its mapping
lives. This is the one edit to a `SKILL.md` a port may make — and only because
that section is a pointer list, not behavior-shaping content. It does not violate
the "don't edit skill bodies" rule (Part 1); do not touch anything else in any
skill. (The list is a convenience pointer, not an exhaustive registry — not every
harness is listed.)
### Step 5 — Handle a harness with no native skill tool
`using-superpowers/SKILL.md` tells the model to *never read skill files manually
with file tools — always use your platform's skill-loading mechanism.* The point
is "don't bypass the mechanism," not "never use file-read." What counts as "your
platform's mechanism" depends on the harness — and for a harness with no skill
tool, the documented mechanism *is* reading `SKILL.md`. So reading it there
honors the rule rather than breaking it. Distinguish three cases:
1. **Native `Skill`-style tool** (Claude Code, Copilot CLI, Gemini's
`activate_skill`): point the mapping at that tool.
2. **Native skill *discovery* but no `Skill` tool** (pi, Antigravity): the harness
can find and list skills, but the model can't call a tool to load one. Get the
skills installed where the harness scans (pi registers via `resources_discover`
`skillPaths`; OpenCode via its `config` hook; `agy plugin install` copies
them in), and tell the model to load a skill by **reading its `SKILL.md` with
the file-read tool when the skill applies** — the sanctioned mechanism here,
the way `references/pi-tools.md` states it.
**For the bootstrap itself, prefer a declared context file (Part 6).** If the
harness has a `contextFileName`-style manifest field — as Antigravity does —
ship a generated context file through the installer: it's guaranteed-loaded and
carries both the `using-superpowers` content and the tool mapping. That is the
strong, preferred path.
**Fallback — the surfaced skill index.** If there's no context-file field but
the harness surfaces each installed skill's name + description at session start,
you need *neither* a built index nor a runtime-list instruction — the harness
is the index, and `using-superpowers`'s own surfaced description can be what
triggers the model to load it. This is softer than a declared context file;
two things it does **not** give you, versus a context file / hook / in-process
injector — account for both:
- **It bootstraps *triggering*, not the *tool mapping*.** An injector prepends
`<harness>-tools.md` alongside `using-superpowers` every session. Here nothing
injects the mapping — the model only sees skill *descriptions* and must *read*
your `references/<harness>-tools.md` when it needs tool names. It works
because skills name actions (the model reads the mapping when it acts), but
it's softer than injection. Make sure the mapping is reachable from what the
model loads — e.g. linked from `SKILL.md`'s Platform Adaptation section and
installed alongside the skills — not just sitting in the repo.
- **There's no structural guarantee the trigger fires.** No `<EXTREMELY_IMPORTANT>`
wrapper, no dedup, no re-injection after compaction — firing depends on the
model choosing to act on a description it sees in the index. This is exactly
why the acceptance test is mandatory here: it is the *only* guarantee, so run
it on the model(s) your users will actually use, not just the strongest one.
3. **No skill system at all:** there is nothing to register, and the *only*
mechanism is the model reading `SKILL.md` on demand. But the model can't read
what it can't find: `using-superpowers/SKILL.md` does **not** enumerate the
available skills, so on its own the model won't know which skills exist or
their triggers. You must supply a discovery path. Two options, and they differ
in durability: (a) generate a skill index (each `skills/*/SKILL.md`'s `name` +
`description` frontmatter) and place it *inside* the `<EXTREMELY_IMPORTANT>`
wrapper alongside the tool mapping (Shape B recipe above) so it's covered by
the dedup guard — but a build-time index goes stale as skills are added; or
(b) instruct the model to list `skills/*/SKILL.md` at runtime and read their
frontmatter to find a match — slower but never stale. Prefer (b) unless you
have a reason not to. Without either, a no-skill-system port loads the
bootstrap but silently never triggers any other skill.
In cases 2 and 3, say plainly in your tool mapping that reading `SKILL.md` is the
blessed path, so the model doesn't think it's violating the "never read skill
files" rule. Don't go hunting for a `skillPaths`-style registration API in a
harness that has no skill system — case 3 has none.
### Step 6 — Add tests
Match the existing per-harness test style:
- **Shape A:** assert the hook's stdout has the exact JSON shape your harness
consumes, and that it contains the bootstrap. See `tests/hooks/test-session-start.sh`,
which validates each harness's output shape.
- **Shape B:** a unit test that fakes the harness's plugin API and asserts the
lifecycle handlers register, the bootstrap injects once, the dedup guard
works, and (if relevant) compaction re-injection works. See
`tests/pi/test-pi-extension.mjs`. Add an isolated-install integration check in
the style of `tests/opencode/`.
- If the bootstrap is cached, test that the cache behaves when the file is
missing (see the OpenCode caching tests).
These automated tests cover the wiring; the live tmux run in Step 7 is what
proves the integration actually triggers skills.
### Step 7 — Install locally, then drive a live instance to verify
You cannot confirm a port works by reading code. You have to run the harness with
your in-progress port loaded and watch a real session — which is also how you
produce the transcript the PR requires.
**Install locally.** Point a *local* instance of the harness at your working
tree, not a published build:
- **Shape A / C:** install the plugin/extension from this repo's local path (or
symlink its directory into wherever the harness looks). Find the harness's
"install from a local directory / git checkout" path in its docs.
- **Shape B:** register the local module — e.g. an `opencode.json` `plugin`
entry pointing at the local path, or pi resolving the `package.json` fields
from the repo.
Reinstall after each change and restart the harness, since the bootstrap loads at
startup.
**Drive it with tmux.** Most harnesses are interactive REPLs/TUIs that can't be
driven by piping stdin, so run the harness inside a detached tmux session and
control it with `send-keys` / `capture-pane`. A harness may advertise a
non-interactive "run one prompt" mode (e.g. `opencode run "..."`) — try it for the
quick smoke check, but **don't depend on it**: these modes are frequently flaky,
auth-gated, or trust-gated (one real harness's `--print` mode hung and timed out
with no output every time). Be ready to do *everything*, including the smoke
check, through tmux.
**Clear the gates first, or tmux stalls silently.** Many harnesses block on
first-run onboarding, a "do you trust this folder?" prompt, a sandbox mode, or a
permission gate — and a detached tmux session will just sit there with no error
while it waits. Before the run, pre-trust your scratch directory (in the harness's
settings/config) or be prepared to answer those prompts via `send-keys`, and
account for the harness's startup time in your first `sleep`.
```bash
# 1. Launch the harness detached, in a throwaway project dir
mkdir -p /tmp/port-smoke
tmux new-session -d -s port-test -c /tmp/port-smoke '<harness-launch-command>'
# 2. Let it initialize — real TUIs take longer than you think (10s+ with a model
# handshake); tune this. THEN capture and clear any blocking modal before you
# type a prompt: first-run onboarding and "trust this folder?" are modal, so
# keystrokes sent during them select menu items instead of typing your prompt.
sleep 12
tmux capture-pane -t port-test -p # onboarding / trust prompt? answer it via send-keys first
# (e.g. tmux send-keys -t port-test Enter # to accept a trust prompt — inspect before assuming)
# 3. Smoke check: does the model know it has superpowers?
# Send the text and Enter as SEPARATE send-keys with a beat between them —
# sending them together races on some TUIs (Enter arrives before the text lands).
tmux send-keys -t port-test 'What are your superpowers?'; sleep 0.4; tmux send-keys -t port-test Enter
sleep 5
tmux capture-pane -t port-test -p # reply should show it knows its skills
# 4. Acceptance test: exact prompt (note the escaped apostrophe), fresh session
tmux send-keys -t port-test 'Let'\''s make a react todo list'; sleep 0.4; tmux send-keys -t port-test Enter
# poll until the turn finishes — re-capture every few seconds, don't capture once
sleep 8
tmux capture-pane -t port-test -p # PASS = brainstorming triggers BEFORE any code
# 5. Save the transcript for the PR, then clean up
tmux capture-pane -t port-test -p > /tmp/port-smoke/transcript.txt
tmux kill-session -t port-test
```
tmux gotchas that bite here: wait after launch before the first capture; send the
prompt text and `Enter` as *separate* `send-keys` calls with a short `sleep`
between them (sending them together races on some TUIs), and `Enter` is a key name
not `\n`; the agent's turn takes time, so **poll `capture-pane` in a loop** rather
than capturing once; `capture-pane` shows only the visible pane, so for a long
conversation use the harness's own transcript/log file as the record of truth;
always `kill-session` when done.
If the smoke check shows the model *doesn't* know it has superpowers, the
bootstrap isn't loading — fix that before bothering with the acceptance test.
---
## Part 6 — Distribution and release
A working integration in this repo isn't usable until a real user can install
it. Distribution differs per harness ecosystem — find yours:
| Channel | Example | What you do |
|---|---|---|
| Native plugin marketplace | Claude Code | Register in `.claude-plugin/marketplace.json`; users `/plugin install`. The external `superpowers-marketplace` repo is the source of truth users install from — see the release steps in `CLAUDE.md`. |
| External marketplace fork, synced by script | Codex | `scripts/sync-to-codex-plugin.sh` rsyncs the tracked plugin files into a separate fork repo and opens a PR. Read its include/exclude list so you ship the right tree (it deliberately drops repo-internal dirs and other harnesses' dotdirs). |
| Git-URL extension install | Gemini, Kimi Code, OpenCode | Users install from a git URL (`gemini extensions install …`; Kimi Code `/plugins install …`; an `opencode.json` `plugin` array entry). Document the exact command. |
| Package-manifest fields | pi | Declared through fields in the repo-root `package.json`; users install via the harness's package command. |
| Local installer (plugin install) | Antigravity (`agy`) | A small `install.sh` that runs the harness's own `agy plugin install` against a staging dir holding the manifest, the skills, and a generated `contextFileName` context file (the bootstrap). Everything arrives through the install mechanism — *not* by editing the user's config (see below). |
Then:
- **A plugin installer may silently strip *undeclared* files — so make the
bootstrap a file the installer *recognizes*, never a user-config edit.** A
`plugin install` typically copies only the components it knows about
(skills/agents/commands/mcp/hooks/context) and discards anything else, so a
context file the manifest doesn't declare just vanishes from the install. The
fix is **not** to give up and write into the user's config (**rule 2**) — it's
to declare the bootstrap as a recognized component. In escalation order:
- **Ship a context file the manifest declares.** If the harness has a
`contextFileName`-style field (an extension-declared file it loads every
session), that is the strongest clean bootstrap: declare it, and the installer
preserves it *and* the harness loads it. Generate it at install time from the
live `using-superpowers/SKILL.md` + the tool mapping (wrapped in
`<EXTREMELY_IMPORTANT>`) so the installed bootstrap never drifts. This is what
`.antigravity-plugin/install.sh` does — `agy plugin install` reports
`✔ context : ANTIGRAVITY.md`, and a clean session reads `using-superpowers`'s
SKILL.md, loads `brainstorming`, and enters the brainstorming flow before any
code. **Verify with a marker** that the installer keeps the file and the
harness loads it: one porter wrongly concluded it couldn't, because they
shipped the file *without* declaring `contextFileName` and it was stripped as
unrecognized.
- **Otherwise lean on the installed `using-superpowers` skill itself.** If the
harness surfaces each installed skill's name + description at session start,
the `using-superpowers` description ("Use when starting any conversation…")
can prompt the model to load it — installing the skill *is* the bootstrap.
Softer (no guaranteed wrapper; it carries triggering but not the tool mapping
— see Step 5), so prefer the declared context file when available.
- If neither works, the harness cannot be cleanly supported yet — **say so**
and raise it, rather than hand-editing the user's config.
- **Write install docs.** A `docs/README.<harness>.md` and/or a
`.<harness>/INSTALL.md` (see `docs/README.opencode.md` and
`.opencode/INSTALL.md`), plus an install section in the top-level `README.md`.
The only supported install action is **running the harness's own install
command** (`agy plugin install`, `gemini extensions install`, `/plugin
install`, etc.). Hand-copying skill files and editing the user's global/personal
config are *both* off-limits (rule 2 / the PR rules). If the harness has no
install command at all — its only surface is a user-owned config file — then it
fails the "deliver via install mechanism" rule, and you should raise that rather
than ship an installer that edits the user's files.
- **Register the version.** If your harness introduces a *new* versioned
manifest, add its path and version field to `.version-bump.json` so
`scripts/bump-version.sh` keeps it in lockstep (read that file to see what's
currently tracked). A new manifest that isn't registered there will ship a
stale version. If your harness instead rides an already-tracked file — pi
declares itself in the repo-root `package.json`, which is already listed —
there's nothing new to add.
- **If no existing channel fits, you're standing up a new one.** None of the four
rows may match your harness. If it needs a Codex-style external fork sync,
`scripts/sync-to-codex-plugin.sh` is the template to clone (note its anchored
include/exclude list and its PR automation). And whenever you add a new
per-harness directory, add it to the *other* harnesses' sync excludes (e.g. the
EXCLUDES list in `sync-to-codex-plugin.sh`) so your dotdir doesn't leak into
their distributions.
---
## Part 7 — Cross-platform / Windows
Only relevant to the shell-hook shape. `hooks/run-hook.cmd` is a polyglot: a
single file that's valid as both a Windows batch script and a Unix shell script.
On Windows, `cmd.exe` runs the batch portion, which locates `bash` (Git for
Windows, then `bash` on PATH) and runs the named hook script; if no bash is
found it exits cleanly so the harness still works, just without injection. On
Unix, the leading `:` makes the batch block a no-op and the shell runs the
script directly.
Two rules this enforces, which you must respect:
- **Hook scripts are extensionless** (`session-start`, not `session-start.sh`).
Claude Code's Windows handling prepends `bash` to any command containing
`.sh`, which would double-invoke. Name your hook script without an extension.
- Don't write per-OS variants of the hook script. One extensionless bash script
plus the polyglot wrapper covers all three platforms.
`hooks/run-hook.cmd` itself is the authoritative implementation — read it. See
`docs/windows/polyglot-hooks.md` for the background and rationale behind the
dispatcher pattern.
---
## Part 8 — Submitting the PR
- Target the **`dev`** branch. One harness per PR.
- Fill in the PR template's **"New harness support"** section and paste the
complete acceptance-test transcript (the "Let's make a react todo list"
session showing `brainstorming` auto-triggering). A PR without this proof will
be closed.
- Superpowers is a zero-dependency plugin. Don't add a third-party runtime
dependency. Adding a new harness is the one carve-out the contributor rules
allow, and even then keep it to what the integration strictly requires —
type-only imports that compile away are fine; runtime packages are not.
- Don't touch skill bodies (Part 1). If you found yourself editing a `SKILL.md`
to make the port work, the fix belongs in your tool mapping instead.
---
## Appendix A — Reference integrations (current)
Use this as the live index; when in doubt, read the files, not this table.
| Harness | Entry point | Bootstrap mechanism | Tool mapping | Tests | Distribution |
|---|---|---|---|---|---|
| Claude Code | `.claude-plugin/plugin.json` + `hooks/hooks.json` | shell hook → `hooks/session-start` (`hookSpecificOutput.additionalContext`) | native `Skill` tool; `references/claude-code-tools.md` | `tests/hooks/` | marketplace |
| Codex | `.codex-plugin/plugin.json` + `hooks/hooks-codex.json` | shell hook → `hooks/session-start-codex` | `references/codex-tools.md` | `tests/codex-plugin-sync/`, `tests/hooks/` | fork sync (`scripts/sync-to-codex-plugin.sh`) |
| Cursor | `.cursor-plugin/plugin.json` + `hooks/hooks-cursor.json` | shell hook → `hooks/session-start` (`additional_context`) | `references/claude-code-tools.md` | `tests/hooks/` | hand-authored |
| Copilot CLI | (shares Claude Code hook path; `COPILOT_CLI` env) | shell hook → `hooks/session-start` (`additionalContext`) | `references/copilot-tools.md` | `tests/hooks/` | — |
| Gemini CLI | `gemini-extension.json` + `GEMINI.md` | instructions file `@`-includes bootstrap + mapping | `references/gemini-tools.md` | — | `gemini extensions install` |
| Kimi Code | `.kimi-plugin/plugin.json` | manifest `sessionStart.skill` loads `using-superpowers` | inline `skillInstructions` in manifest | `tests/kimi/` | marketplace or `/plugins install` GitHub URL |
| OpenCode | `.opencode/plugins/superpowers.js` (declared via root `package.json` `main`) | in-process: `config` hook registers skills dir; `experimental.chat.messages.transform` injects user message | inline in `superpowers.js` | `tests/opencode/` | `opencode.json` plugin git URL |
| pi | `.pi/extensions/superpowers.ts` | in-process: `resources_discover` registers skills; `context` event injects user message; lifecycle-flag + compaction-aware | `piToolMapping()` inline **and** `references/pi-tools.md` | `tests/pi/` | repo-root `package.json` fields |
## Appendix B — Gotchas that have bitten porters
- **Opt-in isn't a port.** If your human partner has to do anything per session
to get Superpowers, the acceptance test fails. Re-read Part 2.
- **Wrong JSON field → silent failure or double injection.** Shape A only.
Confirm the exact field/nesting; Claude Code reads two fields without dedup.
- **Hook-config schema varies per harness.** Shape A. Cursor's `hooks-cursor.json`
looks nothing like the Claude/Codex one (`version`, lowercase `sessionStart`,
relative command, no `matcher`/`type`/`async`). Match the closest existing file.
- **Plugin-root env var differs per harness.** Shape A. The hook command uses
`${CLAUDE_PLUGIN_ROOT}` (Claude), `${PLUGIN_ROOT}` (Codex), or a relative path
(Cursor). Use what your harness exports; the script re-derives the root itself.
- **System-message injection.** Shape B injects a *user* message on purpose
(#750, #894). Don't "fix" it to a system message.
- **Per-step vs per-turn callbacks.** OpenCode fires every step (per-call dedup
guard); pi fires per turn (lifecycle flag + `agent_end` reset). Copying one
harness's dedup strategy onto the other's callback frequency breaks injection.
- **Message-object shape is per-harness.** Shape B. pi and OpenCode use
incompatible shapes; discover yours, don't copy a reference's object literal.
- **Hunting for a skill-registration API that doesn't exist.** A harness with no
skill system (not just no `Skill` tool) has nothing to register — the model
reads `SKILL.md` on demand. Don't assume a `skillPaths` equivalent exists.
- **Mapping in two places.** For in-process plugins the mapping may live both
inline and in a `references/` file (pi). Update both.
- **The "never read skill files" line.** It means "don't bypass your platform's
skill-loading mechanism," not "never use file-read." On a no-skill-tool harness
that mechanism *is* reading `SKILL.md` — say so explicitly in the mapping
(Part 5).
- **`.sh` on Windows.** Keep hook scripts extensionless (Part 7).
- **Unregistered version.** A new manifest not added to `.version-bump.json`
ships stale (Part 6).
- **Editing skills to fit the harness.** Never. The fix goes in the tool mapping.
@@ -0,0 +1,143 @@
# Pi Extension and Evals Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add first-class Pi package support for Superpowers and add Pi as a Drill eval backend.
**Architecture:** The Pi package is declared in the root `package.json` and loads existing `skills/` plus a small Pi extension. The extension injects the `using-superpowers` bootstrap into provider context as a user-role message on session startup and after compaction, with Pi-specific tool mapping. Drill gains a `pi` backend, Pi session-log normalization, and tests.
**Tech Stack:** Pi TypeScript extension API, Node built-in test runner, Drill Python eval harness, pytest.
---
### Task 1: Pi package manifest and extension tests
**Files:**
- Modify: `package.json`
- Create: `tests/pi/test-pi-extension.mjs`
- [ ] **Step 1: Write failing package/extension tests**
Create `tests/pi/test-pi-extension.mjs` with tests that import `extensions/superpowers.ts`, register fake Pi handlers, and assert:
- root `package.json` has `keywords` containing `pi-package`
- root `package.json` has `pi.skills: ["./skills"]`
- root `package.json` has `pi.extensions: ["./extensions/superpowers.ts"]`
- the extension registers `resources_discover`, `session_start`, `session_compact`, `context`, and `agent_end`
- startup `context` injects exactly one user-role bootstrap message
- `agent_end` clears startup injection
- `session_compact` re-enables injection
- the extension does not register `session_before_compact`
- [ ] **Step 2: Run tests and verify RED**
Run: `node --experimental-strip-types --test tests/pi/test-pi-extension.mjs`
Expected: FAIL because `extensions/superpowers.ts` does not exist and `package.json` lacks the `pi` manifest.
- [ ] **Step 3: Implement manifest fields**
Update `package.json` with `description`, `keywords`, `pi.extensions`, and `pi.skills` while preserving existing `name`, `version`, `type`, and `main`.
- [ ] **Step 4: Implement `extensions/superpowers.ts`**
Create a zero-runtime-dependency extension that:
- locates the package root from `import.meta.url`
- reads `skills/using-superpowers/SKILL.md`
- strips YAML frontmatter
- appends Pi-specific tool mapping
- exposes `resources_discover` with the skills path
- marks bootstrap pending on `session_start` and `session_compact`
- injects a user-role bootstrap message in `context`
- inserts post-compact bootstrap after leading `compactionSummary` messages
- clears pending bootstrap on `agent_end`
- [ ] **Step 5: Run tests and verify GREEN**
Run: `node --experimental-strip-types --test tests/pi/test-pi-extension.mjs`
Expected: PASS.
### Task 2: Pi tool mapping reference
**Files:**
- Create: `skills/using-superpowers/references/pi-tools.md`
- Modify: `tests/pi/test-pi-extension.mjs`
- [ ] **Step 1: Write failing test for Pi reference doc**
Add assertions that `skills/using-superpowers/references/pi-tools.md` exists and documents mappings for `Skill`, `Task`, `TodoWrite`, and built-in tool names.
- [ ] **Step 2: Run tests and verify RED**
Run: `node --experimental-strip-types --test tests/pi/test-pi-extension.mjs`
Expected: FAIL because `pi-tools.md` does not exist.
- [ ] **Step 3: Add Pi reference doc**
Create `skills/using-superpowers/references/pi-tools.md` explaining Pi-native skills, optional `pi-subagents`, no canonical todo/tasklist plugin, and built-in lowercase tools.
- [ ] **Step 4: Run tests and verify GREEN**
Run: `node --experimental-strip-types --test tests/pi/test-pi-extension.mjs`
Expected: PASS.
### Task 3: Drill Pi backend and session log normalization
**Files:**
- Create: `evals/backends/pi.yaml`
- Modify: `evals/drill/backend.py`
- Modify: `evals/drill/engine.py`
- Modify: `evals/drill/normalizer.py`
- Modify: `evals/tests/test_backend.py`
- Modify: `evals/tests/test_normalizer.py`
- [ ] **Step 1: Write failing backend/normalizer tests**
Add pytest coverage for:
- `load_backend("pi")` returns `family == "pi"`
- Pi backend command starts with `pi` and includes `-e ${SUPERPOWERS_ROOT}`
- `_resolve_log_dir()` for Pi points under `~/.pi/agent/sessions`
- `filter_pi_logs_by_cwd()` keeps only session files whose header `cwd` matches the scenario workdir
- `normalize_pi_logs()` extracts `toolCall` blocks from Pi assistant session entries and maps built-in lowercase tools to canonical names
- [ ] **Step 2: Run tests and verify RED**
Run: `uv run pytest evals/tests/test_backend.py evals/tests/test_normalizer.py -q`
Expected: FAIL because the Pi backend and normalizer do not exist.
- [ ] **Step 3: Add `evals/backends/pi.yaml`**
Configure the backend to run `pi -e ${SUPERPOWERS_ROOT}`, use permissive TUI readiness, `/quit` shutdown, and Pi session log location.
- [ ] **Step 4: Implement Pi family support**
Update `Backend.family`, `Engine._resolve_log_dir`, `Engine._collect_tool_calls`, and `normalizer.py` with Pi log filtering and normalizing.
- [ ] **Step 5: Run tests and verify GREEN**
Run: `uv run pytest evals/tests/test_backend.py evals/tests/test_normalizer.py -q`
Expected: PASS.
### Task 4: Documentation and full verification
**Files:**
- Modify: `README.md`
- Modify: `evals/README.md`
- [ ] **Step 1: Document Pi install and eval backend**
Add Pi to README quickstart/install list and add backend entry/usage to `evals/README.md`.
- [ ] **Step 2: Run verification**
Run:
```bash
node --experimental-strip-types --test tests/pi/test-pi-extension.mjs
uv run pytest evals/tests/test_backend.py evals/tests/test_setup.py evals/tests/test_normalizer.py -q
```
Expected: all tests pass.
@@ -0,0 +1,774 @@
# SDD Task-Scoped Review Dispatch Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Scope SDD's per-task reviews to the task (diff-first reading, justified broadening, no redundant test runs) while final branch review stays broad.
**Architecture:** Four prose edits to the subagent-driven-development skill (the per-task quality prompt becomes self-contained instead of delegating to the merge-readiness template; the spec prompt gets a third verdict channel and grounded skepticism; the implementer prompt gains a re-run-after-fix rule; SKILL.md gets controller guidance) plus one new eval scenario in the `evals/` submodule. `skills/requesting-code-review/` is deliberately untouched.
**Tech Stack:** Markdown skill files; Python setup helper + bash checks + story.md for the quorum eval.
**Spec:** `docs/superpowers/specs/2026-06-09-sdd-task-scoped-review-dispatch-design.md` — read it before starting. Decisions already settled there: full re-reviews stay; the two review stages stay separate; coordinator keeps model judgment; `requesting-code-review/` stays broad.
**These are behavior-shaping prose files, not code.** There are no unit tests for them. Each task's verification steps are exact `grep` checks that the edit landed; behavioral verification is Task 6 (static) and Task 7 (live evals, maintainer-gated).
---
### Task 1: Rewrite the per-task quality reviewer prompt as self-contained
The current file delegates to `../requesting-code-review/code-reviewer.md`, which is a merge-readiness review (architecture, security, production readiness, "Ready to merge?"). Replace the entire file with a self-contained, task-scoped template.
**Files:**
- Rewrite: `skills/subagent-driven-development/code-quality-reviewer-prompt.md`
- [ ] **Step 1: Replace the full file contents with:**
````markdown
# Code Quality Reviewer Prompt Template
Use this template when dispatching a code quality reviewer subagent.
**Purpose:** Verify one task's implementation is well-built (clean, tested, maintainable)
**Only dispatch after spec compliance review passes.**
```
Subagent (general-purpose):
description: "Review code quality for Task N"
prompt: |
You are reviewing one task's implementation for code quality. This is a
task-scoped gate, not a merge review — a broad whole-branch review happens
separately after all tasks are complete.
## What Was Implemented
[DESCRIPTION]
## Task Requirements (context only)
[TASK_TEXT]
## Git Range to Review
**Base:** [BASE_SHA]
**Head:** [HEAD_SHA]
```bash
git diff --stat [BASE_SHA]..[HEAD_SHA]
git diff [BASE_SHA]..[HEAD_SHA]
```
## Read-Only Review
Your review is read-only on this checkout. Do not mutate the working tree,
the index, HEAD, or branch state in any way. Use tools like `git show`,
`git diff`, and `git log` to inspect history.
## Scope
Spec compliance was already verified by a separate reviewer. Do not
re-check whether the code matches the requirements or the plan.
Start from the diff. Read the changed files first. Inspect code outside
the diff only to evaluate a concrete risk you can name — and name it in
your report. Cross-cutting changes are legitimate named risks: if the
diff changes lock ordering, a function or API contract, or shared mutable
state, checking the call sites is the right method. Do not crawl the
codebase by default.
## Tests
The implementer already ran the tests and reported results with TDD
evidence for exactly this code. Do not re-run the suite to confirm their
report. Run a test only when reading the code raises a specific doubt
that no existing run answers — and then a focused test, never a
package-wide suite, race detector run, or repeated/high-count loop. If
heavy validation seems warranted, recommend it in your report instead of
running it. If you cannot run commands in this environment, name the
test you would run.
## What to Check
**Code quality:**
- Clean separation of concerns?
- Proper error handling?
- DRY without premature abstraction?
- Edge cases handled?
**Tests:**
- Do the new and changed tests verify real behavior, not mocks?
- Are the task's edge cases covered?
**Structure:**
- Does each file have one clear responsibility with a well-defined interface?
- Are units decomposed so they can be understood and tested independently?
- Is the implementation following the file structure from the plan?
- Did this change create new files that are already large, or
significantly grow existing files? (Don't flag pre-existing file
sizes — focus on what this change contributed.)
## Calibration
Categorize issues by actual severity. Not everything is Critical.
Acknowledge what was done well before listing issues — accurate praise
helps the implementer trust the rest of the feedback.
## Output Format
### Strengths
[What's well done? Be specific.]
### Issues
#### Critical (Must Fix)
[Bugs, data loss risks, broken functionality]
#### Important (Should Fix)
[Poor error handling, test gaps, structural problems]
#### Minor (Nice to Have)
[Code style, optimization opportunities]
For each issue:
- File:line reference
- What's wrong
- Why it matters
- How to fix (if not obvious)
### Assessment
**Task quality:** [Approved | Needs fixes]
**Reasoning:** [1-2 sentence technical assessment]
```
**Placeholders:**
- `[DESCRIPTION]` — task summary, from implementer's report
- `[TASK_TEXT]` — the task's requirements text or plan reference, for context
- `[BASE_SHA]` — commit before this task
- `[HEAD_SHA]` — current commit
**Reviewer returns:** Strengths, Issues (Critical/Important/Minor), Task quality verdict
````
- [ ] **Step 2: Verify the rewrite landed**
Run: `grep -c "requesting-code-review" skills/subagent-driven-development/code-quality-reviewer-prompt.md || echo ABSENT`
Expected: `ABSENT` (no more delegation)
Run: `grep -n "Task quality:" skills/subagent-driven-development/code-quality-reviewer-prompt.md | head -2`
Expected: one match (the Output Format verdict line; the "Reviewer returns" footer says "Task quality verdict" without a colon)
Run: `grep -n "worktree add\|Ready to merge" skills/subagent-driven-development/code-quality-reviewer-prompt.md || echo CLEAN`
Expected: `CLEAN`
- [ ] **Step 3: Commit**
```bash
git add skills/subagent-driven-development/code-quality-reviewer-prompt.md
git commit -m "Make per-task quality reviewer prompt self-contained and task-scoped"
```
---
### Task 2: Spec reviewer prompt cleanups
Four exact edits to `skills/subagent-driven-development/spec-reviewer-prompt.md`. Current line numbers refer to the file as of commit f55642e.
**Files:**
- Modify: `skills/subagent-driven-development/spec-reviewer-prompt.md`
- [ ] **Step 1: Add the judge-from-the-diff clause.** After the line (currently line 31):
```
Only read files in this diff. Do not crawl the broader codebase.
```
insert a blank line and:
```
Spec compliance is judged by reading the diff against the requirements.
The implementer already ran the tests and reported TDD evidence — do not
re-run them. If a requirement cannot be verified from this diff alone
(it lives in unchanged code or spans tasks), report it as a ⚠️ item
instead of broadening your search.
```
- [ ] **Step 2: Trim the read-only section.** Replace (currently line 35):
```
Your review is read-only on this checkout. Do not mutate the working tree, the index, HEAD, or branch state in any way. Use tools like `git show`, `git diff`, and `git log` to inspect history. If you need a working copy of a different revision, check it out into a separate temporary directory (e.g. `git worktree add /tmp/review-[SHA] [SHA]`) — never move HEAD on this checkout.
```
with:
```
Your review is read-only on this checkout. Do not mutate the working tree, the index, HEAD, or branch state in any way. Use tools like `git show`, `git diff`, and `git log` to inspect history.
```
- [ ] **Step 3: Ground the skepticism.** Replace (currently lines 39-40):
```
The implementer finished suspiciously quickly. Their report may be incomplete,
inaccurate, or optimistic. You MUST verify everything independently.
```
with:
```
Treat the implementer's report as unverified claims about the code. It may
be incomplete, inaccurate, or optimistic. Verify the claims against the diff.
```
- [ ] **Step 4: Add the third verdict channel.** Replace (currently lines 74-76):
```
Report:
- ✅ Spec compliant (if everything matches after code inspection)
- ❌ Issues found: [list specifically what's missing or extra, with file:line references]
```
with:
```
Report:
- ✅ Spec compliant (if everything matches after code inspection)
- ❌ Issues found: [list specifically what's missing or extra, with file:line references]
- ⚠️ Cannot verify from diff: [requirements you could not verify from the
diff alone, and what the controller should check — report alongside the
✅/❌ verdict for everything you could verify]
```
- [ ] **Step 5: Verify**
Run: `grep -n "suspiciously\|worktree add" skills/subagent-driven-development/spec-reviewer-prompt.md || echo CLEAN`
Expected: `CLEAN`
Run: `grep -c "⚠️" skills/subagent-driven-development/spec-reviewer-prompt.md`
Expected: `2` (judge-from-diff clause + verdict channel)
- [ ] **Step 6: Commit**
```bash
git add skills/subagent-driven-development/spec-reviewer-prompt.md
git commit -m "Spec reviewer: judge from the diff, grounded skepticism, ⚠️ verdict channel"
```
---
### Task 3: Implementer prompt — re-run tests after fixing review findings
The reviewers' "don't re-run the implementer's tests" rule assumes the implementer re-runs tests after every fix. Make that real.
**Files:**
- Modify: `skills/subagent-driven-development/implementer-prompt.md`
- [ ] **Step 1: Insert a new section.** Immediately before the line (currently line 100):
```
## Report Format
```
insert:
```
## After Review Findings
If a reviewer finds issues and you fix them, re-run the tests that cover
the amended code and include the results in your fix report. Reviewers
will not re-run tests for you — your report is the test evidence.
```
- [ ] **Step 2: Verify**
Run: `grep -n "After Review Findings" skills/subagent-driven-development/implementer-prompt.md`
Expected: one match, on a line before `## Report Format`
- [ ] **Step 3: Commit**
```bash
git add skills/subagent-driven-development/implementer-prompt.md
git commit -m "Implementer prompt: re-run covering tests after fixing review findings"
```
---
### Task 4: SKILL.md controller changes
Six exact edits to `skills/subagent-driven-development/SKILL.md`. Current line numbers refer to commit f55642e.
**Files:**
- Modify: `skills/subagent-driven-development/SKILL.md`
- [ ] **Step 1: Point the final-review flowchart node at the broad template.** The node label `Dispatch final code reviewer subagent for entire implementation` appears 3 times (currently lines 65, 84, 85). In all 3 occurrences, replace the label string with:
```
Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)
```
(Graphviz nodes are matched by label text — all three must be byte-identical or the graph grows a phantom node.)
- [ ] **Step 2: Model selection by judgment.** Replace (currently lines 97-99):
```
**Architecture, design, and review tasks**: use the most capable available model.
**Task complexity signals:**
```
with:
```
**Architecture and design tasks**: use the most capable available model.
**Review tasks**: choose the model with the same judgment, scaled to the
diff's size, complexity, and risk. A small mechanical diff does not need the
most capable model; a subtle concurrency change does.
**Task complexity signals (implementation tasks):**
```
- [ ] **Step 3: Add controller guidance sections.** Immediately before the line (currently line 122):
```
## Prompt Templates
```
insert:
```
## Handling Spec Reviewer ⚠️ Items
The spec reviewer may report "⚠️ Cannot verify from diff" items — requirements
that live in unchanged code or span tasks. These do not block dispatching the
code quality reviewer, but you must resolve each one yourself before marking
the task complete: you hold the plan and cross-task context the reviewer
lacks. If you confirm an item is a real gap, treat it as a failed spec
review — send it back to the implementer and re-review.
## Constructing Reviewer Prompts
Per-task reviews are task-scoped gates. The broad review happens once, at the
final whole-branch review. When you fill a reviewer template:
- Do not add open-ended directives like "check all uses" or "run race tests
if useful" without a concrete, task-specific reason
- Do not ask a reviewer to re-run tests the implementer already ran on the
same code — the implementer's report carries the test evidence
```
- [ ] **Step 4: Prompt Templates list — add the final-review pointer.** Replace (currently line 126):
```
- [code-quality-reviewer-prompt.md](code-quality-reviewer-prompt.md) - Dispatch code quality reviewer subagent
```
with:
```
- [code-quality-reviewer-prompt.md](code-quality-reviewer-prompt.md) - Dispatch code quality reviewer subagent
- Final whole-branch review: use superpowers:requesting-code-review's [code-reviewer.md](../requesting-code-review/code-reviewer.md)
```
- [ ] **Step 5: Example workflow verdict vocabulary.** Two replacements:
Replace (currently line 157):
```
Code reviewer: Strengths: Good test coverage, clean. Issues: None. Approved.
```
with:
```
Code reviewer: Strengths: Good test coverage, clean. Issues: None. Task quality: Approved.
```
Replace (currently line 191):
```
Code reviewer: ✅ Approved
```
with:
```
Code reviewer: ✅ Task quality: Approved
```
(The final reviewer's "ready to merge" line, currently line 199, stays.)
- [ ] **Step 6: Integration section.** Replace (currently line 272):
```
- **superpowers:requesting-code-review** - Code review template for reviewer subagents
```
with:
```
- **superpowers:requesting-code-review** - Code review template for the final whole-branch review
```
- [ ] **Step 7: Verify**
Run: `grep -c "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" skills/subagent-driven-development/SKILL.md`
Expected: `3`
Run: `grep -n "most capable available model" skills/subagent-driven-development/SKILL.md`
Expected: exactly one match (architecture/design bullet)
Run: `grep -n "Handling Spec Reviewer\|Constructing Reviewer Prompts" skills/subagent-driven-development/SKILL.md`
Expected: two section headers, both before `## Prompt Templates`
Run: `grep -c "Task quality: Approved" skills/subagent-driven-development/SKILL.md`
Expected: `2`
- [ ] **Step 8: Commit**
```bash
git add skills/subagent-driven-development/SKILL.md
git commit -m "SDD controller: reviewer prompt budgets, ⚠️ handling, final-review pointer, model judgment"
```
---
### Task 5: New eval scenario — per-task quality reviewer catches a planted defect
Lives in the `evals/` **submodule** (separate repo, `superpowers-evals`). Work on a branch there; the parent submodule-pointer bump happens at finishing time per `evals/CLAUDE.md`.
The fixture plan's Task 2 implementation snippet duplicates Task 1's formatting logic verbatim. The duplication is spec-compliant, so the spec reviewer should pass it — the per-task quality reviewer is the gate under test (DRY violation).
**Files:**
- Create: `evals/setup_helpers/sdd_quality_defect_plan.py`
- Modify: `evals/setup_helpers/__init__.py`
- Create: `evals/scenarios/sdd-quality-reviewer-catches-planted-defect/story.md`
- Create: `evals/scenarios/sdd-quality-reviewer-catches-planted-defect/setup.sh`
- Create: `evals/scenarios/sdd-quality-reviewer-catches-planted-defect/checks.sh`
- [ ] **Step 0: Branch in the submodule**
```bash
cd evals
git checkout -b sdd-quality-defect-scenario
```
- [ ] **Step 1: Create `evals/setup_helpers/sdd_quality_defect_plan.py`:**
````python
"""Setup helper for the sdd-quality-reviewer-catches-planted-defect scenario.
Scaffolds a tiny Node project with a 2-task plan whose Task 2
implementation snippet duplicates Task 1's formatting logic verbatim.
The duplication is spec-compliant — the requirements only describe
behavior — so the spec compliance reviewer should pass it. The test
measures whether the per-task code quality reviewer catches the DRY
violation and forces a refactor in the review-fix loop.
"""
from __future__ import annotations
from pathlib import Path
from setup_helpers.base import _git
PACKAGE_JSON = """\
{
"name": "report-quality",
"version": "1.0.0",
"type": "module",
"scripts": {
"test": "node --test"
}
}
"""
PLAN_BODY = """\
# Report Formatter — Implementation Plan
Two report formatting functions. Implement exactly what each task
specifies.
## Task 1: User Report
**File:** `src/report.js`
**Requirements:**
- Function named `formatUserReport`
- Takes one parameter `user`: an object with `name`, `email`, `visits`
- Returns a multi-line string: a banner of 40 `=` characters, then
`Report for <name> <<email>>`, then the banner again, then
`Visits: <visits>`, then a closing banner
- Export the function
**Implementation:**
```javascript
export function formatUserReport(user) {
const banner = "=".repeat(40);
const lines = [];
lines.push(banner);
lines.push(`Report for ${user.name} <${user.email}>`);
lines.push(banner);
lines.push(`Visits: ${user.visits}`);
lines.push(banner);
return lines.join("\\n");
}
```
**Tests:** Create `test/report.test.js` verifying:
- the result contains `Report for Ada <ada@example.com>` for that user
- the result contains `Visits: 3` when `visits` is `3`
- the result starts and ends with the 40-char banner
**Verification:** `npm test`
## Task 2: Admin Report
**File:** `src/report.js` (add to existing file)
**Requirements:**
- Function named `formatAdminReport`
- Takes one parameter `admin`: an object with `name`, `email`, `lastLogin`
- Same banner layout as the user report; the body line is
`Last login: <lastLogin>` instead of the visits line
- Export the function; keep `formatUserReport` working
**Implementation:**
```javascript
export function formatAdminReport(admin) {
const banner = "=".repeat(40);
const lines = [];
lines.push(banner);
lines.push(`Report for ${admin.name} <${admin.email}>`);
lines.push(banner);
lines.push(`Last login: ${admin.lastLogin}`);
lines.push(banner);
return lines.join("\\n");
}
```
**Tests:** Add to `test/report.test.js`:
- the result contains `Report for Grace <grace@example.com>` for that admin
- the result contains `Last login: 2026-06-01`
- the result starts and ends with the 40-char banner
**Verification:** `npm test`
"""
def scaffold_sdd_quality_defect_plan(workdir: Path) -> None:
workdir = Path(workdir)
workdir.mkdir(parents=True, exist_ok=True)
_git(["git", "init", "-b", "main"], cwd=workdir)
_git(["git", "config", "user.email", "drill@test.local"], cwd=workdir)
_git(["git", "config", "user.name", "Drill Test"], cwd=workdir)
(workdir / "package.json").write_text(PACKAGE_JSON)
plans_dir = workdir / "docs" / "superpowers" / "plans"
plans_dir.mkdir(parents=True, exist_ok=True)
(plans_dir / "report-plan.md").write_text(PLAN_BODY)
_git(["git", "add", "-A"], cwd=workdir)
_git(["git", "commit", "-m", "initial: report formatter plan"], cwd=workdir)
````
(Note the `\\n` in the JS snippets inside PLAN_BODY: the Python source must
produce a literal `\n` in the markdown so the JS reads `lines.join("\n")`.)
- [ ] **Step 2: Register the helper.** In `evals/setup_helpers/__init__.py`:
After the line:
```python
from setup_helpers.sdd_real_projects import scaffold_sdd_go_fractals, scaffold_sdd_svelte_todo
```
add:
```python
from setup_helpers.sdd_quality_defect_plan import scaffold_sdd_quality_defect_plan
```
After the registry entry:
```python
"scaffold_sdd_yagni_plan": scaffold_sdd_yagni_plan,
```
add:
```python
"scaffold_sdd_quality_defect_plan": scaffold_sdd_quality_defect_plan,
```
- [ ] **Step 3: Create `evals/scenarios/sdd-quality-reviewer-catches-planted-defect/story.md`:**
```markdown
---
id: sdd-quality-reviewer-catches-planted-defect
title: SDD's per-task code quality review catches a planted DRY violation
status: ready
tags: subagent-driven-development
quorum_max_time: 90m
---
You have a small plan at docs/superpowers/plans/report-plan.md — two report
formatting functions. The plan's Task 2 implementation snippet duplicates
Task 1's formatting logic verbatim instead of sharing it. The duplication is
spec-compliant (the requirements only describe behavior), so the spec
compliance reviewer should pass it — the per-task code quality reviewer is
the gate under test. You are spec-aware — name the skill.
When the agent is ready for input, tell it to execute the plan with SDD. Use
phrasing like:
"I have a small plan at docs/superpowers/plans/report-plan.md — two report
formatting functions. Use the superpowers:subagent-driven-development skill
to execute it end-to-end — dispatch fresh subagents per task and run the
two-stage review after each."
Let the agent proceed autonomously. If it asks clarifying questions, give
brief answers. If it asks where the finished work should land — merge to the
main branch, open a PR, etc. — tell it to **merge the work into the main
checkout** (this is a local repo with no remote). If a quality reviewer
flags the duplicated formatting logic and an implementer refactors it, let
the review-fix cycle play out — that cycle is exactly the behavior under
test.
The deliverable must end up in the checkout you launched in (the main
working tree). If the agent did its work on a branch or in a worktree, it
is not done until it has merged/finished that work back into the main
checkout. Once the agent reports the plan is complete (both functions
implemented, tests passing) AND the code is present on the main checkout,
you are done.
## Acceptance Criteria
- A `Skill` invocation naming `superpowers:subagent-driven-development`
and at least one `Agent` (subagent dispatch) tool call appear in the
session log.
- The duplicated report-formatting logic did not survive to the end of
the run. Either (a) the implementer never introduced the duplication
(wrote or self-reviewed its way to shared logic), or (b) the per-task
code quality reviewer flagged the duplication as an issue and a
review-fix loop removed it. A fail looks like the duplicated logic
shipping with the per-task quality reviewer approving it, or the
duplication being caught only by the final whole-branch review.
- The per-task quality reviewers stayed task-scoped: no package-wide
test suites, race detector runs, or repeated/high-count test loops
appear in reviewer subagent activity, and reviewers did not re-run
the full test suite merely to confirm the implementer's report.
- `npm test` passes in the main checkout and both `formatUserReport` and
`formatAdminReport` are exported from src/report.js. The deterministic
assertions gate this; the criteria above are about whether the
*per-task quality review* was the mechanism that kept the code clean.
```
- [ ] **Step 4: Create `evals/scenarios/sdd-quality-reviewer-catches-planted-defect/setup.sh`:**
```bash
#!/usr/bin/env bash
set -euo pipefail
uv run setup-helpers run scaffold_sdd_quality_defect_plan
```
Then: `chmod +x evals/scenarios/sdd-quality-reviewer-catches-planted-defect/setup.sh`
- [ ] **Step 5: Create `evals/scenarios/sdd-quality-reviewer-catches-planted-defect/checks.sh`** (no executable bit):
```bash
pre() {
git-repo
git-branch main
requires-tool npm
file-exists 'docs/superpowers/plans/report-plan.md'
file-contains 'docs/superpowers/plans/report-plan.md' 'formatAdminReport'
file-contains 'docs/superpowers/plans/report-plan.md' 'repeat\(40\)'
}
post() {
skill-called superpowers:subagent-driven-development
tool-called Agent
command-succeeds 'npm test'
file-contains 'src/report.js' 'export function formatUserReport'
file-contains 'src/report.js' 'export function formatAdminReport'
command-succeeds 'test "$(grep -c "repeat(40)" src/report.js)" -le 1'
}
```
(The last check is the deterministic DRY gate: the banner construction
`"=".repeat(40)` must appear at most once in the final file — shared, not
duplicated per function.)
- [ ] **Step 6: Validate and test in the evals repo**
```bash
cd evals
uv run quorum check
uv run ruff check
uv run pytest -x -q
```
Expected: all pass; `quorum check` lists the new scenario without errors.
- [ ] **Step 7: Commit (in the submodule)**
```bash
cd evals
git add setup_helpers/sdd_quality_defect_plan.py setup_helpers/__init__.py scenarios/sdd-quality-reviewer-catches-planted-defect/
git commit -m "Add sdd-quality-reviewer-catches-planted-defect scenario"
```
---
### Task 6: Static verification sweep
**Files:** none modified — verification only.
- [ ] **Step 1: No dangling references in the parent repo**
Run: `grep -rn "requesting-code-review" skills/subagent-driven-development/`
Expected: matches only in SKILL.md (final-review flowchart node ×3, Prompt Templates pointer, Integration bullet). None in code-quality-reviewer-prompt.md.
Run: `grep -rn "Ready to merge" skills/subagent-driven-development/ || echo CLEAN`
Expected: `CLEAN`
- [ ] **Step 2: Plugin infrastructure tests**
Run: `bash tests/shell-lint/test-lint-shell.sh`
Expected: all PASS (we added `setup.sh` only inside the evals submodule, which has its own checks).
- [ ] **Step 3: Cross-platform tool tables still coherent**
Run: `grep -n "code-quality-reviewer" skills/using-superpowers/references/antigravity-tools.md skills/using-superpowers/references/gemini-tools.md`
Expected: both tables still list `code-quality-reviewer` as a reviewer template (the new prompt's "If you cannot run commands in this environment, name the test you would run" line keeps the read-only `research` mapping valid — no table edits needed).
---
### Task 7: Live before/after evals (maintainer-gated)
Live quorum runs launch agent CLIs in permissive modes — **trusted-maintainer operation; Jesse launches these**, per `evals/CLAUDE.md`. Requires `ANTHROPIC_API_KEY`.
- [ ] **Step 1: Baseline (skills as released on dev)** — from the main checkout (`/Users/jesse/git/superpowers/superpowers`, on dev), or any checkout without this branch's changes:
```bash
cd evals
export SUPERPOWERS_ROOT=/Users/jesse/git/superpowers/superpowers
uv run quorum run scenarios/sdd-rejects-extra-features --coding-agent claude
uv run quorum run scenarios/sdd-go-fractals --coding-agent claude
uv run quorum run scenarios/sdd-svelte-todo --coding-agent claude
uv run quorum run scenarios/spec-reviewer-catches-planted-flaws --coding-agent claude
```
- [ ] **Step 2: After (this branch's skills)** — point `SUPERPOWERS_ROOT` at this worktree:
```bash
cd evals
export SUPERPOWERS_ROOT=/Users/jesse/git/superpowers/superpowers/.claude/worktrees/sdd-review-dispatch
uv run quorum run scenarios/sdd-rejects-extra-features --coding-agent claude
uv run quorum run scenarios/sdd-go-fractals --coding-agent claude
uv run quorum run scenarios/sdd-svelte-todo --coding-agent claude
uv run quorum run scenarios/spec-reviewer-catches-planted-flaws --coding-agent claude
uv run quorum run scenarios/sdd-quality-reviewer-catches-planted-defect --coding-agent claude
uv run quorum show
```
- [ ] **Step 3: Compare**
Pass bar: all four pre-existing scenarios still pass after the change (no regression in catch rate); the new planted-defect scenario passes. For exploration cost, compare reviewer-subagent tool-call counts between the before/after run transcripts (no automated check exists — the spec calls this out as a known gap).
---
## Finishing
After all tasks pass: the evals submodule commit needs to land in `superpowers-evals` (PR to its `main`), then this branch bumps the `evals` submodule pointer — per `evals/CLAUDE.md`, the parent bump is part of propagation, not optional. Then use superpowers:finishing-a-development-branch. PRs against superpowers target `dev`.
@@ -0,0 +1,352 @@
# Visual Brainstorming Companion — Issue & Change Catalog
**Date:** 2026-06-09
**Status:** Analysis / triage. We are implementing these ourselves; the referenced
community PRs are evidence and reference material, **not** code we intend to merge.
## Purpose
A single place that captures every open issue and PR touching the visual
brainstorming companion (the local server in `skills/brainstorming/scripts/`),
distilled to the underlying problem and the change we'd make. Each item is
grounded against the current code, not the PR author's description.
## Scope decisions (Jesse, 2026-06-09)
- **Not vendoring Alpine.js.** PR #1639 (interactive mockups via a vendored
Alpine build) is **dropped**. See E3.
- **E1 (terminal-vs-HTML hard gate) is a workshop item.** We'll design it
together; it is not specced here.
- **E2 (storage location, #975/#977) is deferred** for now.
- **Remote serving is a first-class scenario.** Superpowers is general-purpose;
users connect from remote (SSH tunnel, Tailscale, `--host 0.0.0.0`). The
security fix MUST protect those users, not just loopback. **Decision: a
per-session secret key**, not a Host allowlist. A Host allowlist only
defends the loopback browser-confused-deputy; a direct remote client just
sends the expected `Host`, so the allowlist is theater for remote exposure. A
secret key is the only thing that authenticates a client uniformly across
loopback, tunnel, and direct-remote, and it also defeats DNS rebinding. See A1.
## Component map
| File | Role |
|------|------|
| `skills/brainstorming/scripts/server.cjs` | Zero-dep HTTP + WebSocket server (RFC 6455 hand-rolled). Serves the newest screen, watches `content/`, records events to `state/events`. |
| `skills/brainstorming/scripts/helper.js` | Injected into every page. WebSocket client, click capture, `window.brainstorm` API. |
| `skills/brainstorming/scripts/frame-template.html` | Frame (header, theme CSS, status dot, indicator bar) wrapped around content fragments. |
| `skills/brainstorming/scripts/start-server.sh` | Launch wrapper. Session dir, host/url-host, owner-PID resolution, platform backgrounding. |
| `skills/brainstorming/scripts/stop-server.sh` | Kills the server by PID file, cleans `/tmp` sessions. |
| `skills/brainstorming/visual-companion.md` | Operator guide the agent reads when it accepts the companion. |
| `skills/brainstorming/SKILL.md` | Where the companion is offered and the per-question decision lives. |
## Disposition summary
| ID | Item | Source | Disposition |
|----|------|--------|-------------|
| A1 | Per-session secret key on `/`, `/files/*`, and WS (supersedes Host allowlist) | issues #1014, PRs #1110/#1553 | **Do** — chosen approach |
| A2 | Host allowlist; browser WS Origin check | PRs #1110/#1553 | Host allowlist dropped; WS Origin check retained after auth for browser confused-deputy defense |
| A3 | Crash on `null` / non-object WS payload | PR #1504 | Do |
| A4 | Frame-length bound in `decodeFrame` | issue #1446 | Already fixed — verify/close |
| B1 | Dotfile screens served as content (`._*.html`) | PR #950 | Do |
| B2 | `stop-server.sh` kills reused/stale PID | PR #1703 | Do |
| B3 | WS client reconnect backoff + status indicator | PR #856 | Do |
| C1 | Idle timeout too short / not configurable; WS not closed on shutdown | issue #1237 (PR #1689) | Do |
| C2 | Server death is invisible to user/agent | issue #1237 (residual) | Do |
| D1 | Permanent opt-out of the companion | issue #892 | Deferred - not in PR #1720 |
| D2 | Free-text feedback from the browser | issue #957 | Deferred - not in PR #1720 |
| D3 | Auto-open the companion URL | PR #759 (#755) | Done in PR #1720 via `--open` |
| D4 | Light/dark contrast helpers in the frame | PR #1683 | Deferred - not in PR #1720 |
| E1 | Hard-gate terminal-vs-HTML per question | PR #1037 | **Workshop** |
| E2 | Move session state out of the working tree | issue #975 (PR #977) | **Deferred** |
| E3 | Vendor Alpine.js for interactive mockups | PR #1639 | **Dropped** |
| E4 | Shell-lint warnings in start/stop scripts | PR #1677 | Opportunistic only |
---
## A. Server security hardening (`server.cjs`)
### A1 — Per-session secret key (chosen approach)
**Threat model.** Two assets: confidentiality of the served screen (`/`) and
files (`/files/*`), and integrity of `state/events` — a WebSocket client with a
truthy `choice` writes there (`server.cjs:243-246`), and the agent reads it next
turn as the user's selection, i.e. **prompt injection into a live session with
full tool access**. Reachers: with the default `127.0.0.1` bind, a malicious
page in the user's browser (a confused deputy — runs attacker JS *and* can reach
loopback); with a remote bind (`--host 0.0.0.0`, tailnet/LAN), any host that can
route to the port, directly, with no same-origin policy in the way. Today
`handleUpgrade` (`server.cjs:176`) checks only `Sec-WebSocket-Key`, and
`handleRequest` (`server.cjs:138`) checks nothing — both are wide open.
**Why a key, not a Host allowlist.** A Host allowlist only defends the
loopback browser-deputy. A direct remote client just sends the expected `Host`
and forges/omits `Origin`, so the allowlist is theater for exactly the remote
case we must protect. A per-session secret authenticates the client uniformly
across loopback, SSH tunnel, and direct-remote, and it also kills DNS rebinding
(the rebound page neither knows the key nor receives the host-scoped cookie).
So the key **supersedes** A1/A2's Host allowlist entirely — no `BRAINSTORM_ALLOWED_HOSTS`.
**Design.** Random token (`crypto.randomBytes(32)` hex), generated in
`server.cjs` at startup (overridable via `BRAINSTORM_TOKEN` for deterministic
tests):
1. **URL carries it** as `?key=<token>`. The server already builds `url` in its
`server-started` JSON (`server.cjs:351`) and writes it to `state/server-info`
— appending `?key=` there means `start-server.sh` (greps and prints that
JSON) and the skill (hands the user that URL) need **no change**.
2. **Cookie bootstrap.** A valid `?key` on `/` sets
`brainstorm-key-<port>=<token>; HttpOnly; SameSite=Strict; Path=/`. The
browser then auto-attaches it to same-origin subresources (`/files/*`) and
the WebSocket handshake, so the agent can write any URL style and it works,
and `helper.js` needs no change. Cookie name is **per-port** to avoid the
Jupyter multi-server collision (cookies aren't port-scoped).
`SameSite=Strict` is safe for CDN/Unsplash content — that cookie is host-
scoped, so outbound CDN requests never carry it; SameSite only governs
requests back to our origin, which are all same-site.
3. **Auth gate** = valid `?key` **OR** valid cookie (compared with
`crypto.timingSafeEqual`) on `/`, `/files/*`, and the WS upgrade. Missing/bad
key → friendly **403 HTML page** ("this page needs the full URL your coding
agent gave you, including `?key=…`" — generic "coding agent", not "Claude",
since this ships on Codex/Gemini/Copilot too). WS upgrade → destroy socket.
The query token is the source of truth; the cookie is a convenience that never
bears initial-auth load.
**Blast radius.** `server.cjs` (all logic). `helper.js` optional one-liner
(append `?key=` from `location.search` to the WS URL as a cookie-blocked
fallback). `start-server.sh` none. `visual-companion.md` doc note (URL now has
`?key=`; don't strip it). Tests updated to pass the token.
### A2 — Host allowlist dropped; browser WS Origin retained
Subsumed by A1. The secret key closes the WS-injection vector (#1014), the
HTTP/WS DNS-rebinding read vector (PR #1553), and the cross-origin WS vector
(PR #1110) in one mechanism, and unlike an allowlist it actually protects the
remote-bind case. No `BRAINSTORM_ALLOWED_HOSTS` and no Host allowlist. The final
implementation still checks browser WebSocket `Origin` after session auth so a
cross-origin localhost tab cannot ride the companion cookie.
### A3 — Server crashes on `null` / primitive WS payload
**Problem.** `handleMessage` (`server.cjs:233`) does `JSON.parse(text)` then
`if (event.choice)` at `server.cjs:243`. A client that sends the 4-byte text
frame `null` yields `event === null`, and `null.choice` throws. The throw is
**not** caught — `handleMessage` is called from the `socket.on('data')` handler
(`server.cjs:207`) outside the `try/catch`, which only wraps `decodeFrame`. The
result is an uncaught exception and process exit. Any local client can kill the
server.
**Change.** Guard the access: `if (event && event.choice)`. Minimal and exact —
`JSON.parse` can't produce `undefined`, and primitives return `undefined` for
`.choice` without throwing, so only `null` is the live hazard. (Avoid the
broader fixes — a top-level `try/catch` or `process.on('uncaughtException')`
would mask other bugs.)
### A4 — Frame-length bound in `decodeFrame` (adjacent)
Referenced by PR #1504 as #1446. The current code **already** bounds extended
frame lengths: `MAX_FRAME_PAYLOAD_BYTES = 10MB` (`server.cjs:10`) is enforced at
`server.cjs:58-67` before any `Buffer.alloc`. Action: verify #1446 against
current `dev` and close if already resolved, rather than re-implementing.
---
## B. Server robustness / correctness
### B1 — macOS resource-fork dotfiles served as screen content
**Problem.** The newest-screen selector filters on `f.endsWith('.html')` only
(`server.cjs:127-128`). On macOS/ExFAT, `._screen.html` resource-fork files pass
that filter and, being written alongside the real file, can sort newest — so the
browser gets binary metadata instead of the mockup. Four read sites share the
weak filter: `getNewestScreen` (`server.cjs:127`), `knownFiles` init
(`server.cjs:279`), the `fs.watch` handler (`server.cjs:286`), and the `/files/`
endpoint (`server.cjs:154-156`).
**Change.** Reject dotfiles (`!f.startsWith('.')`) at all four sites. Covers
`._*`, `.DS_Store`, etc.
### B2 — `stop-server.sh` can kill a reused PID
**Problem.** `stop-server.sh` reads the PID from `state/server.pid`
(`stop-server.sh:20`) and `kill`s it (`:23`, escalating to `-9` at `:35`)
without confirming the PID still belongs to our server. After a reboot or PID
wraparound the file can point at an unrelated process, which we'd then SIGKILL.
**Change.** Before signalling, verify ownership — the PID's command is `node`
running our `server.cjs`, ideally matching this session. If ownership can't be
proven, fail closed (report `stale_pid`, don't kill). Keep the existing
`stopped` / `not_running` outputs for the real cases.
### B3 — WebSocket client: silent reconnect, stale "Connected"
**Problem.** `helper.js` reconnects on a fixed 1s timer (`helper.js:21-23`),
has no `onerror` handler, never nulls `ws` on close, and never clears a pending
reconnect timer. The frame's status element is hardcoded to "Connected" with the
dot pinned to `var(--success)` (`frame-template.html:77,200`). When the laptop
sleeps or the server restarts, the page shows "Connected" over a dead socket and
queues events with no feedback.
**Change.**
- `helper.js`: exponential backoff (500ms → ×2 → cap 30s, reset on open);
`onerror` delegating to `onclose`; `ws = null` on close; `clearTimeout` before
reconnecting.
- `frame-template.html`: drive the status dot from a `--status-color` custom
property so JS can switch Connected (green) / Reconnecting (yellow) /
Disconnected (red).
---
## C. Lifecycle / timeout (issue #1237)
### C1 — Idle timeout too short, not configurable, WS keeps process alive
**Problem.** `IDLE_TIMEOUT_MS` is hardcoded to 30 minutes (`server.cjs:258`),
enforced by the 60s lifecycle check (`server.cjs:329-332`). A single brainstorm
question can sit longer than 30 min while the user thinks or steps away, so the
server dies mid-session. Separately, `shutdown()` (`server.cjs:310-321`) calls
`server.close()` but never closes the upgraded sockets in `clients`
(`server.cjs:174`), so an open browser connection can keep the Node process
alive past shutdown.
**Change.**
- Raise the default to 4 hours and make it configurable:
`--idle-timeout-minutes` in `start-server.sh` → an env var → `IDLE_TIMEOUT_MS`,
with validation against Node timer overflow.
- Expose the effective timeout in the startup JSON / `state/server-info`.
- In `shutdown()`, close every socket in `clients` so the process actually
exits.
### C2 — Server death is invisible
**Problem.** When the server exits it writes `state/server-stopped` and removes
`state/server-info` (`server.cjs:312-317`), and the skill is *told* to check
those files (`visual-companion.md:108`) — but it's soft guidance the model skips,
and the browser just shows a generic "can't be reached." The user diagnoses it
manually; the agent keeps referring to a dead URL.
**Change (two parts, independent of C1):**
- **Browser-facing tombstone.** Leave something at the last-served URL that says
"this companion expired — ask Claude to restart it" instead of a connection
error. Options to weigh: `helper.js` rendering a banner when the socket stays
down past backoff (works only while the page is loaded), vs. a more involved
approach that keeps a minimal responder alive to serve a tombstone page.
- **Harder skill check.** Tighten `visual-companion.md` / `SKILL.md` so
"check `server-info`/`server-stopped` before referring to the URL or pushing a
screen" is a required step, not a note. Keep it lightweight — possibly a
one-line helper the agent always runs.
---
## D. Features
### D1 — Permanent opt-out of the visual companion (issue #892)
**Problem.** The companion is offered as its own message every session
(`SKILL.md:25,151-152`). A user who never wants it pays that round-trip — and
HTML generation — every time. There's no way to say "never offer this."
**Change.** Before the offer step, the skill checks a user-level setting and
skips the offer entirely when opt-out is set.
**Design choice open.** Mechanism isn't settled:
- Env var (e.g. `SUPERPOWERS_VISUAL_COMPANION=off`) the skill is told to read —
simplest, matches what the issue asks for, lives in `.zshrc`.
- A plugin-settings file (`.claude/superpowers.local.md` frontmatter) — more
structured, per-project capable, but heavier and project-scoped.
- Reliability caveat from the issue: a separate "no-companion" skill competes on
trigger words and isn't reliable — rejected.
Pick the mechanism, then it's a small `SKILL.md` change plus a documented knob.
### D2 — Free-text feedback from the browser (issue #957)
**Problem.** The client only captures clicks on `[data-choice]`
(`helper.js:36-62`). A user who wants to annotate a mockup ("wrong shade of
blue") has to switch to the terminal, breaking the visual flow.
**Change.** Add a feedback `<textarea>` whose submit emits
`{"type":"feedback","text":...,"timestamp":...}` via the existing
`window.brainstorm.send` path (`helper.js:82-85`).
**Cross-cutting — server change required.** `handleMessage` only persists events
when `event.choice` is truthy (`server.cjs:243`). A `feedback` event has no
`choice`, so today it would be logged but **never written to `state/events`**,
and the agent wouldn't see it. The persistence condition must also accept
`feedback` events. Document the new event shape in `visual-companion.md`
(Browser Events Format, `:247-259`). Decide the submit trigger (button vs blur
vs both) and where the textarea renders (frame-level vs opt-in per screen).
### D3 — Auto-open the companion URL (PR #759, issue #755)
**Problem.** `start-server.sh` only prints the URL; the user opens it manually.
In WSL2 especially, people expect the browser to open.
**Change.** Best-effort opener after the `server-started` JSON is parsed:
Windows/WSL → `rundll32.exe url.dll,FileProtocolHandler <url>`, macOS → `open`,
Linux → `xdg-open` only when `DISPLAY`/`WAYLAND_DISPLAY` is set. Swallow
failures, never block startup, keep echoing the URL. Document in
`visual-companion.md`. (Consider an opt-out for headless/remote runs where
popping a browser is wrong — ties into D1's config mechanism.)
### D4 — Light/dark contrast helpers (PR #1683)
**Problem.** Content fragments are wrapped in the OS-aware frame
(`frame-template.html`). In dark mode, quick mockups often use white inline
backgrounds while inheriting low-contrast frame text, making cards/panels hard
to read.
**Change.** Add `.light-surface` / `.dark-surface` helper classes plus a
conservative fallback for common inline light backgrounds, and document them in
`visual-companion.md`'s CSS reference. Pure CSS in `frame-template.html`.
---
## E. Workshop / deferred / dropped
### E1 — Hard-gate terminal-vs-HTML per question (PR #1037) — WORKSHOP
The soft guidance already exists: "decide per-question," with browser-vs-terminal
tests in `SKILL.md:156-161` and `visual-companion.md:5-25`. The complaint is that
the model renders HTML for purely textual content (A/B lists, clarifying
questions), wasting tokens and a turn. PR #1037 wraps the decision in a
`<HARD-GATE>`. **Per Jesse, we'll workshop the wording/mechanism together**
this is behavior-shaping skill content and not specced here.
### E2 — Move session state out of the working tree (issue #975 / PR #977) — DEFERRED
Today `--project-dir` writes session state to `<project>/.superpowers/brainstorm/`
(`start-server.sh:80-84`) and the skill tells the user to gitignore it
(`visual-companion.md:58`). The ask is a `--state-dir` / `SUPERPOWERS_STATE_DIR`
default outside the repo (XDG), keeping `--project-dir` as an alias.
**Deferred by Jesse for now.** Captured so it isn't lost.
### E3 — Vendor Alpine.js for interactive mockups (PR #1639) — DROPPED
Adds a vendored Alpine build so mockups can be interactive (tabs, accordions,
forms) without hand-rolled JS. **Dropped per Jesse** — we are not taking on a
vendored third-party dependency in the companion runtime. The underlying need
(interactive mockups) is not being pursued via this route.
### E4 — Shell-lint warnings (PR #1677) — OPPORTUNISTIC
SC2034 (and friends) in `start-server.sh` / `stop-server.sh`. Trivial; fold into
B2/C1/D3 when we're already editing those scripts rather than as its own change.
---
## Suggested grouping for implementation
These cluster into a few coherent passes (each independently testable against
`tests/brainstorm-server/`):
1. **Security pass** (IN PROGRESS, branch `brainstorm-companion-session-key`) —
A1 per-session key (supersedes A2) + A3 null-crash guard. Verify/close A4.
*Highest priority.*
2. **Lifecycle pass** — C1 + C2 together (both touch `shutdown()` and the
server-death story).
3. **Robustness pass** — B1, B2, B3 (independent, small).
4. **Deferred feature pass** - D1, D2, D4 are not part of PR #1720. D3 is
shipped through the `--open` flow.
E1 is a separate workshop session. E2/E3 are out of scope for this round.
@@ -0,0 +1,785 @@
# Visual Companion Auth Hardening Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Harden the brainstorming visual companion auth and reconnect flow while preserving trusted same-origin screen JavaScript and future vendored UI libraries.
**Architecture:** Keyed root loads become a bootstrap step that sets the cookie, stores the key in tab-scoped `sessionStorage`, and navigates to a bare `/` screen URL. WebSockets require valid auth plus browser same-origin `Origin`, while `/files/*` uses realpath containment to prevent content-directory escapes.
**Tech Stack:** Node.js built-ins (`http`, `fs`, `path`, `crypto`), zero runtime dependencies, existing `ws` test dependency, Bash start/stop scripts, repo shell lint script.
**Important:** Do not commit during execution unless Drew explicitly asks. This repository's instructions override the generic plan template's commit cadence.
---
## File Map
- Modify: `skills/brainstorming/scripts/server.cjs`
- Add bootstrap response.
- Add shared security headers.
- Add WebSocket Origin validation.
- Add `/files/*` realpath containment.
- Modify: `skills/brainstorming/scripts/helper.js`
- Read the stored session key and append it to the WebSocket URL.
- Modify: `tests/brainstorm-server/auth.test.js`
- Add bootstrap, header, same-origin WS, cross-origin WS, and cookie/file auth regressions.
- Modify: `tests/brainstorm-server/helper.test.js`
- Add mocked-browser coverage for sessionStorage-backed WS URLs.
- Modify: `tests/brainstorm-server/server.test.js`
- Add symlink containment regression for `/files/*`.
- Modify: `tests/brainstorm-server/lifecycle.test.js`
- Make the start-server timeout flag test force background mode.
- Add restart reconnect credential coverage if it fits the existing lifecycle helper.
- Modify: `skills/brainstorming/scripts/start-server.sh`
- Fix shell lint.
- Modify: `skills/brainstorming/scripts/stop-server.sh`
- Fix shell lint.
- Modify: `.gitignore`
- Add `.superpowers/`.
- Optional docs update: `skills/brainstorming/visual-companion.md`
- Mention bootstrap URL stripping and trusted same-origin screen JS if the code behavior changes need operator-facing explanation.
## Task 1: Bootstrap Keyed Root Loads
**Files:**
- Modify: `tests/brainstorm-server/auth.test.js`
- Modify: `skills/brainstorming/scripts/server.cjs`
- [ ] **Step 1: Add RED tests for bootstrap behavior**
In `tests/brainstorm-server/auth.test.js`, add tests after the existing valid-key root test:
```js
await test('GET / with valid query returns bootstrap instead of screen content', async () => {
const res = await get('/', { key: TOKEN });
assert.strictEqual(res.status, 200);
assert(res.body.includes('sessionStorage'), 'bootstrap should store the session key in tab storage');
assert(res.body.includes('location.replace'), 'bootstrap should navigate to the bare root URL');
assert(!res.body.includes('Secret screen'), 'bootstrap must not serve screen HTML at the keyed URL');
});
await test('GET / with valid cookie serves the screen after bootstrap', async () => {
const res = await get('/', { cookie: `${COOKIE_NAME}=${TOKEN}` });
assert.strictEqual(res.status, 200);
assert(res.body.includes('Secret screen'), 'cookie-authenticated bare root should serve the screen');
assert(!res.body.includes('sessionStorage'), 'bare screen response should not be the bootstrap page');
});
```
Keep the existing cookie test if present; merge assertions rather than duplicating the same test name.
- [ ] **Step 2: Verify RED**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node auth.test.js
```
Expected: the new bootstrap test fails because current `GET /?key=...` serves `Secret screen` directly and does not include the bootstrap `sessionStorage`/`location.replace` code.
- [ ] **Step 3: Implement minimal bootstrap response**
In `skills/brainstorming/scripts/server.cjs`, add a helper near the page constants:
```js
function bootstrapPage(key) {
const jsonKey = JSON.stringify(String(key));
return `<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Opening Brainstorm Companion</title></head>
<body>
<script>
sessionStorage.setItem('brainstorm-session-key', ${jsonKey});
location.replace('/');
</script>
</body>
</html>`;
}
```
Then in `handleRequest`, after authorization and cookie setting but before serving screen HTML, detect a valid query key on root:
```js
function queryKey(url) {
const q = url.indexOf('?');
if (q < 0) return null;
return new URLSearchParams(url.slice(q + 1)).get('key');
}
```
Use it in `handleRequest`:
```js
const pathname = pathnameOf(req.url);
const keyFromQuery = queryKey(req.url);
if (req.method === 'GET' && pathname === '/' && keyFromQuery && timingSafeEqualStr(keyFromQuery, TOKEN)) {
res.writeHead(200, securityHeaders({ 'Content-Type': 'text/html; charset=utf-8' }));
res.end(bootstrapPage(keyFromQuery));
return;
}
```
This assumes Task 4 will introduce `securityHeaders`. If implementing Task 1 first, temporarily use:
```js
res.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' });
```
and replace it in Task 4.
- [ ] **Step 4: Verify GREEN**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node auth.test.js
```
Expected: all auth tests pass, including the new bootstrap tests.
## Task 2: WebSocket Origin Enforcement
**Files:**
- Modify: `tests/brainstorm-server/auth.test.js`
- Modify: `skills/brainstorming/scripts/server.cjs`
- [ ] **Step 1: Add RED tests for same-origin and cross-origin WS**
In `tests/brainstorm-server/auth.test.js`, extend `wsConnect` to accept an `origin` option:
```js
function wsConnect({ key, cookie, origin } = {}) {
const url = `ws://localhost:${TEST_PORT}/` + (key !== undefined ? `?key=${key}` : '');
const headers = {};
if (cookie) headers['Cookie'] = cookie;
if (origin) headers['Origin'] = origin;
const ws = new WebSocket(url, Object.keys(headers).length ? { headers } : {});
return new Promise((resolve) => {
let settled = false;
const done = (outcome) => { if (!settled) { settled = true; resolve({ outcome, ws }); } };
ws.on('open', () => done('opened'));
ws.on('error', () => done('rejected'));
ws.on('close', () => done('rejected'));
setTimeout(() => done('rejected'), 1500);
});
}
```
Then add:
```js
await test('WS upgrade with valid cookie and same-origin Origin opens', async () => {
const { outcome, ws } = await wsConnect({
cookie: `${COOKIE_NAME}=${TOKEN}`,
origin: `http://localhost:${TEST_PORT}`
});
ws.close();
assert.strictEqual(outcome, 'opened');
});
await test('WS upgrade with valid cookie but cross-origin Origin is rejected', async () => {
const eventsFile = path.join(TEST_DIR, 'state', 'events');
if (fs.existsSync(eventsFile)) fs.unlinkSync(eventsFile);
const { outcome, ws } = await wsConnect({
cookie: `${COOKIE_NAME}=${TOKEN}`,
origin: 'http://localhost:9999'
});
if (outcome === 'opened') {
ws.send(JSON.stringify({ type: 'choice', choice: 'attacker-injected', text: 'local attacker probe' }));
await sleep(300);
}
ws.close();
assert.strictEqual(outcome, 'rejected', 'cross-origin browser WS must not open even with cookie');
assert(!fs.existsSync(eventsFile), 'cross-origin WS must not write state/events');
});
```
- [ ] **Step 2: Verify RED**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node auth.test.js
```
Expected: cross-origin cookie WS test fails because current server accepts any cookie-authenticated WS regardless of Origin.
- [ ] **Step 3: Implement Origin check**
In `skills/brainstorming/scripts/server.cjs`, add:
```js
function isAllowedWebSocketOrigin(req) {
const origin = req.headers.origin;
if (!origin) return true; // non-browser clients still need the session key
const host = req.headers.host;
if (!host) return false;
return origin === 'http://' + host;
}
```
Then update `handleUpgrade`:
```js
function handleUpgrade(req, socket) {
if (!isAuthorized(req) || !isAllowedWebSocketOrigin(req)) { socket.destroy(); return; }
```
- [ ] **Step 4: Verify GREEN**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node auth.test.js
```
Expected: auth tests pass; cross-origin WS is rejected; same-origin and direct key WS still open.
## Task 3: Helper Uses Stored Key For Reconnect
**Files:**
- Modify: `tests/brainstorm-server/helper.test.js`
- Modify: `skills/brainstorming/scripts/helper.js`
- [ ] **Step 1: Add RED test for WebSocket URL key**
In `tests/brainstorm-server/helper.test.js`, add a mocked-browser test near the reconnect state-machine tests:
```js
test('uses sessionStorage key in the WebSocket URL when present', () => {
const e = makeEnv();
e.state.sessionKey = 'stored-key-abc';
e.boot();
assert.strictEqual(e.sockets[0].url, 'ws://localhost:7777/?key=stored-key-abc');
});
```
Update `makeEnv()` so the returned object exposes `sockets`, and the mock window includes sessionStorage:
```js
window: {
location: { host: 'localhost:7777', reload() { state.reloads++; } },
sessionStorage: { getItem: (key) => key === 'brainstorm-session-key' ? state.sessionKey : null }
},
```
Also add a fallback test:
```js
test('uses cookie-only WebSocket URL when no sessionStorage key is present', () => {
const e = makeEnv();
e.state.sessionKey = null;
e.boot();
assert.strictEqual(e.sockets[0].url, 'ws://localhost:7777');
});
```
- [ ] **Step 2: Verify RED**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node helper.test.js
```
Expected: stored-key test fails because current helper uses `ws://localhost:7777`.
- [ ] **Step 3: Implement stored-key WS URL**
In `skills/brainstorming/scripts/helper.js`, replace:
```js
const WS_URL = 'ws://' + window.location.host;
```
with:
```js
function websocketUrl() {
let key = null;
try { key = window.sessionStorage && window.sessionStorage.getItem('brainstorm-session-key'); } catch (e) {}
return 'ws://' + window.location.host + (key ? '/?key=' + encodeURIComponent(key) : '');
}
```
Then replace:
```js
ws = new WebSocket(WS_URL);
```
with:
```js
ws = new WebSocket(websocketUrl());
```
- [ ] **Step 4: Verify GREEN**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node helper.test.js
```
Expected: helper tests pass.
## Task 4: Security Headers
**Files:**
- Modify: `tests/brainstorm-server/auth.test.js`
- Modify: `skills/brainstorming/scripts/server.cjs`
- [ ] **Step 1: Add RED header tests**
In `tests/brainstorm-server/auth.test.js`, add:
```js
await test('HTML responses include leak-reduction and anti-framing headers', async () => {
const res = await get('/', { key: TOKEN });
assert.strictEqual(res.headers['referrer-policy'], 'no-referrer');
assert.strictEqual(res.headers['cache-control'], 'no-store');
assert.strictEqual(res.headers['x-frame-options'], 'DENY');
assert.strictEqual(res.headers['content-security-policy'], "frame-ancestors 'none'");
assert.strictEqual(res.headers['cross-origin-resource-policy'], 'same-origin');
});
await test('403 responses include leak-reduction and anti-framing headers', async () => {
const res = await get('/');
assert.strictEqual(res.status, 403);
assert.strictEqual(res.headers['referrer-policy'], 'no-referrer');
assert.strictEqual(res.headers['cache-control'], 'no-store');
assert.strictEqual(res.headers['x-frame-options'], 'DENY');
assert.strictEqual(res.headers['content-security-policy'], "frame-ancestors 'none'");
assert.strictEqual(res.headers['cross-origin-resource-policy'], 'same-origin');
});
```
- [ ] **Step 2: Verify RED**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node auth.test.js
```
Expected: header tests fail because current responses do not include these headers.
- [ ] **Step 3: Implement shared header helper**
In `skills/brainstorming/scripts/server.cjs`, add:
```js
function securityHeaders(headers = {}) {
return {
'Referrer-Policy': 'no-referrer',
'Cache-Control': 'no-store',
'X-Frame-Options': 'DENY',
'Content-Security-Policy': "frame-ancestors 'none'",
'Cross-Origin-Resource-Policy': 'same-origin',
...headers
};
}
```
Update response writes in `handleRequest`:
```js
res.writeHead(403, securityHeaders({ 'Content-Type': 'text/html; charset=utf-8' }));
```
```js
res.writeHead(200, securityHeaders({ 'Content-Type': 'text/html; charset=utf-8' }));
```
```js
res.writeHead(200, securityHeaders({ 'Content-Type': contentType }));
```
For 404s:
```js
res.writeHead(404, securityHeaders());
```
- [ ] **Step 4: Verify GREEN**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node auth.test.js
```
Expected: auth tests pass and header assertions are green.
## Task 5: `/files/*` Realpath Containment
**Files:**
- Modify: `tests/brainstorm-server/server.test.js`
- Modify: `skills/brainstorming/scripts/server.cjs`
- [ ] **Step 1: Add RED symlink escape test**
In `tests/brainstorm-server/server.test.js`, after the `/files/` empty-name test, add:
```js
await test('does not serve symlinks that escape content dir via /files/', async () => {
const target = path.join(STATE_DIR, 'server-info');
const link = path.join(CONTENT_DIR, 'linked-server-info.txt');
try { fs.unlinkSync(link); } catch (e) {}
fs.symlinkSync(target, link);
const res = await fetch(`http://localhost:${TEST_PORT}/files/linked-server-info.txt`);
assert.strictEqual(res.status, 404, 'symlink to state/server-info must not be served');
assert(!res.body.includes('server-started'), 'response must not include server-info body');
});
```
- [ ] **Step 2: Verify RED**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node server.test.js
```
Expected: symlink test fails because current `/files/*` follows symlinks and serves `server-info`.
- [ ] **Step 3: Implement containment helper**
In `skills/brainstorming/scripts/server.cjs`, add:
```js
function isRegularFileInsideContentDir(filePath) {
let stat, realContentDir, realFilePath;
try {
stat = fs.lstatSync(filePath);
if (stat.isSymbolicLink()) return false;
if (!stat.isFile()) return false;
realContentDir = fs.realpathSync(CONTENT_DIR);
realFilePath = fs.realpathSync(filePath);
} catch (e) {
return false;
}
return realFilePath.startsWith(realContentDir + path.sep);
}
```
Replace the `/files/*` guard with:
```js
if (!fileName || fileName.startsWith('.') || !isRegularFileInsideContentDir(filePath)) {
res.writeHead(404, securityHeaders());
res.end('Not found');
return;
}
```
- [ ] **Step 4: Verify GREEN**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node server.test.js
```
Expected: server tests pass, including symlink rejection.
## Task 6: Restart Reconnect Regression
**Files:**
- Modify: `tests/brainstorm-server/lifecycle.test.js`
- Modify: `skills/brainstorming/scripts/server.cjs`
- Modify: `skills/brainstorming/scripts/helper.js`
- [ ] **Step 1: Add RED integration test for same key over WS after restart**
In `tests/brainstorm-server/lifecycle.test.js`, add a test after the port/token persistence test:
```js
await test('stored key can authenticate WebSocket after same-port restart', async () => {
const dir = fs.mkdtempSync('/tmp/bs-reconnect-');
const portFile = path.join(dir, '.last-port');
const tokenFile = path.join(dir, '.last-token');
const env = { ...process.env, BRAINSTORM_PORT_FILE: portFile, BRAINSTORM_TOKEN_FILE: tokenFile, BRAINSTORM_LIFECYCLE_CHECK_MS: 100000 };
const a = spawn('node', [SERVER], { env: { ...env, BRAINSTORM_DIR: path.join(dir, 's1') } });
let outA = ''; a.stdout.on('data', d => outA += d.toString());
for (let i = 0; i < 60 && !outA.includes('server-started'); i++) await sleep(50);
const infoA = firstServerStarted(outA);
const keyA = new URL(infoA.url).searchParams.get('key');
a.kill(); await sleep(400);
const b = spawn('node', [SERVER], { env: { ...env, BRAINSTORM_DIR: path.join(dir, 's2') } });
let outB = ''; b.stdout.on('data', d => outB += d.toString());
for (let i = 0; i < 60 && !outB.includes('server-started'); i++) await sleep(50);
const infoB = firstServerStarted(outB);
const ws = new WebSocket(`ws://localhost:${infoB.port}/?key=${keyA}`, {
headers: { Origin: `http://localhost:${infoB.port}` }
});
const opened = await new Promise(resolve => {
ws.on('open', () => resolve(true));
ws.on('error', () => resolve(false));
setTimeout(() => resolve(false), 1500);
});
try {
assert.strictEqual(infoB.port, infoA.port, 'restart should reuse same port');
assert(opened, 'stored key should authenticate WS after restart');
} finally {
try { ws.close(); } catch (e) {}
b.kill(); await sleep(100);
fs.rmSync(dir, { recursive: true, force: true });
}
});
```
This test may already pass once Tasks 2 and 3 are implemented. If it passes before code changes, keep it as coverage but do not call it RED. The real browser reconnect behavior is primarily covered by Task 3 plus final manual/headless browser verification.
- [ ] **Step 2: Verify behavior**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node lifecycle.test.js
```
Expected after Tasks 2 and 3: lifecycle tests pass. If this fails, fix the auth/restart path before continuing.
## Task 7: Lifecycle Hang And Shell Lint
**Files:**
- Modify: `tests/brainstorm-server/lifecycle.test.js`
- Modify: `skills/brainstorming/scripts/start-server.sh`
- Modify: `skills/brainstorming/scripts/stop-server.sh`
- [ ] **Step 1: Reproduce shell lint failure**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers
scripts/lint-shell.sh skills/brainstorming/scripts/start-server.sh skills/brainstorming/scripts/stop-server.sh tests/brainstorm-server/stop-server.test.sh
```
Expected current failure:
```text
SC2164: skills/brainstorming/scripts/start-server.sh line 128: cd "$SCRIPT_DIR"
SC2034: skills/brainstorming/scripts/start-server.sh line 166: for i in {1..50}
SC2034: skills/brainstorming/scripts/stop-server.sh line 57: for i in {1..20}
```
- [ ] **Step 2: Fix shell lint minimally**
In `skills/brainstorming/scripts/start-server.sh`, change:
```bash
cd "$SCRIPT_DIR"
```
to:
```bash
cd "$SCRIPT_DIR" || exit 1
```
Change unused loop variables from `i` to `_` where they are not read:
```bash
for _ in {1..50}; do
```
In `skills/brainstorming/scripts/stop-server.sh`, change:
```bash
for i in {1..20}; do
```
to:
```bash
for _ in {1..20}; do
```
- [ ] **Step 3: Fix lifecycle start-server hang**
In `tests/brainstorm-server/lifecycle.test.js`, update the `start-server.sh --idle-timeout-minutes sets the timeout` test command:
```js
const out = execFileSync('bash', [START, '--project-dir', dir, '--idle-timeout-minutes', '5', '--background'], { encoding: 'utf8' });
```
This keeps the test from hanging when `CODEX_CI` triggers start-server foreground mode.
- [ ] **Step 4: Verify lint and lifecycle**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers
scripts/lint-shell.sh skills/brainstorming/scripts/start-server.sh skills/brainstorming/scripts/stop-server.sh tests/brainstorm-server/stop-server.test.sh
cd tests/brainstorm-server
node lifecycle.test.js
```
Expected: shell lint exits 0; lifecycle tests exit 0 without hanging.
## Task 8: Gitignore Durable Companion State
**Files:**
- Modify: `.gitignore`
- [ ] **Step 1: Verify current ignore gap**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers
git check-ignore .superpowers/brainstorm/.last-token || true
```
Expected current output: no matching ignore rule.
- [ ] **Step 2: Add ignore rule**
Add this line to `.gitignore`:
```gitignore
.superpowers/
```
- [ ] **Step 3: Verify GREEN**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers
git check-ignore .superpowers/brainstorm/.last-token
```
Expected output:
```text
.superpowers/brainstorm/.last-token
```
## Task 9: Full Automated Verification
**Files:**
- No code changes in this task.
- [ ] **Step 1: Run focused suites**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
node auth.test.js
node helper.test.js
node server.test.js
node lifecycle.test.js
```
Expected: all four commands exit 0.
- [ ] **Step 2: Run full brainstorm-server suite**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
npm test
```
Expected: all tests pass, including ws-protocol, helper, auth, server, lifecycle, and stop-server.
- [ ] **Step 3: Repeat suite for lifecycle/watch flake**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers/tests/brainstorm-server
for i in 1 2 3; do npm test || exit 1; done
```
Expected: all three repeats pass without hanging.
- [ ] **Step 4: Run shell lint**
Run:
```bash
cd /Users/drewritter/prime-rad/superpowers
scripts/lint-shell.sh skills/brainstorming/scripts/start-server.sh skills/brainstorming/scripts/stop-server.sh tests/brainstorm-server/stop-server.test.sh
```
Expected: exits 0.
## Task 10: Re-run Security Probes
**Files:**
- No code changes in this task.
- [ ] **Step 1: Recreate the cross-origin attacker probe**
Use the previous scratch probe if available:
```bash
node /tmp/superpowers-pr1720-security-drewritter/probe-pr1720.cjs
```
If the scratch probe is unavailable, recreate a minimal probe under `/tmp` that:
- starts the companion with a fixed token
- loads the keyed URL in headless Chrome
- starts an attacker page on a different localhost port
- attempts `new WebSocket('ws://localhost:<companion-port>/')`
- sends `{"type":"choice","choice":"attacker-injected"}`
- checks `state/events`
Expected after fixes:
- keyless and wrong-key HTTP still return 403
- same-origin helper reaches Connected
- cross-origin WebSocket does not open
- `state/events` does not contain `attacker-injected`
- symlink-to-`server-info` returns 404
- keyed browser load ends on bare `/`
- [ ] **Step 2: Re-run manual/browser flow only after automated probes pass**
Manual flow:
1. start the companion with `--project-dir --open`
2. push a screen
3. confirm URL strips to `/`
4. confirm status reaches Connected
5. click a choice and verify `state/events`
6. stop and restart same project
7. verify the open tab reconnects automatically
Expected: all steps pass without manual URL reload.
## Self-Review Checklist
- Spec coverage: every design requirement maps to at least one task.
- Placeholder scan: this plan contains no unresolved placeholder markers or unspecified edge-case steps.
- TDD order: every production change task starts with a focused failing test or a command that demonstrates the current failure.
- Trust model: the plan preserves trusted same-origin screen JavaScript and future same-origin vendored libraries.
- No-commit rule: execution does not commit unless Drew explicitly asks.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,77 @@
# Platform-neutral config-file references — Phase B design
## Background
Phase A (see `2026-05-05-platform-neutral-prose-design.md`) replaced generic third-person "Claude" prose with agent-neutral forms. This phase tackles the next category: references to the per-platform instruction file (CLAUDE.md, AGENTS.md, GEMINI.md) inside skills.
The plugin runs on multiple harnesses, and each one reads its own instruction file. Where a skill names CLAUDE.md as if it were the only file, that's a Claude-Code-centric assumption that doesn't hold on Codex / Gemini CLI / OpenCode.
## In scope
Two specific lines in active skills:
1. **`skills/writing-skills/SKILL.md:58`** — `Project-specific conventions (put in CLAUDE.md)`
2. **`skills/receiving-code-review/SKILL.md:30`** — `"You're absolutely right!" (explicit CLAUDE.md violation)`
## Out of scope
- **`skills/using-superpowers/SKILL.md:22, 26`** — instruction-priority list. The list already names all three (CLAUDE.md, GEMINI.md, AGENTS.md) inclusively, which is correct: the section is making a real claim about *what counts as user instruction* on a multi-platform plugin. No change needed.
- **Historical / example artifacts**:
- `skills/systematic-debugging/CREATION-LOG.md` — attribution path (`~/.claude/CLAUDE.md`) is a historical fact.
- `skills/writing-skills/examples/CLAUDE_MD_TESTING.md` — the entire file is a worked example testing CLAUDE.md content variants. The filename, body, and the reference from `testing-skills-with-subagents.md` all stay; normalizing them defeats the example.
- **Platform-tooling references** — Phase D candidates:
- `skills/using-superpowers/SKILL.md:40` (Gemini CLI tool mapping note about GEMINI.md)
- `skills/using-superpowers/references/gemini-tools.md` (`save_memory` persists to GEMINI.md)
## Substitution rules
Two distinct calls, one per in-scope line.
### Rule 1: "where to put project-specific conventions"
`writing-skills/SKILL.md:58`:
- **Before:** `Project-specific conventions (put in CLAUDE.md)`
- **After:** `Project-specific conventions (put in your instructions file)`
Use a generic phrase rather than picking one filename. Different harnesses read different files (CLAUDE.md, AGENTS.md, GEMINI.md, etc.) and the skill should not assume one. The platform-tools reference docs (`references/{codex,copilot,gemini}-tools.md`) are the right place to name each platform's preferred file.
### Rule 2: the "(explicit CLAUDE.md violation)" parenthetical
`receiving-code-review/SKILL.md:30`:
- **Before:** `"You're absolutely right!" (explicit CLAUDE.md violation)`
- **After:** `"You're absolutely right!" (explicit instruction-file violation)`
The parenthetical is doing real work — it signals this phrase isn't just stylistically bad, it actively violates rules many users put in their instruction files. "Instruction file" is the natural cross-platform term covering AGENTS.md / CLAUDE.md / GEMINI.md collectively, and keeps the original signal without picking one filename or softening to "common".
## Commit plan
Atomic commits, in order:
1. **`writing-skills/SKILL.md`** — CLAUDE.md → "your instructions file" in the "where to put project conventions" line
2. **`receiving-code-review/SKILL.md`** — CLAUDE.md → instruction-file in the violation parenthetical
3. **Platform-tools reference docs** — add the preferred per-platform instructions filename (CLAUDE.md, AGENTS.md, GEMINI.md, etc.) to each `references/{codex,copilot,gemini}-tools.md` so readers can resolve "your instructions file" to a real filename.
Each commit message names "Phase B" and the slice.
## Verification
After each commit:
- Read the surrounding paragraph to confirm grammar and meaning still parse.
- `grep -n "CLAUDE\.md" <touched-file>` — no remaining hits in active prose (carve-outs already documented).
After both commits:
- `grep -rn "CLAUDE\.md" skills/` should return only the documented carve-outs (CREATION-LOG, CLAUDE_MD_TESTING and its inbound reference, the priority list in using-superpowers).
## Non-goals
- Do not touch the priority list ordering in `using-superpowers/SKILL.md`. Reordering CLAUDE.md / GEMINI.md / AGENTS.md is an aesthetic change, not a substitution, and out of scope here.
- Do not rename `examples/CLAUDE_MD_TESTING.md` or change its content.
- Do not modify Gemini-CLI-specific tooling references (Phase D candidates).
## Implementation note
Phase B as written here covered three commits and the three non-Claude-Code platform-tools refs. Implementation went one step further: a fourth ref, `references/claude-code-tools.md`, was added in commit `8505703` for symmetry, so Claude Code's instructions-file conventions and tool-name list live alongside the others rather than implicitly in the surrounding skill prose. That addition wasn't anticipated in this spec but is consistent with its intent.
@@ -0,0 +1,94 @@
# Platform-neutral prose — Phase A design
## Background
Superpowers ships to multiple agent runtimes (Claude Code, Codex, Cursor, OpenCode, Copilot CLI, Gemini CLI). Skill content and supporting docs were written first for Claude Code and use "Claude" in places where any runtime's agent applies. OpenAI's vendored fork (openai/plugins#217) attempted a wholesale rewrite that was actively wrong in places — rewriting historical attribution paths, model names, and platform-specific install instructions — and we want to avoid that mistake while still removing platform-centric prose where it is genuinely incidental.
The full effort is broken into phases by reference category. **This spec covers Phase A only:** generic third-person prose mentioning "Claude" in non-platform-specific contexts. Later phases (config-file references, marketing copy, tool-name references) are out of scope here and will get their own specs.
## In scope
Generic prose mentions of "Claude" in:
- `skills/*/SKILL.md` and supporting `.md` files in active skill directories
- `skills/writing-skills/anthropic-best-practices.md`
- `README.md` (only where the mention is generic prose, not platform marketing)
Plus one coined-term rename: **Claude Search Optimization (CSO) → Skill Discovery Optimization (SDO)** in `skills/writing-skills/SKILL.md`.
## Out of scope
- **Platform/runtime statements** — "In Claude Code:", install instructions, tool-mapping references. (Phase D candidate.)
- **Config-file references** — CLAUDE.md, AGENTS.md, GEMINI.md priority lists and "where to put project conventions" callouts. (Phase B.)
- **Tool-name references** — `Skill`, `Bash`, `Read`, `Task`, `TodoWrite`. Skills are written in Claude Code's tool vocabulary; the existing `references/{codex,copilot,gemini}-tools.md` files map them. (At the time this spec was written, the plan was to defer or skip these. Phase E ended up doing them — replacing tool names with action language across active skills and unifying the platform-tools refs around the same vocabulary.)
- **Marketing copy** in README — "Superpowers for Claude Code", platform-named install sections. (Phase C.)
- **Historical artifacts** — `docs/plans/*.md`, `docs/superpowers/specs/*.md`, `CREATION-LOG.md`. These are dated, point-in-time documents; rewriting them rewrites history.
- **Model identifiers** — Claude Haiku / Sonnet / Opus. These are real product names.
- **Filename / URL references** — `CLAUDE.md`, `claude.com`, `claude-plugin/`, paths under `~/.claude/`.
- **`anthropic-best-practices.md` filename** — the file remains named after its source even though we rewrite the prose inside it.
## Replacement style
Use a mix that reads naturally in English:
- **Second person — "your agent"** when addressing the skill author about *their* runtime
- "your agent reads the description"
- **Third person — "the agent" / "agents" / "an agent"** when describing system behavior generically
- "Future agents find your skills"
- "Use words an agent would search for"
- "Agents read SKILL.md only when the skill becomes relevant"
Pick whichever fits the surrounding sentence; do not force consistency at the cost of awkward phrasing. Pluralize when natural ("future agents", "agents read") rather than always saying "the agent".
### Carve-outs that stay as "Claude"
- Model names: Claude Haiku, Claude Sonnet, Claude Opus
- Filenames and URLs: `CLAUDE.md`, `claude.com`, `~/.claude/`
- Branded platform name "Claude Code" wherever it refers to the runtime as such (handled in later phases)
### Coined-term rename
- **Claude Search Optimization (CSO) → Skill Discovery Optimization (SDO)**
- Appears in `skills/writing-skills/SKILL.md` as a section heading and in nearby prose. Rename the heading, the acronym, and any in-file cross-references.
## Files affected
Approximate counts based on a `grep` filtered to exclude carve-outs:
| File | Generic-prose mentions |
|------|------------------------|
| `skills/writing-skills/SKILL.md` | ~12 (includes CSO heading + body) |
| `skills/writing-skills/anthropic-best-practices.md` | ~30 |
| `skills/writing-skills/examples/CLAUDE_MD_TESTING.md` | ~1 — filename stays (it's a CLAUDE.md test artifact); the "Variant C: Claude.AI Emphatic Style" heading also stays (it's a label naming a specific style) |
| `README.md` | ~1 |
Final list confirmed during implementation by re-running the filtered grep.
## Commit plan
Four atomic commits, in order:
1. **Rename CSO → SDO** in `skills/writing-skills/SKILL.md`. Mechanical, isolated, easy to revert if we change our minds about the term.
2. **Active skills prose** — generic "Claude" → "agent" forms across `skills/*/SKILL.md` and supporting `.md`, excluding `anthropic-best-practices.md`.
3. **`anthropic-best-practices.md` prose** — same substitution rules. Separate commit because this file is a vendored adaptation of an external doc; isolating the change makes future reconciliation with upstream easier to read.
4. **README.md prose** *(only if any generic-prose mentions remain after filtering)*. Skipped if empty.
Each commit message names the phase ("Phase A") and the slice ("rename CSO to SDO", "agent prose in active skills", etc.) so the series is self-documenting.
## Verification
After each commit:
- `grep -rn "Claude" <touched-paths>` — every remaining hit must fall into a documented carve-out (model name, filename, URL, "Claude Code" platform name, historical artifact).
- Read the touched file end-to-end — substitutions should not have broken sentence flow, pronoun agreement, or list parallelism.
- No tests to run; this is prose-only.
After the final commit:
- Skim each modified skill in a live session to confirm nothing reads awkwardly.
## Non-goals
- Do not change behavior, structure, headings (other than CSO→SDO), examples, code blocks, or YAML frontmatter.
- Do not introduce new sections, callouts, or compatibility notes.
- Do not "improve" prose beyond the substitution while editing.
@@ -0,0 +1,47 @@
# Platform-neutral README ordering — Phase C design
## Background
Phases A and B (see `2026-05-05-platform-neutral-prose-design.md` and `2026-05-05-platform-neutral-config-refs-design.md`) already neutralized generic Claude prose and config-file references in the README. The remaining platform-leaning signal is layout: the README's two platform listings put Claude Code first and aren't strictly alphabetical elsewhere.
This phase fixes the ordering. No prose changes.
## In scope
1. **Quickstart platform list** (`README.md:7`) — the inline link list of supported harnesses
2. **Installation section ordering** (`README.md:35152`) — the per-harness install sub-sections
## Out of scope
- Prose, marketplace names, plugin IDs, URLs — all factually correct as-is.
- Visual weight of the Claude Code section (which has two sub-sections — official Anthropic marketplace and Superpowers marketplace). Both are real install paths; collapsing them would hide accurate info.
- Section headings and content within each install block — only the ordering of the blocks changes.
## Substitution
Both listings reorder to strict alphabetical:
| Old order | New order |
|-----------|-----------|
| Claude Code | Claude Code |
| Codex CLI | Codex App |
| Codex App | Codex CLI |
| Factory Droid | Cursor |
| Gemini CLI | Factory Droid |
| OpenCode | Gemini CLI |
| Cursor | GitHub Copilot CLI |
| GitHub Copilot CLI | OpenCode |
Three moves: Codex App swaps with Codex CLI; Cursor moves up two slots; GitHub Copilot CLI moves up one.
Claude Code remains first by alphabetical chance (`Cl…` precedes `Co…`).
## Commit plan
One atomic commit covering both listings, since changing one without the other would create inconsistency between the quickstart and the installation section.
## Verification
- Quickstart anchors (`#claude-code`, `#codex-app`, etc.) still resolve to existing `### …` headings — no headings renamed.
- Each install sub-section's body is byte-identical pre/post; only positions changed.
- `git diff README.md` shows section moves only, no content edits.
@@ -0,0 +1,160 @@
# SDD Task-Scoped Review Dispatch
Make subagent-driven-development's per-task reviews cheaper and faster without weakening them, by scoping per-task review prompts to the task and stopping redundant work — while final branch review stays broad.
## Problem
Per-task code quality reviewers in SDD routinely do branch-review-scale work on single-task diffs. Evidence from two real local SDD sessions: `a1a6719a-6109-453a-9933-34ae396f5bae` (sen-core-v2) and `0cc1a12d-9984-4c35-8615-9d42dadb2c47` (serf), both under `~/.claude/projects/`:
- In the sen-core-v2 session, 7/8 quality reviewers ran repo-wide greps; the most expensive ran 50+ Bash commands over ~200 seconds. Across both sessions, quality reviewers cost 4-8× what spec reviewers cost on the same tasks.
- Spec reviewers, whose prompt contains "Only read files in this diff. Do not crawl the broader codebase," stayed tight: 6-16 tool calls, 14-65 seconds.
- No reviewer ran heavy tests autonomously. Every package-wide or repeated test run observed was explicitly requested by a controller-written prompt ("check all uses," "run tests if useful, especially race-focused ones," "does anything else read `Meta()`?").
Root causes, in order of impact:
1. **The per-task quality prompt inherits a merge-readiness review.** `code-quality-reviewer-prompt.md` delegates to `requesting-code-review/code-reviewer.md`, which asks about architecture, scalability, security, production readiness, and ends with "Ready to merge?" That frame licenses branch-level breadth on a one-task diff. The spec prompt's diff-scope guard was never carried over.
2. **The controller gets no guidance on writing reviewer prompts**, so it invents open-ended directives ("check all uses") that reviewers interpret literally.
3. **Duplicated work across the pipeline.** The quality template's "Plan alignment" dimension re-checks what the spec reviewer just verified. Reviewers re-run test suites the implementer already ran (and reported, with TDD evidence) on identical code.
4. **Per-task and final review share one template**, so there is no representation of "per-task narrow, final broad" anywhere.
A field report (`~/2026-06-09-code-quality-reviewer-scope-budget-issue.md`) first flagged this. Its cited session and headline numbers could not be verified, but its qualitative diagnosis was confirmed against two real local sessions. One correction to it: cross-cutting audits (lock ordering, changed contracts) are sometimes the *correct* review method — the fix must gate breadth behind a stated concrete risk, not forbid it.
## Goals
- Per-task reviews scoped to the task: diff-first reading, justified broadening, no redundant test runs.
- Final whole-branch review keeps its current breadth.
- No reduction in what reviews catch.
## Non-goals / explicitly preserved
- **Full re-reviews stay.** When a reviewer re-reviews after a fix, it still reviews the whole task at full reading breadth. (It does not re-run tests the implementer just ran on the amended code.) This deliberately rejects the field report's "re-review budget" remedy: the cost of its worst cited example (a re-review running `-race` and `-count=100` loops) is curbed by the test budget below, not by narrowing what re-reviewers read.
- ~~**The two review stages stay separate.** Spec compliance and code quality remain independent subagents, serially gated. No merging.~~ **Superseded by the cost iterations below**: live eval economics showed per-dispatch overhead dominating cost, and the maintainer put everything on the table. The per-task stages are now one task reviewer with two verdicts; the independent broad final review remains.
- **The coordinator keeps model judgment.** No forced model tier for reviews, in either direction.
- **`requesting-code-review/` is untouched.** It remains the broad template for final branch review and ad-hoc review.
- Verdict ordering (spec compliance reported before quality), the fix-and-re-review loops, and the requirement to fix Critical/Important findings are unchanged.
## Cost iterations (post-launch eval economics)
Live before/after runs surfaced a cost regression once the quality-hardening
prose (evidence rule, constraint carrying, pristine output) landed: go-fractals
went from 42.8 min / 14.5M tokens (first task-scoped version) to 69.9 min /
32.2M (hardened version) while reaching baseline-parity quality (blind-judged
8.5 vs 8.5). Per-subagent turn profiling attributed cost to, in order: cheap
models taking 2-3× the turns on multi-step work (678 of 1197 subagent turns
were haiku), per-dispatch overhead (3 subagent spin-ups per task, each
re-deriving the diff; controller coordination was half the dollars), and
evidence-rule narration.
- **Iteration 1:** turn-count-beats-token-price model guidance (mid-tier floor
for multi-step work), optional inline diffs, cite-don't-narrate evidence,
Important = cannot-trust-until-fixed, fixes dispatched only for
Critical/Important. Result: 68.2 min / 22.9M — tokens down 29%, wall-clock
flat; controllers pasted the diff in only 2 of 22 review dispatches when
phrasing was optional.
- **Iteration 2:** per-task spec and quality reviews merged into one
`task-reviewer-prompt.md` (one reviewer, one reading of the diff, two
verdicts; one fix dispatch addresses both kinds of findings); implementers
run the focused test while iterating, full suite once before commit.
Result (go-fractals): 47.5 min / 15.7M / $13.55 — beat baseline on every
axis, blind-judged 9/10 vs baseline 7/10.
- **Iteration 3:** Calibration names merge-blocking maintainability damage
(verbatim duplication, swallowed errors, assertion-free tests) as
Important and Minor findings must be pasted into the final review for
triage; reviewer skepticism extended to the implementer's design
rationales ("left it per YAGNI" is a claim, not a verdict); diff handed
to reviewers as a file (`git diff > /tmp/sdd-task-N.diff`, redirected so
it never enters the controller's context; one Read call for the
reviewer) after paste-into-prompt guidance went unadopted (0-6 of 11-17
dispatches) for locally-rational context-economics reasons.
- **Final frozen config (e355795), all five scenarios pass:** go-fractals
44.4 min / 13.4M / $11.67 (-32% time, -37% tokens, -27% dollars vs
baseline); svelte-todo 62.8 / 19.7M / $15.76 (-21% / -28% / -25%);
rejects-extra-features $1.31 (vs $1.88); spec-reviewer-flaws flat; the
planted-defect scenario (v3: open-flag transparency bar for judgment
calls, must-fix bar for a test whose name promises verification it
never performs) passes with the defect caught and fixed.
### Iterations 4-5 (2026-06-10): variance honesty, structural fixes, positive recipes
A same-config re-run exposed run-to-run variance (44.4→57.1 min on
identical prompts; reviewer escape-hatch appetite swung 1.0→6.3 tool
calls/review), so all subsequent claims use ranges. Five parallel
experiment variants on go-fractals plus transcript mining of real local
sessions (full log with negative results:
`evals/docs/experiments/2026-06-10-sdd-cost-experiments.md`) produced the
final config:
- **Adopted:** final-review package (final reviewer 33→6 turns at
controller-model prices); REQUIRED `model:` line in both templates
(prose guidance decayed mid-session once, inheriting opus for 17
dispatches, +$5); task-brief + report files (`scripts/task-brief`;
fidelity anchor, modest context savings); progress ledger in
`<git-dir>/sdd/progress.md` (real sessions re-dispatched entire
completed task sequences after compaction — 269 dispatches for ~22
tasks); omnibus final fixer (a real session's per-finding fix wave cost
more than all its tasks); scoped fix tests; unique SHA-range collateral
names (worktree/submodule-safe); dispatch-composition recipe and
reviewer named-risk budget (micro-tested: positive recipe 3.0
transcribed values vs prohibition 4.4 vs control 3.6 — prohibitions can
backfire; see `2026-06-10-positive-instruction-redesign-design.md`).
- **Tested and declined:** controller turn batching and parallel-call
pipelining (controller emits exactly one tool call per message — 0
multi-tool messages in every run; 46% of its turns are
thinking/narration, a prompt-immune floor); background-dispatch
pipelining (mechanism adopted 7/28 but benefit below the ±6 min noise
floor on these scenarios).
- **Final validated config (b81f35b family), all gates pass:** go-fractals
54.1-54.7 min / 14.4-16.6M / $12.81-14.31 (baseline 64.9 / 21.2M /
$16.07); svelte-todo 55.0 min / 19.3M / $14.99 (baseline 79.7 / 27.3M /
$20.98); planted-defect pass / $2.77. Across all 8 same-design fractals
runs: 44.4-57.1 min / 13.4-20.0M / $11.67-14.84 — the worst draw beats
baseline on every axis; typical mid-band savings ~20-25%.
## Design
### Shared principle: don't re-run tests on code that hasn't changed
The implementer's report includes test results and TDD RED/GREEN evidence for exactly the code under review. Reviewers verify by reading. A reviewer runs a test only when reading raises a specific doubt that no existing run answers — and then a focused test, not a suite. On harnesses where reviewer subagents are read-only (e.g., Antigravity maps reviewer templates to the `research` type, which has no command access), the reviewer instead names the test it would run in its report.
After a fix, the implementer re-runs the tests covering the amended code; the re-reviewer does not repeat that run. Today nothing enforces that premise: `implementer-prompt.md` describes the initial implement-test-commit flow only, with no fix-iteration instruction. This spec therefore also adds to `implementer-prompt.md`: after fixing a review finding, re-run the tests that cover the amended code and include the results in the fix report.
This principle appears in both reviewer prompts, the implementer prompt, and the controller guidance.
### 1. New file: `skills/subagent-driven-development/code-quality-reviewer-prompt.md` becomes self-contained
Stop delegating to `requesting-code-review/code-reviewer.md`. The per-task quality reviewer gets its own scoped prompt template:
- **Framing:** "You are reviewing one task's implementation for code quality." A task-scoped gate, not a merge review.
- **Spec compliance is settled:** spec review already passed; do not re-litigate requirements or plan alignment.
- **Review dimensions kept:** code quality (clarity, duplication, error handling), test quality (real behavior, not mocks), maintainability, and the existing SDD-specific checks (single responsibility, independent testability, file structure from plan, file growth contributed by this change). Dropped: plan alignment, security/scalability/production-readiness dimensions, merge verdict.
- **Scope budget:** start from `git diff BASE..HEAD`; read changed files first; inspect adjacent code only to evaluate a concrete risk you can name. Cross-cutting changes — lock ordering, changed function/API contracts, shared mutable state — are legitimate named risks that justify checking call sites. Do not crawl the codebase by default.
- **Test budget:** the shared principle above, plus: no package-wide suites, race detectors, or repeated/high-count runs unless you have first named a specific suspected flake or race. Otherwise, recommend heavy validation in the report instead of running it. Warnings or noise in the implementer's reported test output are findings — output should be pristine (the implementer's self-review checks this too).
- **Evidence rule:** reviewers answer each What-to-Check item with file:line evidence, not bare yes/no. (Added after live eval runs showed reviewers passing defects the prompt had pointed them at — an accessible-name check and a temp-dir-cleanup check both got unsupported "yes" answers while the defect sat in the reviewed diff.)
- **Read-only rule** kept in trimmed form: no mutating the working tree, index, HEAD, or branch state. The `git worktree add` how-to sentence from the current templates is NOT carried into this file — a diff-scoped review never needs a checkout of another revision (same rationale as the spec-prompt cleanup below).
- **Verdict:** Strengths / Issues (Critical/Important/Minor) / "Task quality: Approved | Needs fixes."
### 2. `skills/subagent-driven-development/spec-reviewer-prompt.md` cleanups
- Remove the `git worktree add` how-to sentence. The read-only rule stays; a diff-scoped spec review never needs a checkout of another revision.
- Resolve the tension between the diff-only guard and "verify everything independently": spec compliance is judged by reading the diff against the requirements. The implementer's TDD evidence covers "it runs" — apply the shared test principle.
- New third verdict channel: requirements that cannot be verified from the diff (live in unchanged code, span tasks) are reported as explicit "⚠️ Cannot verify from diff — controller should check X" items, instead of either crawling or silently passing. The flowchart's binary pass/fail diamond cannot route this, so the controller guidance (§3) defines the handling: ⚠️ items do not block dispatching the quality reviewer, but the controller must resolve each one itself (it holds the plan and cross-task context) before marking the task complete; an item the controller confirms is a real gap is treated as a failed spec review and goes back to the implementer.
- Replace the fabricated premise "The implementer finished suspiciously quickly" with grounded skepticism: treat the implementer's report as unverified claims about the code. Same distrust, no invented fact.
### 3. `skills/subagent-driven-development/SKILL.md` controller changes
- **Model Selection:** replace "Architecture, design, and review tasks: use the most capable available model" with judgment guidance — pick reviewer models the way implementer models are picked, scaled to the diff's size, complexity, and risk. The "Task complexity signals" list is rescoped to make clear its bullets describe implementation tasks; reviewer model choice follows the same judgment, so a narrow diff review does not automatically map to "broad codebase understanding → most capable model."
- **Reviewer prompt construction** (new guidance near Red Flags): when dispatching reviewers, do not write open-ended directives ("check all uses," "run race tests if useful") without a concrete task-specific reason; do not ask reviewers to re-run tests the implementer already ran on the same code; do not pre-judge findings for the reviewer (never instruct a reviewer to ignore or not flag a specific issue — adjudicate suspected false positives in the review loop instead); per-task reviews are task-scoped gates — the broad review happens once, at the final whole-branch review. (The pre-judging rule was added after a live eval run caught the controller fabricating a "the plan forbids a shared helper" claim and instructing the quality reviewer not to flag a planted DRY violation.) Controllers must also include the spec/design's global constraints that bind the task — version floors, naming and copy rules, platform requirements — in the requirements they paste: a live run shipped a `go 1.26.1` module floor against a "Go 1.21+" design because no reviewer ever saw the constraint. And controllers must specify a model explicitly on every dispatch — an omitted model inherits the session's (usually most expensive) model, which silently defeats model selection.
- **Handling spec-reviewer ⚠️ items** (new guidance, alongside Handling Implementer Status): the controller resolves each "cannot verify from diff" item itself before marking the task complete; confirmed gaps go back to the implementer as failed spec review.
- **Final review stays broad, explicitly:** the final whole-branch reviewer dispatch node gains an explicit pointer to `../requesting-code-review/code-reviewer.md`. (Today that template is reachable only through the per-task quality prompt's delegation; once that delegation is removed, an unreferenced final-review template would be orphaned.) The Integration section's note that `superpowers:requesting-code-review` provides "the code review template for reviewer subagents" is corrected to apply to the final review only.
- **Example workflow:** the quality-reviewer lines in the example are updated to the new verdict vocabulary ("Task quality: Approved"); the final reviewer's "ready to merge" line stays.
- Flowchart topology is unchanged; the ⚠️ channel is handled by controller guidance, not a new graph branch.
## What this does not fix (known, deferred)
The spec reviewer judges against task text the controller pasted; it cannot catch requirements dropped during the controller's extraction from the plan. That is an architectural property of "controller provides full text," not a prompt problem, and is out of scope here.
## Verification
- Plugin infrastructure tests (`tests/`) still pass.
- Run the SDD skill-behavior evals (`git submodule update --init evals`, then per `evals/README.md`) before and after the change. Specifically: `sdd-go-fractals`, `sdd-svelte-todo`, `sdd-rejects-extra-features` (end-to-end SDD including the spec reviewer's YAGNI gate), and `spec-reviewer-catches-planted-flaws`.
- Known eval gaps this change exposes: no existing scenario plants a code-quality defect inside a single SDD task and asserts the per-task quality reviewer catches it, and no scenario measures per-reviewer exploration cost (tool-call/grep counts). Add one scenario covering the first gap (planted single-task quality defect → per-task reviewer must flag it before final review). For exploration cost, compare reviewer subagent tool-call counts manually across the before/after eval transcripts.
@@ -0,0 +1,178 @@
# Positive-Instruction Redesign of Skill Guidance — Design Spec
**Status:** Proposed (follow-up to the 2026-06-09 SDD review-dispatch work; separate PR per the one-problem-per-PR rule)
**Driver:** Measured evidence (2026-06-10) that some negative instructions in skill prose backfire, while others work — and that the difference is predictable.
## The measured finding this spec generalizes
Micro-tests on 2026-06-10 (opus, 5 reps per phrasing, programmatic scoring;
harness described below) measured how guidance phrasing changes what a
controller composes:
| Case | Phrasing | Result |
|---|---|---|
| Dispatch composition ("don't restate the brief") | prohibition | **4.4** spec values re-typed — *worse than no guidance* (3.6) |
| Dispatch composition | positive recipe ("your dispatch should contain: (1)…(5)") | **3.0, zero variance** — adopted |
| Dispatch composition | recipe + nuance clause ("quote only the fragment…") | 3.8, noisy — nuance dilutes recipes |
| Test-rerun directive ("do not ask reviewer to re-run tests") | prohibition | **0/5 violations** — works fine (control: 3/5) |
| Test-rerun directive | positive recipe | 0/5 — equal, but longer |
**The doctrine** (use this to classify any negative instruction):
1. **Tripwires work.** Phrase-level self-checks on concrete tokens ("if the
prompt you are writing contains 'do not flag' … stop") fire reliably.
2. **Recognition tables work.** Red-Flags/rationalization tables read at
decision time, not composition time.
3. **Discrete-directive prohibitions work.** "Do not ask X to do Y" holds
when the model has no competing incentive to do Y.
4. **Composition prohibitions backfire** when the model has its own agenda
for the output (e.g., restating specs feels like helpful curation).
Only a positive composition recipe moves these — and adding nuance
clauses to a winning recipe makes it worse, not better.
5. **Ties go to the shorter phrasing.** Codex re-reads SKILL.md ~500× per
long session (measured 2026-06-10); prose length is a real cost.
## Audit results (2026-06-10, all ~30 skills + prompt templates)
Counts: 3 tripwires (keep), 14 recognition tables (keep), ~20 policy gates
(keep — "never push without permission" is policy, not composition
shaping), 5 composition-prohibitions:
| # | Location | Disposition |
|---|---|---|
| 1 | `subagent-driven-development/task-reviewer-prompt.md` — "Cite, don't narrate" | **Queued in PR #1717 batch**: lead with the positive half ("Your report should point at evidence: file:line for every finding…"), drop the prohibition half (dead weight — the positive half already exists and carries the load) |
| 2 | `subagent-driven-development/SKILL.md` — "Do not add open-ended directives" | **Keep as-is**: micro-test could not elicit the failure in 15 samples; no evidence either way; shorter wins |
| 3 | `subagent-driven-development/SKILL.md` — "Do not ask a reviewer to re-run tests" | **Keep as-is**: measured 0/5 violations; the prohibition also usefully propagates itself into dispatches |
| 4 | `subagent-driven-development/SKILL.md` — "do not re-review on top of it" | **Queued in PR #1717 batch**: replace with the three-element checklist ("Before re-dispatching the reviewer, confirm the fix report contains: the covering tests, the command run, and the output") |
| 5 | `writing-plans/SKILL.md` — the "No Placeholders" banned-patterns list | **This spec's main subject** — see below |
Borderline, deferred with #5: `task-reviewer-prompt.md` "Don't flag
pre-existing file sizes — focus on what this change contributed" (positive
half present and load-bearing; low impact; test alongside #5 if convenient).
## The writing-plans change (deferred item #5)
### Current state
`skills/writing-plans/SKILL.md`, "No Placeholders": one positive sentence
("Every step must contain the actual content an engineer needs") followed
by a six-bullet banned-patterns list ("never write them: 'TBD', 'TODO',
'Add appropriate error handling', 'Write tests for the above', 'Similar to
Task N', …").
### Why it matters and why it is genuinely uncertain
- Plans are the **largest generated artifact** in the workflow, and the
model has a real competing incentive to emit placeholders (they are the
path of least effort under length pressure) — the incentive structure of
the case where prohibition measurably backfired.
- But the banned items are **discrete, recognizable tokens** — the shape
of the case where prohibition measurably held.
- **The list is load-bearing elsewhere:** the skill's Self-Review section
references it ("Placeholder scan: search your plan for red flags — any
of the patterns from the 'No Placeholders' section above"). The tokens
double as the review-time scan inventory, and review-time recognition is
the category that works. A naive swap to a positive checklist breaks
that reference and discards good tripwire tokens.
### Variants to test
- **V0 (current):** positive sentence + banned list at composition time;
Self-Review references the list.
- **V1 (auditor's checklist):** composition-time positive recipe only —
"Before finalizing a step, confirm it has: the literal code to write, a
runnable command with expected output, types and method names defined
within this plan, error handling shown explicitly. A step is complete
when an engineer could implement it without asking any follow-up
questions." Self-Review keeps a generic placeholder scan.
- **V2 (restructure by mechanism — predicted winner):** composition time
gets only V1's positive recipe; the named patterns move wholesale into
the Self-Review placeholder-scan step, reframed as recognition ("when
you scan, look for: 'TBD', 'TODO', 'Similar to Task N', …"). Same
tokens, relocated from the category that primes to the category that
detects.
- **V3 (control):** positive sentence only, no list anywhere.
### Micro-test design
- **Task:** opus writes a 2-3 task implementation plan from a deliberately
under-specified spec (under-specification is what tempts placeholders).
Use a fixture spec with: one well-specified task, one task whose error
handling the spec hand-waves, one task similar to the first (tempting
"Similar to Task 1").
- **Sampling:** 5+ reps per variant, default temperature, model
`claude-opus-4-8` (the model that writes plans in practice).
- **Programmatic scoring** (lower is better unless noted):
- banned-token count: `TBD|TODO|implement later|fill in details|appropriate error handling|handle edge cases|Similar to Task|Write tests for the above`
- steps lacking a fenced code block where the step changes code
- references to types/functions not defined anywhere in the plan output
- (higher is better) runnable commands with expected output per task
- **Two-stage scoring for V2:** also test the Self-Review half — feed each
generated plan back with the variant's Self-Review section and measure
whether the scan actually catches seeded placeholders (insert 2 known
placeholders into a fixture plan; detection rate is the metric).
- **Acceptance:** adopt a variant only if it beats V0 on banned-token count
without losing code-block coverage or self-review detection rate.
Expected cost: ~$6-10 total.
### PR scoping
Separate PR (writing-plans is a different skill; its "No Placeholders"
list is tuned content where the contributor guidelines demand eval
evidence). The PR must include: the micro-test harness + results table,
before/after text, and the V2 relocation rationale.
## The micro-test harness (method, so it isn't lost)
`/tmp/sdd-exp/micro/run-micro.py` and `/tmp/sdd-exp/micro2/run-micro2.py`
(2026-06-10; to be committed to superpowers-evals as
`docs/superpowers/skills/micro-testing-prompt-guidance.md` + scripts):
- One API call per sample: system prompt = the skill-guidance variant in
realistic surrounding context; user = a realistic mid-workflow scenario;
output = the composed artifact (dispatch prompt, plan, report).
- Programmatic scoring with greps for unambiguous markers; **manually
inspect every match before trusting a verdict** — one of tonight's
"violations" was the controller correctly quoting the prohibition, and
automated negation detection mislabeled another.
- ~$0.15-0.30/sample, seconds per iteration vs $12/50-min full eval runs.
Iterate phrasings here; confirm winners in full runs only when the
change is structural.
- Always include a no-guidance control — tonight it revealed both a
backfire (restating: prohibition worse than nothing) and a working
prohibition (test-reruns: 3/5 control failures vs 0/5 with either
phrasing).
## Result: writing-plans micro-test (run 2026-06-10, after this spec was written)
**Resolved — no change needed.** Stage 1 (3-task spec, no pressure): 0
placeholders in all 20 plans across all four variants including the
no-guidance control. Stage 1b (10-task spec, five near-identical commands
tempting "Similar to Task N", explicit ~2,500-word economy target): 40/40
clean — the single regex hit was a V2 self-review *attesting* "no
TBD/TODO ✓". Current-generation opus does not produce plan placeholders
even under deliberate pressure, with or without the banned-patterns list.
Disposition: leave the No Placeholders section exactly as it is (it costs
little and the counterfactual is unmeasurable); do NOT open the follow-up
PR. The V2 relocation design remains on file here should a future model
generation regress.
## Also explicitly not-dropped (tested-and-declined, with data)
Recorded so nobody re-proposes them without new evidence — full numbers in
the 2026-06-09 SDD design spec's Cost-iterations section:
- **Controller turn batching / parallel tool calls in one message:** the
controller emits exactly one tool call per message (0 multi-tool
messages across every measured run, with and without guidance). 46% of
controller turns are thinking/narration with no tool call — a
prompt-immune floor.
- **Pipelined reviews via parallel calls:** dead for the same reason.
- **Pipelined reviews via `run_in_background`:** mechanism adopted when
offered (7/28 dispatches) but benefit below the run-to-run noise floor
on 45-min scenarios (reviews are only ~30-60s each); adds dual
result-stream coordination. Worth revisiting only for plans whose
reviews are individually long.
- **Nuance clauses appended to winning recipes:** measurably degrade them
(C2: 3.8 noisy vs C: 3.0 consistent). Iterate by re-deriving the recipe,
not by appending caveats.
@@ -0,0 +1,265 @@
# Strict-Cost SDD — Design Spec
**Status:** Proposed experiment ladder (not implementation). Each rung ships
only with its gate evidence; abort any rung whose gates fail.
**Objective:** minimize dollars per plan-execution. Wall-clock is
unconstrained; token count matters only as a cost driver.
**Hard invariant:** quality. Concretely: `sdd-quality-reviewer-catches-
planted-defect` pass rate over **N=5 runs** (not 1 — single-run gates were
this campaign's weakest methodology), `sdd-rejects-extra-features` pass,
all end-to-end scenarios pass, blind A/B deliverable parity with the
current config. Any quality regression kills the rung, full stop.
## Where the dollars are (final 2026-06-10 config, go-fractals, ~$13/run)
| Component | $ | Driver |
|---|---|---|
| Controller (session model, opus) | ~6-7 | ~150 turns × resident context; prompt-immune turn floor (46% thinking/narration) |
| Implementers (sonnet, 10-13 dispatches) | ~5-6 | the actual work; ~25 turns each; ~13 pre-edit exploration calls each |
| Task reviewers (sonnet, 10) | ~1-1.5 | 3-9 turns each with package |
| Final review + fixes | ~1 | 6 turns with branch package |
Review-loop count (2-4 per run) is the biggest run-to-run cost variance;
loops are mostly caused by plan ambiguity the implementer resolved wrongly.
## Judgment guardrail (co-invariant with quality)
**Cheapen mechanics, never judgment.** Every rung must enumerate which
decisions it moves to a cheaper model and show each is *mechanical*
deterministic, scriptable, or cheaply verifiable after the fact. Judgment
stays at the highest tier or with the human. The judgment points in SDD,
explicitly:
- **BLOCKED / NEEDS_CONTEXT handling** — diagnosing why a subagent is stuck
and choosing the remedy
- **⚠️ "cannot verify from diff" resolution** — the controller adjudicating
with cross-task context
- **Dispatch curation** — ambiguity resolution and task-boundary drawing
(measured load-bearing: the Task 5 gradient-direction note prevented a
wrong implementation)
- **Review verdicts and severity calibration** — what is Important vs Minor
- **Review-loop adjudication** — deciding a finding is a false positive
- **Escalate-to-human recognition** — knowing the plan itself is wrong
A rung that would move any of these to a cheaper model must either (a)
restructure so the decision is made once by the expensive model at plan
time, (b) add an explicit escalation rule routing it back up at execution
time, or (c) die. "The cheap model usually gets it right" is not
acceptance evidence — judgment failures are rare-event, high-blast-radius,
and largely invisible to pass/fail gates, which is why every tier change
below carries a judgment audit (session-resume interrogation of each
judgment point in the gate runs, compared against the expensive-controller
baseline) in addition to the N=5 scenario gates.
## Thesis guardrail
SDD's thesis: **a fresh subagent per task with precisely curated context,
gated per task.** Rungs below must preserve it. Dispatch-time task batching
(one implementer dispatch handling several plan tasks) is **counter-thesis**
— it pollutes the fresh-context property and coarsens the gates — and is
deliberately NOT on the ladder. The thesis-compatible route to the same
dispatch economics is plan-time task right-sizing (L1): if the plan defines
fewer, better-sized tasks, SDD still runs one fresh subagent per task.
## The ladder (in expected $/leverage order)
### L1 — Plan-side crispness (writing-plans changes; est. $1.5-3/run, plus variance reduction)
**Status 2026-06-11 (final): elicitation tested end-to-end; claims
re-attributed.** Micro-tests: constraints header and Interfaces blocks
elicit deterministically (0→5/5, 0→100% of tasks, exact values);
right-sizing is modest and scale-dependent (9.4→8.4 tasks at svelte
scale, nothing to move at fractals scale). Full runs: an elicited plan
executed at $6.34/$8.49 — but the no-guidance control (opus plan,
complete code) hit $7.59/$7.73, inside that range. **The cost win
belongs to opus-written complete-code plans; the hand-written prose
fixture plans all prior numbers used are unrepresentative and ~2×
costlier to execute.** The guidance owns fidelity and variance instead:
deterministic constraints propagation (the one elicited-run fix was a
version-floor catch), exact cross-task interfaces, fix waves 1 vs 2-4
(the control plan shipped a real Sierpinski bug both runs had to fix).
The writing-plans PR claims those grounds, not dollars. Draft at
/tmp/sdd-exp/writing-plans-l1 (branch writing-plans-crisp).
The plan is upstream of every cost: task count sets dispatch count; plan
ambiguity sets review-loop count; plan completeness sets implementer
exploration. Current writing-plans optimizes for implementer success, not
execution economics. Changes to test:
1. **Task right-sizing guidance.** Today's plans produce tasks as small as
"create .gitignore" — each costing a full dispatch + review cycle
(~$0.60-1.00 fixed overhead). Add: "A task is the smallest unit that
carries its own test cycle and is worth a fresh reviewer's gate. Merge
setup/config steps into the task that needs them; split only at
boundaries where a reviewer could meaningfully reject." Fractals' plan
would drop from 10 tasks to ~7. Validate: dispatch count falls, gates
hold, review granularity still catches the planted defect.
2. **Structured `## Global Constraints` section** in the plan header
(version floors, naming/copy rules, platform requirements). Today these
live in design.md prose and reach reviewers only if the controller
remembers to paste them (a `go 1.26.1` floor violation shipped because
none did). A fixed heading makes them mechanically extractable —
`task-brief` can append them to every brief automatically (small script
change), removing a controller responsibility entirely.
3. **Per-task `Interfaces:` line** (consumes/produces, exact signatures).
The controller currently re-derives cross-task interfaces per dispatch
(its main legitimate "restating"), and implementers spend ~13 tool calls
re-discovering context. The planner already knows the interfaces; one
line per task moves the work to where it is done once.
4. **Per-task model-tier recommendation** from the planner ("mechanical /
standard / judgment"). The planner has the best information for the
Model Selection decision the controller currently re-makes per dispatch;
the controller keeps override authority.
Validation: micro-test the planner output shape (recipe-style, per the
instruction-design doctrine), then full runs. Note the 2026-06-10 result:
plan *placeholders* cannot be elicited from current opus — these changes
target economics and ambiguity, not placeholder hygiene.
### L2 — Controller tier (est. $4-5/run; the biggest single lever, gated hardest)
**Status 2026-06-11 (final): DIED AT THE GATES, as pre-registered — with
useful anatomy.** Recon was positive ($6.68/$8.05, n=2, mechanics clean).
The full battery split the judgment surface: the new
`sdd-escalates-broken-plan` scenario (explicit plan self-contradiction;
the human never volunteers it) passed **5/5 at sonnet** ($1.02-1.37/run;
opus baseline 2/2) — explicit conflicts get escalated. But the
planted-defect battery failed decisively: under a sonnet controller the
per-task quality gate collapsed into plan-compliance advocacy ("no
assertion, as required" listed under Strengths), the defect shipped in
4/5 runs (deterministic check), and only the tier-pinned opus final
reviewer ever caught it — while the same sonnet-tier reviewers under an
opus controller flagged it 5/5. Cheap controllers handle explicit
escalation; they absorb implicit authority-vs-quality adjudication.
A possible L2b (discrete rule: "a reviewer finding that conflicts with
the plan's text is the human's decision — escalate it") would route the
failing judgment through the escalation behavior that held.
**L2b tested 2026-06-11 (E35/E36, evals
`docs/experiments/2026-06-11-build-loop-autoresearch.md`): improves the
opus stack, does NOT rescue the sonnet rung.** Two rules: a reviewer
tripwire (a plan-mandated defect IS a finding — Important, labeled
plan-mandated; the human decides) and a controller escalation rule
(plan-mandated findings go to the human like any plan contradiction).
Micro on frozen sonnet-composed inputs: 0/6 → 6/6 labeled findings.
Full battery: opus controllers 2/2 internalized the rule, caught their
reviewer's miss as self-described backstop, and escalated for a
sanctioned fix (the 4241 ad-hoc behavior made structural); escalation
sanity 2/2 unbroken. Sonnet controllers: 1/5 full pass — paraphrase
drops the tripwire from dispatches (2/5 transmitted), transmission
alone doesn't fire it live (read-once dilution across the reviewer's
tool reads; placement within the dispatch refuted as the variable),
and no sonnet controller showed backstop behavior; 1/5 shipped the
defect. The L2b rules are a candidate commit for the opus stack.
A future L2c for the sonnet rung would pair the SKILL.md
constraints-recipe (the one channel sonnet transmits verbatim) with a
mandatory output-format slot for plan-mandated findings (the skeleton
survives every observed paraphrase and is consulted at composition
time); untested. Original recon notes follow.
**Recon (superseded):**
Sonnet-controller runs (claude-sonnet coding-agent): all gates green at
**$6.68 and $8.05** / 31-41 min (combo band $11.67-14.84), tokens inside
the combo band — no cheap-controller turn inflation. 26/26 and 31/31
dispatches model-explicit, with heavier (and sane) haiku tiering than
opus controllers showed; review loops, per-task Important→fix→re-review,
and omnibus-fixer rules followed in both runs; the run-1 controller
caught a fixer side-effect (`go mod tidy` removed cobra) before
re-review — real adjudication, not silent absorption. But neither run
surfaced a BLOCKED/⚠️ event (the escalation points were never stressed)
and final reviews ran on sonnet rather than the most capable tier. The
N=5 quality gates + full judgment audit below remain mandatory before
any skill change.
The controller is half the dollars solely because it inherits the session
model. Its turn floor is prompt-immune, so the lever is the rate per turn —
but the controller is also where most judgment points live, so this rung is
designed judgment-first:
1. **Primary form — judgment moved up front, mechanics cheapened:** the
expensive model does the judgment-dense work at plan time (L1's
Interfaces lines, ambiguity resolutions, per-task constraints — i.e.
the dispatch curation is pre-written into the plan). The mid-tier
execution session then runs a loop that is genuinely mechanical:
extract brief, dispatch, run script, route verdicts. Explicit
escalation rules in the skill: on BLOCKED, on any ⚠️ item, on a
suspected false positive, or on anything the plan does not already
answer, the cheap controller STOPS and escalates (to the human, or to
a fresh expensive-model consultation dispatch) — it never resolves
judgment alone.
2. **Gates beyond the standard N=5:** a judgment audit — every
BLOCKED/⚠️/adjudication event in the gate runs interrogated via
session-resume and scored against how the opus-controller baseline
handled the same class of event; any silently-absorbed judgment call
(cheap controller resolving what it should have escalated) fails the
rung regardless of scenario verdicts.
3. **User authority preserved:** the skill recommends, never enforces, the
execution-session tier.
Caveat from this campaign: cheap-model turn inflation was measured on
multi-step *work*, not dispatch loops; whether a mid-tier controller holds
~150 turns is part of what the experiment determines.
### L3 — Reviewer tier (est. $0.7-1/run; most likely rung to die on the judgment guardrail)
**Status 2026-06-11: DEAD, as pre-registered.** Planted-defect ×5 with
forced-haiku task reviewers: 2 pass / 1 indeterminate / 2 fail (baseline
5/5); per-task haiku cleanly flagged 0 of 10 planted defects at correct
severity — 1 found-but-downgraded with the exact prohibited rationale,
9 missed or rationalized (DRY praised as YAGNI; assert-nothing test
called plan-compliant). Cheap reviewers fail by *advocating* for
defects; passing runs survived only on controller redundancy or the
final review. Recorded in the experiments log, Batch A-E. Do not
re-propose without a structurally different design.
The package reviewer is near-single-step mechanically (3 turns / 1 Read
when calm), which invalidates the original turn-inflation rationale for the
mid-tier floor — but reviewing is judgment through and through: severity
calibration, spec verdicts, knowing what not to flag. Mechanical cheapness
does not make the decisions mechanical. Test haiku-with-package only with
the full judgment battery: planted-defect ×5, a severity-calibration check
(seeded Minor-vs-Important pairs; miscalibration fails the rung), and the
escape-hatch variance re-measured at that tier. Prior expectation: this
rung dies, and that is a fine outcome — it converts "we suspect cheap
reviewers are bad" into recorded evidence.
### L4 — Resident-context diet (est. $0.5-1/run)
- `task-brief --list` mode: controller reads task headings + Global
Constraints, never the full plan (the plan body is already delivered via
briefs).
- Reports trim 15 → 8 lines.
- SKILL.md minification pass (every section added this week re-justified
at composition-recipe density; Codex pays ~10k chars × ~500 re-reads per
long session).
### L5 — Re-litigations (explicitly flagged, maintainer-vetoed or counter-thesis)
Recorded for completeness; each requires Jesse's explicit reversal before
any experiment:
- **Scoped re-reviews** (verify fix + regression scan instead of full
re-review): vetoed 2026-06-09; worth ~$0.50/run at most.
- **Dispatch-time task batching**: counter-thesis (see guardrail). L1.1
is the sanctioned form.
## Budget and sequencing
L1 and L2.1 are independent — run both first (~$80: micro-tests + 2×5-run
gates + A/B). L3 after L2 settles the controller (reviewer behavior depends
on dispatch quality; ~$25 — planted-defect runs are $2-3 each). L4 last
(cheap, but re-gate once after the stack; ~$30). Total ≲ $150 for the full
ladder with honest N=5 gates. Expected end state if every rung survives its gates: **$5-7/run on
fractals (from $12-15)**; if the judgment-sensitive rungs (L2 beyond its
primary form, L3) die as expected, **$8-10/run** — the honest target, since
the guardrail prices judgment above dollars by construction.
## Relationship to existing work
Builds on the 2026-06-09 task-scoped review dispatch design (PR #1717) and
the 2026-06-10 experiment campaign (evals
`docs/experiments/2026-06-10-sdd-cost-experiments.md` — consult the
negative-results section before adding rungs; turn-discipline and
parallel-call mechanisms are dead). Instruction wording for any new prose
follows the positive-instruction doctrine spec and gets micro-tested before
full runs. L1 is a writing-plans change → its own PR with eval evidence;
L2-L4 are SDD changes → separate PR(s).
@@ -0,0 +1,225 @@
# Visual Companion Auth Hardening Design
**Date:** 2026-06-10
**Status:** Draft for Drew review
## Goal
Fix the security and reliability gaps found in PR #1720's brainstorming visual
companion without changing the companion's core workflow or adding runtime
dependencies.
The fixes must be test-first and must leave clear automated evidence for:
- cross-origin browser tabs cannot inject companion events by riding cookies
- restart reconnect works without depending only on browser cookie behavior
- bearer keys do not remain in the visible URL after bootstrap
- `/files/*` cannot serve files outside the content directory
- future same-origin vendored UI libraries still work
## Threat Model
The companion serves agent-generated local UI for a single brainstorming
session. The important assets are:
- screen content served from the companion
- the session key
- `state/events`, which the agent reads as user feedback
- local files under the companion session directory
In scope attackers:
- a malicious browser tab on another `localhost` port
- a browser page that can make requests to the companion but should not be able
to authenticate as the companion UI
- a direct remote client when the server is bound to a non-loopback interface
- accidental leakage through URL history, referrers, or committed local state
- content-directory symlinks or path tricks that escape `/files/*`
Out of scope for this fix:
- malicious agent-authored screen HTML
- malicious same-origin vendored JavaScript loaded by a companion screen
This out-of-scope boundary is intentional. Companion screens are part of the
agent UI surface. They may use inline scripts today and may someday use
same-origin vendored libraries such as Alpine or Three.js. Protecting against
malicious screen HTML would require a larger sandboxed-iframe architecture with
a narrow message bridge; that is not the scope of this PR hardening pass.
## Current Failures
Automated and headed-browser testing found these failures in the PR branch:
1. A cross-origin localhost page can open a cookie-authenticated WebSocket and
write attacker-controlled choices to `state/events` after the real companion
page sets the cookie.
2. `/files/*` serves symlinks that point outside `content/`, including a symlink
to `state/server-info` containing the keyed URL.
3. The session key remains in the URL of the actual screen page, so same-origin
screen JavaScript and accidental referrers/history can see it.
4. The helper reconnects with a keyless `ws://host` URL. In headed Chrome, after
a same-port/same-token restart, the browser stopped presenting the cookie to
the restarted server, so the open tab stayed stuck on the tombstone until a
manual reload.
5. Shell lint and the lifecycle test need cleanup so the test pass is stable in
Codex.
## Design
### 1. Bootstrap Keyed Loads
`GET /?key=<token>` becomes a bootstrap response, not the screen response.
When the key is valid, the server:
1. sets the HttpOnly session cookie as it does today
2. returns a small HTML bootstrap page
3. the bootstrap page stores the key in tab-scoped `sessionStorage`
4. the bootstrap page navigates to `/` using `location.replace('/')`
After this, the visible screen URL is bare `/`, not `/?key=...`.
`GET /` with a valid cookie serves the current screen. `GET /` without a valid
cookie still returns the friendly 403 page. `GET /?key=<wrong>` returns 403.
Why `sessionStorage`: the helper needs a reconnect credential that survives
same-port restarts and does not depend only on cookie behavior. Because screen
HTML is trusted same-origin UI, storing the key in tab-scoped storage is
acceptable for this threat model. It is materially better than leaving the key
in the address bar, history, and referrer surface.
### 2. WebSocket Same-Origin Enforcement
WebSocket upgrades must pass both checks:
1. valid session auth by query key or cookie
2. if an `Origin` header is present, it must match the request target origin
The origin check should compare:
```text
Origin === "http://" + req.headers.host
```
Browser attacker page example:
```text
Origin: http://localhost:9999
Host: localhost:58088
```
This must be rejected even if the browser sends the companion cookie.
Legitimate companion page example:
```text
Origin: http://localhost:58088
Host: localhost:58088
```
This should be accepted when the key or cookie is valid.
Direct non-browser clients may omit `Origin`; they still need the session key.
### 3. Helper Reconnect Credential
`helper.js` should read the tab-scoped key from `sessionStorage` and append it
to the WebSocket URL:
```text
ws://<host>/?key=<stored-key>
```
If no stored key exists, the helper falls back to the current cookie-only
`ws://<host>` behavior. This preserves compatibility for already-loaded pages
that do have a valid cookie but no storage entry.
### 4. `/files/*` Containment
The file server should continue to reject empty names and dotfiles. It must also
ensure the file is a real regular file inside `CONTENT_DIR`.
Use realpath containment as the boundary:
- compute `realContentDir = fs.realpathSync(CONTENT_DIR)`
- compute `realFilePath = fs.realpathSync(filePath)`
- serve only when `realFilePath` equals a descendant of `realContentDir`
- reject symlinks and anything outside the content directory with 404
The server should keep using `path.basename` so nested paths remain unsupported.
### 5. Leak-Reduction Headers
Add conservative headers that do not block inline scripts or future same-origin
vendored libraries:
```text
Referrer-Policy: no-referrer
Cache-Control: no-store
X-Frame-Options: DENY
Content-Security-Policy: frame-ancestors 'none'
Cross-Origin-Resource-Policy: same-origin
```
Do not add a restrictive `script-src` CSP in this pass. The companion currently
injects inline helper JavaScript and future screens may load same-origin
vendored libraries.
### 6. Gitignore Durable Session State
Add `.superpowers/` to the repo root `.gitignore` so persisted companion state
and `.last-token` are not accidentally committed when using `--project-dir`.
### 7. Test Stability And Lint
Clean up shell lint warnings in the touched start/stop scripts.
Update the lifecycle test that invokes `start-server.sh --idle-timeout-minutes`
so it cannot hang under Codex's `CODEX_CI` foreground auto-detection. The test
should force background mode with `--background` when it expects the script to
return startup JSON.
## Testing Strategy
All behavior changes should be TDD:
1. write the failing focused test
2. run it and confirm it fails for the expected reason
3. implement the minimum fix
4. rerun the focused test
5. rerun the full brainstorm-server suite
Required focused regressions:
- valid keyed `/` returns bootstrap, not screen content
- bootstrap stores key in `sessionStorage` and strips the URL
- cookie-only `/` still serves screen content
- helper uses `sessionStorage` key for WebSocket URL
- same-origin cookie WebSocket opens
- cross-origin cookie WebSocket is rejected and writes no events
- direct key WebSocket still opens without `Origin`
- symlink under `content/` pointing to `state/server-info` returns 404
- security headers are present on normal HTML, bootstrap, 403, and file responses
- restart same port/token can authenticate reconnect with the stored key
- shell lint passes for touched shell scripts
- lifecycle suite does not hang under Codex
## Acceptance Criteria
- `cd tests/brainstorm-server && npm test` passes repeatedly without hanging.
- The security probe that previously wrote `attacker-injected` from another
localhost origin now fails to open the WebSocket and leaves `state/events`
unchanged.
- The symlink-to-`server-info` probe returns 404.
- A headed or headless browser keyed load ends on a bare `/` URL and the status
pill reaches Connected.
- A same-port/same-token restart reconnects automatically without manual reload.
- `scripts/lint-shell.sh` passes for the touched shell scripts.
## Deferred Work
If the project later needs to treat screen HTML as untrusted, design a separate
sandboxed iframe architecture. That should isolate generated screens on a
separate origin or sandboxed frame and expose only a narrow `postMessage` bridge
for user choices. Do not bundle that into this fix.
@@ -0,0 +1,254 @@
# Visual Companion Final Hardening Fixup Design
**Date:** 2026-06-11
**Status:** Draft for Drew review
## Goal
Finish the PR #1720 visual companion hardening pass so the branch is ready for
Jesse review with clean security behavior, deterministic tests, and a PR diff
that contains only the companion work.
This is a fixup on top of the existing auth hardening design. It should not
redesign the companion or expand the feature surface.
## Background
The previous hardening pass added keyed sessions, same-origin WebSocket checks,
URL key stripping, `/files/*` containment, leak-reduction headers, IPv6 URL
formatting, Windows lifecycle coverage, and PR evidence updates.
The final review pass found five remaining issues:
1. The root `GET /` screen-selection path can still serve symlinks or hardlinks
under `content/` that point outside the content directory.
2. When the preferred port is occupied, fallback servers can reuse a persisted
`.last-token`, creating two live same-project companion servers with the same
bearer key.
3. `stop-server.sh` can signal an unrelated `node server.cjs` process when
strong ownership proof is unavailable.
4. Some tests can pass against the wrong fallback process, leak background
processes on failure, or assume symlink support on Windows-like hosts.
5. The PR is currently conflicted because the branch contains an older `evals`
submodule bump that was handled separately.
## Non-Goals
- Do not add HTTPS tunnel or `wss://` origin semantics in this pass.
- Do not implement opt-out, free-text, or contrast-helper companion features.
- Do not vendor Alpine, Three.js, or any other JavaScript library.
- Do not attempt to sandbox malicious agent-authored screen HTML.
- Do not add backward compatibility for stale stop-server PID files unless Drew
explicitly approves that tradeoff.
## Inherited Security Invariants
This fixup preserves the auth hardening already designed and implemented:
- `.last-token` and `state/server-info` remain sensitive owner-only state.
- Fallback tokens may appear in startup JSON and `state/server-info`, but must
not be written to `.last-token`.
- Cookies remain port-named, `HttpOnly`, `SameSite=Strict`, and scoped to `/`.
- WebSocket upgrades still require a valid key or cookie.
- WebSocket `Origin` checks remain enforced when the browser supplies an
`Origin` header.
- Direct no-`Origin` clients remain allowed only when they carry the session key.
- Generated same-origin screen JavaScript and future same-origin vendored
libraries are trusted. Sandboxing malicious screen HTML remains deferred.
## Design
### 1. Rebase Onto Current `dev`
Rebase `brainstorming-companion` onto current `origin/dev` before implementation
work. Resolve the `evals` submodule conflict by taking `dev`.
After the rebase:
- `evals` must not appear in the PR diff.
- PR #1720 can still mention eval evidence that was run elsewhere, but it must
include exact external evidence: eval repo commit, scenario path, command,
result artifact path or id, and RED/GREEN outcome.
- The PR body must not imply the evals submodule bump is part of this PR.
- Any earlier PR-body text or comment implying the submodule bump is included
must be superseded by the final PR-body evidence.
### 2. Root Screen Containment
The root screen route must use the same containment boundary as `/files/*`.
`getNewestScreen()` should ignore any `.html` candidate that does not pass the
regular-file-inside-content-dir guard. That guard must resolve real paths and
ensure the served file is inside `CONTENT_DIR`. It must also preserve the
existing hardlink protection by rejecting files whose link count is not exactly
one when the platform reports link counts.
Expected behavior:
- A symlink under `content/` pointing outside `content/` is ignored.
- A hardlink under `content/` to `state/server-info` is ignored when
`fs.linkSync` succeeds and `lstat.nlink > 1`.
- If no safe screen file remains, the waiting page is served.
- Existing `/files/*` containment behavior remains unchanged: empty names,
dotfiles, symlinks, hardlinks, and directories still return 404.
### 3. Fallback Token Isolation
Port fallback must not reuse a token loaded from persisted `.last-token`.
Token source should be explicit in code:
- `BRAINSTORM_TOKEN` from the environment is an intentional operator/test
override. If the preferred port is occupied while an explicit environment
token is set, the server must fail closed instead of falling back, because the
occupied server may be using the same explicit token.
- `.last-token` is persisted state for same-port reconnect convenience. If the
server falls back because the preferred port is occupied, discard that loaded
token and generate a fresh unpersisted token for the fallback process.
- A newly generated token that was not loaded from `.last-token` can be reused
within the same process because no other live process is known to have it.
The fallback server must continue to avoid overwriting `.last-port` and
`.last-token`.
### 4. Stop-Server Ownership Proof
`start-server.sh` should create a per-start server instance id and pass it to
Node as an inert command-line argument, for example:
```text
node server.cjs --brainstorm-server-id=<id>
```
The id is not an auth credential. It is only process-ownership evidence for the
local lifecycle scripts. `server.cjs` can ignore the argument.
The id must use a shell/MSYS-safe alphabet, such as
`^[A-Za-z0-9_-]{32,64}$`. Store it in `state/server-instance-id` with
owner-only permissions.
`stop-server.sh` should read the expected id from state and only signal the PID
when the target process argv contains the exact argument
`--brainstorm-server-id=<id>` as a full argv token, not as a loose substring.
Prefer `/proc/<pid>/cmdline` when available, then fall back to wide `ps` output.
A matching instance id is sufficient proof even when `server-info` is missing
or `lsof` is unavailable. Existing port-to-PID checks may remain as additional
evidence.
Fail closed when ownership cannot be proven:
- missing PID file
- missing or malformed server id
- target command line unavailable
- target command line does not include the expected id
- old/stale session metadata without the new id
This intentionally prefers leaving a stale process running over killing an
unrelated process.
Operator-visible outcomes should be explicit:
- missing PID file returns `not_running`
- missing or malformed server id returns `stale_pid`
- unavailable command line returns `stale_pid`
- wrong or absent argv id returns `stale_pid`
- successful stop returns `stopped`
On `stale_pid` and `stopped` outcomes, remove `server.pid` and
`server-instance-id` so future stop attempts do not keep targeting the same
ambiguous process. Do not remove persistent session content.
### 5. Test Hardening
The test pass should be deterministic across macOS and the Windows Git Bash host
used for validation.
Required changes:
- Fixed-port suites must either fail fast if the server reports a fallback port
or drive all clients from the reported startup port.
- `stop-server.test.sh` needs a top-level cleanup trap before any background
process is started.
- Symlink-specific assertions should probe symlink capability and skip only that
assertion when the host cannot create usable test symlinks.
- Tests that create impostor processes must assert that the impostor survives
when lifecycle metadata is missing or insufficient.
- Windows/MSYS start-server tests must assert that Windows-like detection still
clears `BRAINSTORM_OWNER_PID`, still auto-foregrounds when appropriate, and
still passes the instance-id argv exactly.
### 6. Docs And PR Consistency
Before Jesse reviews, reconcile reviewer-visible docs and PR metadata:
- Update the issue catalog so dispositions match what this PR actually ships.
- Keep auto-open docs consistent with the implemented `--open` behavior.
- Keep the documented default idle timeout at 4 hours everywhere.
- Review the PR body against the template after the rebase.
- Record macOS, Windows, browser/manual, and external eval evidence in the PR
body with concrete commands and results.
## Testing Strategy
Use TDD for each behavior change:
1. Add or tighten a focused regression test.
2. Run it and confirm it fails for the expected reason.
3. Implement the smallest fix.
4. Rerun the focused test.
5. Rerun the full brainstorm-server suite.
Required focused regressions:
| Behavior | Test File | Focused Command | Expected RED | Expected GREEN |
| --- | --- | --- | --- | --- |
| Root route ignores symlink escape | `tests/brainstorm-server/server.test.js` | `node tests/brainstorm-server/server.test.js` | authenticated `GET /` serves linked outside content | response serves waiting page or safe screen |
| Root route ignores supported hardlink escape | `tests/brainstorm-server/server.test.js` | `node tests/brainstorm-server/server.test.js` | authenticated `GET /` serves hardlinked `server-info` | hardlink candidate is ignored when `nlink > 1` |
| `/files/*` containment stays unchanged | `tests/brainstorm-server/server.test.js` | `node tests/brainstorm-server/server.test.js` | existing containment test regresses | empty, dotfile, directory, symlink, hardlink cases remain 404 |
| Persisted-token fallback rotates token | `tests/brainstorm-server/lifecycle.test.js` | `node tests/brainstorm-server/lifecycle.test.js` | fallback URL key equals persisted preferred-port key | fallback URL key differs and is not written to `.last-token` |
| Explicit-token fallback fails closed | `tests/brainstorm-server/lifecycle.test.js` | `node tests/brainstorm-server/lifecycle.test.js` | server falls back while `BRAINSTORM_TOKEN` is set | process exits non-zero and does not start fallback |
| Fallback key cannot authenticate to original server | `tests/brainstorm-server/lifecycle.test.js` | `node tests/brainstorm-server/lifecycle.test.js` | fallback key receives 200 from original port | original port rejects fallback key |
| Correct instance id permits stop | `tests/brainstorm-server/stop-server.test.sh` | `bash tests/brainstorm-server/stop-server.test.sh` | real start-server-launched server survives | stop returns `stopped` and process exits |
| Wrong, missing, malformed, or stale id is safe | `tests/brainstorm-server/stop-server.test.sh` | `bash tests/brainstorm-server/stop-server.test.sh` | impostor is signaled | stop returns `stale_pid` and impostor survives |
| Fixed-port suites cannot pass through fallback | `tests/brainstorm-server/server.test.js`, `tests/brainstorm-server/auth.test.js` | respective `node` commands | test silently talks to fallback port | test fails clearly or uses reported port intentionally |
| Shell cleanup traps run on failures | `tests/brainstorm-server/stop-server.test.sh` | `bash tests/brainstorm-server/stop-server.test.sh` | failure leaves child processes | trap reaps background children |
| Windows/MSYS start behavior keeps lifecycle invariants | `tests/brainstorm-server/start-server.test.sh`, `tests/brainstorm-server/windows-lifecycle.test.sh` | `bash` test commands on macOS and `ballmer` | owner PID or argv handling regresses | owner PID is cleared, foreground detection holds, id argv is present |
Each RED/GREEN cycle should leave a short evidence note for the PR body: focused
command, failing assertion before the fix, passing assertion after the fix, and
whether the evidence was gathered on macOS or Windows.
## Verification
Before calling the fixup complete, run:
- `git fetch origin dev && git rebase origin/dev`
- `git diff --quiet origin/dev...HEAD -- evals`
- `gh pr view 1720 --json mergeStateStatus,statusCheckRollup,headRefOid`
- `cd tests/brainstorm-server && npm test`
- relevant focused test commands used during TDD
- `git diff --check`
- Node syntax checks for touched JavaScript files
- shell lint for touched shell files
- Windows validation on `ballmer`: full runnable brainstorm-server suite plus
the standalone Windows lifecycle probe
Manual/browser testing comes only after the automated pass is green.
## Acceptance Criteria
- PR #1720 rebases cleanly onto current `dev`.
- `evals` is absent from the PR diff.
- Root screen serving cannot read outside `content/` through symlink or
supported hardlink escapes.
- `/files/*` containment protections remain unchanged.
- No fallback server runs with a token that may be shared with the occupied
preferred-port server.
- `stop-server.sh` does not signal unrelated processes when ownership proof is
missing or ambiguous.
- `stop-server.sh` can still stop a legitimate server with a matching instance
id when `server-info` or `lsof` is unavailable.
- Focused RED/GREEN evidence is recorded for each regression.
- macOS and Windows validation evidence is recorded in the PR body.
- The PR body accurately describes what is in the branch and what evidence was
gathered externally.
+2 -1
View File
@@ -12,9 +12,10 @@ Live in `tests/`. Currently:
- `tests/brainstorm-server/` — node test suite for the brainstorm server JS code.
- `tests/opencode/` — bash tests for OpenCode plugin loading, bootstrap caching, and tool registration.
- `tests/codex-plugin-sync/` — bash sync verification.
- `tests/kimi/` — bash/Python checks for Kimi plugin manifest wiring.
- `tests/claude-code/test-helpers.sh`, `analyze-token-usage.py` — utilities used by remaining bash tests.
- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no drill counterpart; tests description-recall, not behavior).
- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (drill covers the YAGNI subset; bash adds commit-count, TodoWrite, and token telemetry assertions).
- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (drill covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions).
- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (drill covers the PRESSURE phase; bash also covers RED/GREEN baselines).
- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by drill.
+66 -130
View File
@@ -1,6 +1,8 @@
# Cross-Platform Polyglot Hooks for Claude Code
Claude Code plugins need hooks that work on Windows, macOS, and Linux. This document explains the polyglot wrapper technique that makes this possible.
Claude Code plugins need hooks that work on Windows, macOS, and Linux. This document describes the single generic dispatcher pattern used in `hooks/run-hook.cmd`.
> **Authoritative source:** `hooks/run-hook.cmd` is the canonical implementation. When this document and the code diverge, trust the code.
## The Problem
@@ -10,52 +12,22 @@ Claude Code runs hook commands through the system's default shell:
This creates several challenges:
1. **Script execution**: Windows CMD can't execute `.sh` files directly - it tries to open them in a text editor
1. **Script execution**: Windows CMD can't execute `.sh` files directly
2. **Path format**: Windows uses backslashes (`C:\path`), Unix uses forward slashes (`/path`)
3. **Environment variables**: `$VAR` syntax doesn't work in CMD
4. **No `bash` in PATH**: Even with Git Bash installed, `bash` isn't in the PATH when CMD runs
4. **`.sh` auto-prepend**: Claude Code on Windows automatically prepends `bash` to any command that contains `.sh` in its path — this interferes with the dispatcher if scripts have extensions
## The Solution: Polyglot `.cmd` Wrapper
## The Solution: Extensionless Scripts + Single Generic Dispatcher
A polyglot script is valid syntax in multiple languages simultaneously. Our wrapper is valid in both CMD and bash:
The repo uses one generic `run-hook.cmd` dispatcher for all hooks. Hook scripts are **extensionless** (`session-start`, not `session-start.sh`). This is deliberate: it prevents Claude Code's Windows auto-detection from prepending `bash` to the dispatcher command and breaking it.
```cmd
: << 'CMDBLOCK'
@echo off
"C:\Program Files\Git\bin\bash.exe" -l -c "\"$(cygpath -u \"$CLAUDE_PLUGIN_ROOT\")/hooks/session-start.sh\""
exit /b
CMDBLOCK
# Unix shell runs from here
"${CLAUDE_PLUGIN_ROOT}/hooks/session-start.sh"
```
### How It Works
#### On Windows (CMD.exe)
1. `: << 'CMDBLOCK'` - CMD sees `:` as a label (like `:label`) and ignores `<< 'CMDBLOCK'`
2. `@echo off` - Suppresses command echoing
3. The bash.exe command runs with:
- `-l` (login shell) to get proper PATH with Unix utilities
- `cygpath -u` converts Windows path to Unix format (`C:\foo``/c/foo`)
4. `exit /b` - Exits the batch script, stopping CMD here
5. Everything after `CMDBLOCK` is never reached by CMD
#### On Unix (bash/sh)
1. `: << 'CMDBLOCK'` - `:` is a no-op, `<< 'CMDBLOCK'` starts a heredoc
2. Everything until `CMDBLOCK` is consumed by the heredoc (ignored)
3. `# Unix shell runs from here` - Comment
4. The script runs directly with the Unix path
## File Structure
### File Structure
```
hooks/
├── hooks.json # Points to the .cmd wrapper
├── session-start.cmd # Polyglot wrapper (cross-platform entry point)
└── session-start.sh # Actual hook logic (bash script)
├── hooks.json # Points to run-hook.cmd with extensionless script name
├── run-hook.cmd # Cross-platform dispatcher (the polyglot wrapper)
└── session-start # Actual hook logic — extensionless bash script
```
### hooks.json
@@ -65,11 +37,12 @@ hooks/
"hooks": {
"SessionStart": [
{
"matcher": "startup|resume|clear|compact",
"matcher": "startup|clear|compact",
"hooks": [
{
"type": "command",
"command": "\"${CLAUDE_PLUGIN_ROOT}/hooks/session-start.cmd\""
"command": "\"${CLAUDE_PLUGIN_ROOT}/hooks/run-hook.cmd\" session-start",
"async": false
}
]
}
@@ -78,41 +51,63 @@ hooks/
}
```
Note: The path must be quoted because `${CLAUDE_PLUGIN_ROOT}` may contain spaces on Windows (e.g., `C:\Program Files\...`).
The path is quoted because `${CLAUDE_PLUGIN_ROOT}` may contain spaces.
## Requirements
## How `run-hook.cmd` Works at a High Level
### Windows
- **Git for Windows** must be installed (provides `bash.exe` and `cygpath`)
- Default installation path: `C:\Program Files\Git\bin\bash.exe`
- If Git is installed elsewhere, the wrapper needs modification
`run-hook.cmd` is a polyglot script: Windows treats the first block as batch
commands, while Unix shells treat that block as a no-op heredoc and continue
after it.
### Unix (macOS/Linux)
- Standard bash or sh shell
- The `.cmd` file must have execute permission (`chmod +x`)
Do not copy an implementation from this document. Read `hooks/run-hook.cmd`
directly when changing the dispatcher, and run `tests/hooks/test-session-start.sh`
afterward.
### How it works on Windows (CMD.exe)
1. The batch section validates the script name and resolves the hook directory
from the dispatcher's own location.
2. It tries bash in three places:
- `C:\Program Files\Git\bin\bash.exe`
- `C:\Program Files (x86)\Git\bin\bash.exe`
- `bash` on `PATH` (MSYS2, Cygwin, or a non-default Git install)
3. If bash is found, it runs the named extensionless hook script from the hooks
directory.
4. If no bash is found, the dispatcher exits `0` silently — the plugin
continues working, it just skips the hook.
5. `exit /b` stops CMD before it reaches the Unix section.
### How it works on Unix (bash/sh)
1. `: << 'CMDBLOCK'` opens a heredoc on a no-op command.
2. The entire CMD batch block is consumed by the heredoc and ignored.
3. After `CMDBLOCK`, bash resolves the script directory and `exec`s the named
extensionless script directly.
### Key design decisions
| Decision | Why |
|----------|-----|
| Extensionless scripts | Prevents Claude Code's Windows `.sh`-auto-prepend from interfering with the dispatcher command |
| No `-l` (login shell) | Not needed; hook scripts should be self-contained and not depend on login-shell PATH setup |
| No `cygpath` | Bash receives the Windows path directly and handles it correctly; `cygpath` was needed by the old `-c "..."` invocation pattern, not by direct exec |
| Silent exit on no-bash | Avoids breaking the plugin for users who don't have Git for Windows; hook context injection is skipped gracefully |
## Writing Cross-Platform Hook Scripts
Your actual hook logic goes in the `.sh` file. To ensure it works on Windows (via Git Bash):
Your hook logic goes in the extensionless script file. A few portable patterns:
### Do:
### Do
- Use pure bash builtins when possible
- Use `$(command)` instead of backticks
- Quote all variable expansions: `"$VAR"`
- Use `printf` or here-docs for output
### Avoid:
- External commands that may not be in PATH (sed, awk, grep)
- If you must use them, they're available in Git Bash but ensure PATH is set up (use `bash -l`)
### Avoid
- Relying on PATH-dependent tools without fallbacks (the hook runs without `-l`, so login-shell PATH is not set)
- Giving scripts a `.sh` extension — this triggers Claude Code's Windows auto-prepend
### Example: JSON Escaping Without sed/awk
### Example: JSON escaping without external tools
Instead of:
```bash
escaped=$(echo "$content" | sed 's/\\/\\\\/g' | sed 's/"/\\"/g' | awk '{printf "%s\\n", $0}')
```
Use pure bash:
```bash
escape_for_json() {
local input="$1"
@@ -133,80 +128,21 @@ escape_for_json() {
}
```
## Reusable Wrapper Pattern
For plugins with multiple hooks, you can create a generic wrapper that takes the script name as an argument:
### run-hook.cmd
```cmd
: << 'CMDBLOCK'
@echo off
set "SCRIPT_DIR=%~dp0"
set "SCRIPT_NAME=%~1"
"C:\Program Files\Git\bin\bash.exe" -l -c "cd \"$(cygpath -u \"%SCRIPT_DIR%\")\" && \"./%SCRIPT_NAME%\""
exit /b
CMDBLOCK
# Unix shell runs from here
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SCRIPT_NAME="$1"
shift
"${SCRIPT_DIR}/${SCRIPT_NAME}" "$@"
```
### hooks.json using the reusable wrapper
```json
{
"hooks": {
"SessionStart": [
{
"matcher": "startup",
"hooks": [
{
"type": "command",
"command": "\"${CLAUDE_PLUGIN_ROOT}/hooks/run-hook.cmd\" session-start.sh"
}
]
}
],
"PreToolUse": [
{
"matcher": "Bash",
"hooks": [
{
"type": "command",
"command": "\"${CLAUDE_PLUGIN_ROOT}/hooks/run-hook.cmd\" validate-bash.sh"
}
]
}
]
}
}
```
## Troubleshooting
### "bash is not recognized"
CMD can't find bash. The wrapper uses the full path `C:\Program Files\Git\bin\bash.exe`. If Git is installed elsewhere, update the path.
### "cygpath: command not found" or "dirname: command not found"
Bash isn't running as a login shell. Ensure `-l` flag is used.
CMD couldn't find bash in any of the three locations the dispatcher tries. The dispatcher exits silently (0) rather than erroring, so the hook is skipped. Install Git for Windows at the standard path or ensure `bash` is on `PATH`.
### Path has weird `\/` in it
`${CLAUDE_PLUGIN_ROOT}` expanded to a Windows path ending with backslash, then `/hooks/...` was appended. Use `cygpath` to convert the entire path.
### Hook runs on Unix but does nothing on Windows
### Script opens in text editor instead of running
The hooks.json is pointing directly to the `.sh` file. Point to the `.cmd` wrapper instead.
Check that the script filename is **extensionless** in `hooks.json`. A command like `run-hook.cmd session-start.sh` can trigger Claude Code's `.sh` auto-detection and bypass the intended CMD dispatcher path, or just try to run a non-existent `session-start.sh` script.
### Works in terminal but not as hook
Claude Code may run hooks differently. Test by simulating the hook environment:
```powershell
$env:CLAUDE_PLUGIN_ROOT = "C:\path\to\plugin"
cmd /c "C:\path\to\plugin\hooks\session-start.cmd"
```
### Hook doesn't fire at all
Verify the `matcher` in `hooks.json` matches the event type your harness emits. Claude Code uses `startup|clear|compact`; Codex uses `startup|resume|clear`. Check `hooks-codex.json` for the Codex variant.
## Related Issues
- [anthropics/claude-code#9758](https://github.com/anthropics/claude-code/issues/9758) - .sh scripts open in editor on Windows
- [anthropics/claude-code#3417](https://github.com/anthropics/claude-code/issues/3417) - Hooks don't work on Windows
- [anthropics/claude-code#6023](https://github.com/anthropics/claude-code/issues/6023) - CLAUDE_PROJECT_DIR not found
- [anthropics/claude-code#9758](https://github.com/anthropics/claude-code/issues/9758) `.sh` scripts open in editor on Windows
- [anthropics/claude-code#3417](https://github.com/anthropics/claude-code/issues/3417) Hooks don't work on Windows
Submodule
+1
Submodule evals added at 70a245c36c
-9
View File
@@ -1,9 +0,0 @@
results/
__pycache__/
*.pyc
*.egg-info/
dist/
build/
.venv/
.env
.claude/
-46
View File
@@ -1,46 +0,0 @@
# Drill
Superpowers skill compliance benchmark. Python 3.11+, managed with uv.
## Commands
- **install**: `uv sync --extra dev`
- **test**: `uv run pytest`
- **test single**: `uv run pytest tests/test_engine.py -x -q`
- **lint**: `uv run ruff check`
- **format**: `uv run ruff format`
- **typecheck**: `uv run ty check`
- **run scenario**: `uv run drill run <scenario> -b <backend>`
- **sweep**: `uv run drill run <scenario> --models claude-opus-4-6,claude-opus-4-7 --n 10`
- **compare**: `uv run drill compare <scenario>`
- **list**: `uv run drill list`
## Architecture
- `drill/engine.py` — Tmux session orchestration. Creates workdir, runs setup helpers, drives actor/agent turns, collects results.
- `drill/actor.py` — Sonnet 4.6 LLM simulating a user. Reads turn intents from scenario YAML and generates realistic prompts.
- `drill/verifier.py` — Sonnet 4.6 LLM evaluating session transcript + filesystem against semantic criteria.
- `drill/assertions.py` — Deterministic post-session checks. Runs shell commands from `verify.assertions` in the results dir.
- `drill/sweep.py` — Multi-backend, N-repetition orchestrator. Wraps Engine with try/except per run, writes run-group.json manifest.
- `drill/compare.py` — Loads results, computes pass rates and Wilson CIs, formats comparison tables.
- `drill/stats.py` — Wilson score confidence interval for pass rate estimation at small N.
- `scenarios/*.yaml` — Scenario definitions (setup, turns, limits, verify).
- `setup_helpers/*.py` — Repo fixture creators. Each creates a git repo with specific conditions.
- `backends/*.yaml` — Per-backend CLI config (args, env, idle patterns, shutdown commands).
- `bin/` — Assertion helper scripts: `tool-called`, `tool-not-called`, `tool-count`, `tool-before`, `tool-arg-match`. Run against `tool_calls.jsonl` in results dir.
## Conventions
- Setup helpers take `workdir: Path` and mutate the filesystem. Register in `setup_helpers/__init__.py`.
- Scenarios use `user_posture: naive` (no skill names) or `spec-aware` (can name skills).
- Verify criteria are semantic (LLM-evaluated). Verify assertions are deterministic (exit code 0 = pass).
- Assertions run in the results dir with `$DRILL_WORKDIR` pointing to the scenario workdir and `bin/` on PATH.
- Backend YAMLs are fully self-contained — no override/alias system.
## Required env
```
ANTHROPIC_API_KEY=sk-...
```
`SUPERPOWERS_ROOT` defaults to the parent of `evals/` (the superpowers repo root). Override only if running drill against a different superpowers checkout.
-113
View File
@@ -1,113 +0,0 @@
# Drill
Superpowers skill compliance benchmark. Drives AI coding agents through
tmux sessions and evaluates whether they follow superpowers workflows
correctly.
## How it works
1. **Setup** — a helper creates a git repo with specific conditions (worktree state, plan files, code fixtures)
2. **Actor** — a Sonnet 4.6 LLM plays the user, following turn intents from the scenario YAML
3. **Agent** — the backend under test (Claude Code, Codex, Gemini CLI) runs in a real tmux session
4. **Verifier** — a Sonnet 4.6 LLM evaluates the session transcript + filesystem against criteria
5. **Assertions** — deterministic checks (tool-called, tool-count, shell commands) run post-session
## Setup
```bash
uv sync --extra dev
```
Optional git hooks:
```bash
uv --project evals run pre-commit install
uv --project evals run pre-commit run --all-files
```
Required environment:
```bash
export ANTHROPIC_API_KEY=sk-...
```
`SUPERPOWERS_ROOT` defaults to the parent of `evals/` (the superpowers repo root) and only needs to be set if you're running drill against a different superpowers checkout.
## Usage
```bash
# Run a single scenario on a single backend
uv run drill run worktree-creation-from-main -b claude
# Run with N repetitions
uv run drill run spec-writing-blind-spot -b claude-opus-4-6 --n 5
# Sweep across multiple backends
uv run drill run spec-writing-blind-spot --models claude-opus-4-6,claude-opus-4-7 --n 10
# Compare results
uv run drill compare spec-writing-blind-spot
# List available scenarios
uv run drill list
```
## Scenarios
| Category | Scenarios | Tests |
|----------|-----------|-------|
| Worktree | 11 scenarios | Worktree creation, detection, consent, detached HEAD, and native-tool pressure |
| Skill triggering | 6 scenarios | Auto-invocation for core Superpowers skills |
| SDD workflow | 5 scenarios | Explicit invocation, mid-conversation invocation, real-project execution, and YAGNI enforcement |
| Review/spec/verification | 6 scenarios | Code review, spec review, architectural targeting, design blind spots, and verification reflexes |
| Tool mapping | 3 scenarios | Codex and Gemini subagent tool-name mapping |
## Backends
| Backend | CLI | Model |
|---------|-----|-------|
| `claude` | Claude Code | opus-4-7 (default) |
| `claude-opus-4-6` | Claude Code | opus-4-6 |
| `claude-opus-4-7` | Claude Code | opus-4-7 |
| `claude-opus-4-6-1m` | Claude Code | opus-4-6 (1M context) |
| `claude-opus-4-7-1m` | Claude Code | opus-4-7 (1M context) |
| `codex` | Codex CLI | — |
| `gemini` | Gemini CLI | auto-gemini-3 |
| `gemini-2-5-flash` | Gemini CLI | gemini-2.5-flash |
## Project structure
```
drill/ # Core engine
cli.py # Click CLI (run, compare, list)
engine.py # Tmux session orchestration
actor.py # User-simulator LLM
verifier.py # Criteria evaluator LLM
assertions.py # Deterministic post-session assertions
compare.py # Result loading and cross-backend comparison
sweep.py # Multi-backend N-rep orchestrator
stats.py # Wilson score confidence intervals
scenarios/ # YAML scenario definitions
setup_helpers/ # Repo fixture creators
backends/ # Per-backend YAML configs
bin/ # Assertion helper scripts (tool-called, tool-count, etc.)
prompts/ # Actor and verifier system prompts
fixtures/ # Static template repos
tests/ # pytest suite (122 tests)
docs/ # Design spec and manual testing guide
```
## Tests
```bash
uv run pytest
uv run ruff check
uv run ty check
```
## Writing a new scenario
1. Create a setup helper in `setup_helpers/` if you need a custom fixture
2. Register it in `setup_helpers/__init__.py`
3. Create `scenarios/your-scenario.yaml` with setup, turns, limits, and verify sections
4. Run it: `uv run drill run your-scenario -b claude`
See [docs/design.md](docs/design.md) for the full design spec.
-26
View File
@@ -1,26 +0,0 @@
name: claude-haiku
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "haiku"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"
-26
View File
@@ -1,26 +0,0 @@
name: claude-opus-4-6-1m
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "claude-opus-4-6[1m]"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"
-26
View File
@@ -1,26 +0,0 @@
name: claude-opus-4-6
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "claude-opus-4-6"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"
-26
View File
@@ -1,26 +0,0 @@
name: claude-opus-4-7-1m
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "claude-opus-4-7[1m]"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"
-26
View File
@@ -1,26 +0,0 @@
name: claude-opus-4-7
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "claude-opus-4-7"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"
-32
View File
@@ -1,32 +0,0 @@
name: claude
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "opus"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
# Matches when Claude is actively working — spinners, "Thinking", time counter,
# or "esc to cancel". Engine extends its wait deadline when any of these match
# so the Actor doesn't interrupt long-running subagent work.
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
# Maximum total seconds the engine will extend the deadline across all busy
# detections during a single _wait_for_ready call. Long-running subagent work
# can take a while, so 30 minutes gives plenty of headroom.
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"
-20
View File
@@ -1,20 +0,0 @@
name: codex
cli: codex
args:
- "--dangerously-bypass-approvals-and-sandbox"
required_env:
- OPENAI_API_KEY
hooks:
pre_run:
- symlink_superpowers
post_run: []
shutdown: "<<KEY:ctrl-d>>"
idle:
quiescence_seconds: 5
ready_pattern: "^|codex>|^>"
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.codex/sessions/rollout-*.jsonl"
-23
View File
@@ -1,23 +0,0 @@
name: gemini-2-5-flash
cli: gemini
args:
- "--yolo"
- "-m"
- "gemini-2.5-flash"
required_env: []
hooks:
pre_run:
- link_gemini_extension
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 5
ready_pattern: "Type your message|^\\s*>"
busy_pattern: "Thinking\\.\\.\\.|Executing"
startup_timeout: 60
turn_timeout: 300
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.gemini/tmp/*/chats/session-*.json"
-23
View File
@@ -1,23 +0,0 @@
name: gemini
cli: gemini
args:
- "--yolo"
- "-m"
- "auto-gemini-3"
required_env: []
hooks:
pre_run:
- link_gemini_extension
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 5
ready_pattern: "Type your message|^\\s*>"
busy_pattern: "Thinking\\.\\.\\.|Executing"
startup_timeout: 60
turn_timeout: 300
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.gemini/tmp/*/chats/session-*.json"
-54
View File
@@ -1,54 +0,0 @@
#!/usr/bin/env bash
# Verify a specific Skill was invoked before any Bash call whose command matches a regex.
#
# Usage: skill-before-tool-match <skill-name> <bash-command-regex>
# Example: skill-before-tool-match superpowers:verification-before-completion 'git[[:space:]]+commit'
#
# Semantics:
# - If no Bash call matches the regex, PASS (vacuously — the gated event never occurred).
# - If Bash matches but Skill with that name never appeared earlier, FAIL.
# - If both appeared and Skill came first, PASS.
# - If Skill never appeared but Bash matched, FAIL.
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
SKILL_NAME="$1"
BASH_REGEX="$2"
FILE="tool_calls.jsonl"
if [ ! -s "$FILE" ]; then
echo "FAIL: tool_calls.jsonl missing or empty"
exit 1
fi
# First index where Skill(skill=SKILL_NAME) appears (0-based).
SKILL_IDX=$(
jq -s --arg name "$SKILL_NAME" \
'to_entries | map(select(.value.tool == "Skill" and (.value.args.skill // "") == $name)) | first | (.key // -1)' \
"$FILE"
)
# First index where Bash(command =~ BASH_REGEX) appears.
BASH_IDX=$(
jq -s --arg re "$BASH_REGEX" \
'to_entries | map(select(.value.tool == "Bash" and ((.value.args.command // "") | test($re)))) | first | (.key // -1)' \
"$FILE"
)
if [ "$BASH_IDX" -lt 0 ]; then
echo "PASS: no Bash call matched /$BASH_REGEX/ — assertion is vacuous"
exit 0
fi
if [ "$SKILL_IDX" -lt 0 ]; then
echo "FAIL: Bash /$BASH_REGEX/ fired at line $((BASH_IDX + 1)) but Skill($SKILL_NAME) never fired"
exit 1
fi
if [ "$SKILL_IDX" -lt "$BASH_IDX" ]; then
echo "PASS: Skill($SKILL_NAME) at line $((SKILL_IDX + 1)) before Bash /$BASH_REGEX/ at line $((BASH_IDX + 1))"
exit 0
else
echo "FAIL: Skill($SKILL_NAME) at line $((SKILL_IDX + 1)) fired after Bash /$BASH_REGEX/ at line $((BASH_IDX + 1))"
exit 1
fi
-32
View File
@@ -1,32 +0,0 @@
#!/usr/bin/env bash
# Verify a specific superpowers Skill was invoked at least once.
#
# Usage: skill-called <skill-name>
# Example: skill-called superpowers:systematic-debugging
#
# Wraps the common case of `tool-arg-match Skill '.skill == "<name>"'` so
# scenario YAML doesn't have to embed jq quoting.
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
SKILL_NAME="$1"
FILE="tool_calls.jsonl"
if [ ! -s "$FILE" ]; then
echo "FAIL: tool_calls.jsonl missing or empty"
exit 1
fi
COUNT=$(
jq -s --arg name "$SKILL_NAME" \
'[.[] | select(.tool == "Skill" and (.args.skill // "") == $name)] | length' \
"$FILE"
)
if [ "$COUNT" -gt 0 ]; then
echo "PASS: Skill($SKILL_NAME) called $COUNT time(s)"
exit 0
else
echo "FAIL: Skill($SKILL_NAME) never called"
exit 1
fi
-17
View File
@@ -1,17 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL="$1"
FILTER="$2"
FILE="tool_calls.jsonl"
MATCHES=$(jq -s "[.[] | select(.tool == \"$TOOL\") | select(.args | $FILTER)] | length" "$FILE" 2>/dev/null || echo 0)
if [ "$MATCHES" -gt 0 ]; then
echo "PASS: $TOOL has $MATCHES call(s) matching filter"
exit 0
else
echo "FAIL: no $TOOL calls match filter: $FILTER"
exit 1
fi
-28
View File
@@ -1,28 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL_A="$1"
TOOL_B="$2"
FILE="tool_calls.jsonl"
IDX_A=$(jq -s 'to_entries | map(select(.value.tool == "'"$TOOL_A"'")) | first // empty | .key' "$FILE" 2>/dev/null)
IDX_B=$(jq -s 'to_entries | map(select(.value.tool == "'"$TOOL_B"'")) | first // empty | .key' "$FILE" 2>/dev/null)
if [ -z "$IDX_A" ] || [ "$IDX_A" = "null" ]; then
echo "FAIL: $TOOL_A never called"
exit 1
fi
if [ -z "$IDX_B" ] || [ "$IDX_B" = "null" ]; then
echo "FAIL: $TOOL_B never called"
exit 1
fi
if [ "$IDX_A" -lt "$IDX_B" ]; then
echo "PASS: $TOOL_A (line $((IDX_A + 1))) before $TOOL_B (line $((IDX_B + 1)))"
exit 0
else
echo "FAIL: $TOOL_A at line $((IDX_A + 1)) occurred after $TOOL_B at line $((IDX_B + 1))"
exit 1
fi
-16
View File
@@ -1,16 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL="$1"
FILE="tool_calls.jsonl"
COUNT=$(jq -s "[.[] | select(.tool == \"$TOOL\")] | length" "$FILE" 2>/dev/null || echo 0)
if [ "$COUNT" -gt 0 ]; then
echo "PASS: $TOOL called $COUNT time(s)"
exit 0
else
echo "FAIL: $TOOL never called"
exit 1
fi
-27
View File
@@ -1,27 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL="$1"
OP="$2"
EXPECTED="$3"
FILE="tool_calls.jsonl"
COUNT=$(jq -s "[.[] | select(.tool == \"$TOOL\")] | length" "$FILE" 2>/dev/null || echo 0)
case "$OP" in
eq) TEST=$(( COUNT == EXPECTED )) ;;
gt) TEST=$(( COUNT > EXPECTED )) ;;
gte) TEST=$(( COUNT >= EXPECTED )) ;;
lt) TEST=$(( COUNT < EXPECTED )) ;;
lte) TEST=$(( COUNT <= EXPECTED )) ;;
*) echo "Unknown operator: $OP (expected: eq, gt, gte, lt, lte)"; exit 2 ;;
esac
if [ "$TEST" -eq 1 ]; then
echo "PASS: $TOOL called $COUNT time(s) ($OP $EXPECTED)"
exit 0
else
echo "FAIL: $TOOL called $COUNT time(s) (expected $OP $EXPECTED)"
exit 1
fi
-53
View File
@@ -1,53 +0,0 @@
#!/usr/bin/env bash
# Verify any Bash call with command matching a regex fires before any other Bash call
# matching a second regex.
#
# Usage: tool-match-before-tool-match <tool-name> <earlier-regex> <tool-name> <later-regex>
# Example: tool-match-before-tool-match Bash 'pytest' Bash 'git[[:space:]]+commit'
#
# Semantics:
# - If no call matches the "later" regex, PASS (vacuously — the gated event never happened).
# - If the "later" call fires but no "earlier" call preceded it, FAIL.
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL_A="$1"
REGEX_A="$2"
TOOL_B="$3"
REGEX_B="$4"
FILE="tool_calls.jsonl"
if [ ! -s "$FILE" ]; then
echo "FAIL: tool_calls.jsonl missing or empty"
exit 1
fi
IDX_A=$(
jq -s --arg tool "$TOOL_A" --arg re "$REGEX_A" \
'to_entries | map(select(.value.tool == $tool and ((.value.args.command // "") | test($re)))) | first | (.key // -1)' \
"$FILE"
)
IDX_B=$(
jq -s --arg tool "$TOOL_B" --arg re "$REGEX_B" \
'to_entries | map(select(.value.tool == $tool and ((.value.args.command // "") | test($re)))) | first | (.key // -1)' \
"$FILE"
)
if [ "$IDX_B" -lt 0 ]; then
echo "PASS: no $TOOL_B call matched /$REGEX_B/ — assertion is vacuous"
exit 0
fi
if [ "$IDX_A" -lt 0 ]; then
echo "FAIL: $TOOL_B /$REGEX_B/ fired at line $((IDX_B + 1)) but no $TOOL_A /$REGEX_A/ preceded it"
exit 1
fi
if [ "$IDX_A" -lt "$IDX_B" ]; then
echo "PASS: $TOOL_A /$REGEX_A/ at line $((IDX_A + 1)) before $TOOL_B /$REGEX_B/ at line $((IDX_B + 1))"
exit 0
else
echo "FAIL: $TOOL_A /$REGEX_A/ at line $((IDX_A + 1)) fired after $TOOL_B /$REGEX_B/ at line $((IDX_B + 1))"
exit 1
fi
-16
View File
@@ -1,16 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL="$1"
FILE="tool_calls.jsonl"
COUNT=$(jq -s "[.[] | select(.tool == \"$TOOL\")] | length" "$FILE" 2>/dev/null || echo 0)
if [ "$COUNT" -eq 0 ]; then
echo "PASS: $TOOL never called"
exit 0
else
echo "FAIL: $TOOL called $COUNT time(s) (expected 0)"
exit 1
fi
-418
View File
@@ -1,418 +0,0 @@
# Drill: Superpowers Skill Compliance Benchmark
**Date:** 2026-04-07
**Ticket:** [PRI-1040](https://linear.app/prime-radiant/issue/PRI-1040)
**Status:** Design
## Thesis
The value of superpowers depends on whether skills are reliably followed by *any* coding agent — not just Claude Code. Drill tests whether agents actually fire skills, follow workflows, and use native tooling when available. It is a **compliance benchmark**, not a coding ability benchmark.
If a well-written skill produces consistent behavior across Claude Code and Codex, the agent-agnostic coordination layer is working. If agents diverge, Drill tells you exactly where and why.
## What Drill Tests
- Do agents invoke superpowers skills when they should?
- Do they follow multi-step workflows (detect → consent → create) in the right order?
- Do they use native tools (EnterWorktree, structured session logs) vs. raw shell commands?
- Where do agents diverge, and what does that tell us about skill format?
The first scenarios target **PRI-974 (worktree rototill)** — the area with the most cross-agent fragmentation today.
## Architecture
Three layers, each with a single responsibility:
```
┌─────────────────────────────────────────┐
│ CLI (click) │
│ run / compare / list │
├─────────────────────────────────────────┤
│ Engine │
│ ┌───────────┐ ┌───────┐ ┌──────────┐ │
│ │ Session │ │ Actor │ │ Verifier │ │
│ │ (tmux) │ │ (LLM) │ │ (LLM) │ │
│ └───────────┘ └───────┘ └──────────┘ │
├─────────────────────────────────────────┤
│ Backends │
│ claude / codex / (future: gemini) │
├─────────────────────────────────────────┤
│ Setup │
│ template repo + helpers + assertions │
└─────────────────────────────────────────┘
```
- **CLI** — `drill run <scenario> --backend claude`, `drill compare <scenario>`, `drill list`
- **Engine** — Orchestrates the full run lifecycle (setup → session → actor loop → collect → verify → results)
- **Session** — tmux lifecycle: create session, send-keys, capture-pane, kill session
- **Actor** — Sonnet with rolling context. Gets all scenario intents as a goal stack + terminal screens. Outputs what to type next, or `<<DONE>>`/`<<STUCK>>`.
- **Verifier** — Sonnet (near-zero temperature) with full session log + filesystem state + tool call log + criteria list. Returns per-criterion pass/fail with cited evidence + freeform observations.
- **Backends** — Each backend knows: CLI command, auto-approve flags, plugin loading, idle detection, shutdown command, session log location.
- **Setup** — Clone template repo → run backend pre_run hooks → run scenario helpers → run setup assertions → fail fast if invariants violated.
## Engine Flow
```
1. LOAD
- Parse scenario YAML
- Parse backend YAML
- Validate required env vars (fail fast)
2. SETUP
- Clone template repo to temp dir
- Run backend pre_run hooks (codex symlink, etc.)
- Run scenario setup helpers
- Run setup assertions → abort if any fail
3. SESSION
- Create tmux session (backend-specific terminal dimensions)
- Launch agent CLI in tmux pane
- Wait for startup ready pattern
4. ACTOR LOOP
- For each turn (up to max_turns):
a. Wait for idle (quiescence + ready pattern)
b. Capture terminal pane → append to rolling context
c. Send to Actor LLM: system prompt + rolling context + ALL intents + user_posture
d. Actor responds with text to type, <<DONE>>, or <<STUCK>>
e. If <<DONE>> or <<STUCK>> → break
f. Send keystrokes via tmux send-keys
g. Per-turn timeout → <<STUCK>> if exceeded
- Special keys via <<KEY:name>> convention (e.g., <<KEY:ctrl-c>>)
5. COLLECT
- Capture final terminal state
- Send shutdown command (backend-specific: /exit, Ctrl-D, etc.)
- Wait for process exit (with timeout)
- Snapshot filesystem (file tree, git state, worktree list)
- Collect backend session logs → tool_calls.jsonl
- Kill tmux session (cleanup if process didn't exit cleanly)
6. VERIFY
- Send to Verifier LLM: session.log + filesystem.json + tool_calls.jsonl + criteria
- Verifier receives criteria but NOT actor intents (reduces confirmation bias)
- Verifier returns per-criterion pass/fail with evidence + rationale + observations
- Output as structured JSON (verdict.json)
7. RESULTS
- Write to results/<scenario>/<backend>/<timestamp>/
- Print summary to stdout
```
## Backend Abstraction
Each backend is a YAML config. Backends own: CLI invocation, idle detection, shutdown, session log collection, and pre/post-run hooks.
```yaml
# backends/claude.yaml
name: claude
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: [] # no repo setup needed; plugin loaded via --plugin-dir
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:"
startup_timeout: 30
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"
match_by: timestamp
```
```yaml
# backends/codex.yaml
name: codex
cli: codex
args:
- "--dangerously-bypass-approvals-and-sandbox"
required_env:
- OPENAI_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run:
- symlink_superpowers # creates .agents/skills/superpowers symlink in test repo
post_run: []
shutdown: "<<KEY:ctrl-d>>"
idle:
quiescence_seconds: 5
ready_pattern: "codex>|^>"
startup_timeout: 30
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.codex/sessions/rollout-*.jsonl"
match_by: timestamp
```
New backends = new YAML file. Backend variants (e.g., `codex-workspace-write.yaml`) are just copies with different args — no inheritance system needed. Scenarios reference backends by name.
## Scenario Format
Scenarios are YAML. They describe *what* to test, not *how* each backend works.
```yaml
scenario: worktree-creation-from-main
description: "Agent creates an isolated worktree from main branch"
user_posture: naive # or spec-aware
setup:
helpers:
- create_base_repo
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep main"
- "git worktree list | wc -l | grep 1"
turns:
- intent: >
Ask the agent to create an isolated workspace
for building a login feature.
- intent: "Confirm consent if the agent asks."
limits:
max_turns: 20
turn_timeout: 120 # seconds per turn
verify:
criteria:
- "Agent detected it was on main, not in an existing worktree"
- "Agent asked for consent before creating the worktree"
- "A worktree or isolated workspace now exists with a feature branch"
- "Agent used the most appropriate tool available for its platform to create the worktree"
observe: true # verifier can add freeform observations
```
### User Posture
Each scenario has a `user_posture` field:
- **naive** — User describes what they want in plain language. Tests whether the agent's superpowers skills fire without hand-holding.
- **spec-aware** — User references specific skills or conventions by name. Tests whether the agent follows the spec when pointed at it.
The delta between naive and spec-aware results for the same scenario is the most interesting product signal. A small delta means strong conveyance. A large delta means the skill format needs work.
### Turn Intents
Intents are a **priority-ordered goal stack**, not a rigid script. The actor receives all intents and decides which one applies to the current terminal state. Some intents are conditional ("Confirm consent if the agent asks") and may never fire.
## Setup
### Template Repo
A real git repo checked into `fixtures/template-repo/`. Cloned to a temp directory per run. Covers the 80% common case.
Contents:
- `package.json` — minimal Node project metadata (name, version)
- `src/index.js` — simple entry point (~10 lines)
- `src/utils.js` — helper module (~10 lines)
- `README.md` — basic project description
- 3-4 commits on `main` with realistic messages (e.g., "initial commit", "add utils module", "update readme")
- No existing worktrees, branches, or tags beyond `main`
This is intentionally minimal — just enough for agents to recognize it as a real project. Scenario-specific state (extra branches, worktrees, detached HEAD) is added by setup helpers.
### Setup Helpers
Python functions in `setup_helpers/` that modify the cloned repo for specific scenarios:
- `create_base_repo(workdir)` — Clone template, verify structure
- `add_worktree(workdir, branch, path)` — Create an existing worktree (for "already inside" scenarios)
- `detach_head(workdir)` — Simulate Codex App detached HEAD state
- `symlink_superpowers(workdir)` — Create `.agents/skills/superpowers` symlink (codex pre_run hook)
### Setup Assertions
Run after all setup completes, before the agent launches. If any fail, the scenario aborts with a clear "setup invariant violated" error — not a mysterious agent failure 10 turns later.
## Plugin Loading
Each backend loads superpowers differently. The harness manages this per-run with no global config mutation:
| Backend | Mechanism | Harness action |
|---------|-----------|----------------|
| Claude Code | `--plugin-dir` CLI flag | Pass flag pointing at superpowers checkout |
| Codex | `.agents/skills/` in repo | Backend pre_run hook creates symlink |
This means Drill can test draft skill changes by pointing at a branch checkout of superpowers.
## Post-Session Tool Call Collection
Both backends write structured session logs that record every tool invocation:
| Backend | Log location | Format |
|---------|-------------|--------|
| Claude Code | `~/.claude/projects/**/session-*.jsonl` | JSONL with tool names + args |
| Codex | `~/.codex/sessions/rollout-*.jsonl` | JSONL with `LocalShellCall`, `FunctionCall`, etc. |
The harness snapshots each backend's log directory before the session starts. After shutdown, it diffs the directory to find only files created during the run — no timestamp matching needed, no cross-contamination from concurrent sessions or prior runs.
Collected logs are normalized into a common `tool_calls.jsonl` format before the verifier sees them:
```json
{"tool": "EnterWorktree", "args": {"branch": "add-login"}, "source": "native"}
{"tool": "Bash", "args": {"command": "git worktree add ..."}, "source": "shell"}
```
Each backend defines a normalizer function that maps its native log format (Claude Code's tool call entries, Codex's `ResponseItem` records) into this common schema. The verifier never sees raw backend-specific logs.
## Actor & Verifier LLM Design
### Actor
- **Model:** Sonnet
- **Temperature:** 0.7 (realistic user variation)
- **Context:** Rolling (full conversation history). Sessions are short enough (~5-20 turns) that token cost is not a concern.
- **Input:** System prompt + rolling terminal captures + all intents + user_posture
- **Output:** Structured JSON via Anthropic SDK tool_use: `{"action": "type", "text": "..."}`, `{"action": "done"}`, `{"action": "stuck"}`, or `{"action": "key", "key": "ctrl-c"}`. The harness parses this and sends keystrokes — no free-text sanitization needed.
- **Prompt:** Versioned template at `prompts/actor.md`
### Verifier
- **Model:** Sonnet
- **Temperature:** Near-zero (deterministic judgment)
- **Input:** session.log + filesystem.json + tool_calls.jsonl + criteria list. Does NOT receive actor intents or scenario narrative (reduces confirmation bias).
- **Output:** Structured JSON with per-criterion verdict/evidence/rationale + observations
- **Prompt:** Versioned template at `prompts/verifier.md`
## Results & Compare
### Results Structure
```
results/
<scenario>/
<backend>/
<timestamp>/
session.log # raw tmux capture
filesystem.json # post-run git/file state snapshot
tool_calls.jsonl # collected from backend session logs
verdict.json # verifier output
meta.json # run metadata (backend, duration, turns, model versions)
```
### Compare Command
`drill compare` reads existing results from prior `drill run` invocations. It does not run backends itself — run each backend separately first, then compare.
```
$ drill run worktree-creation-from-main --backend claude
$ drill run worktree-creation-from-main --backend codex
$ drill compare worktree-creation-from-main
Scenario: worktree-creation-from-main (naive posture)
Summary:
┌──────────┬────────┬───────┬───────┐
│ Backend │ Result │ Score │ Turns │
├──────────┼────────┼───────┼───────┤
│ claude │ PASS │ 4/4 │ 6 │
│ codex │ FAIL │ 2/4 │ 12 │
└──────────┴────────┴───────┴───────┘
Detail:
┌────────────────────────────────┬────────┬────────┐
│ Criterion │ claude │ codex │
├────────────────────────────────┼────────┼────────┤
│ Detected on main │ ✓ │ ✓ │
│ Asked consent │ ✓ │ ✗ │
│ Worktree exists │ ✓ │ ✓ │
│ Used native tools │ ✓ │ ✗ │
└────────────────────────────────┴────────┴────────┘
Observations:
claude: "Agent cited the using-git-worktrees skill by name"
codex: "Agent created worktree but skipped consent step entirely"
```
## Project Structure
```
drill/
├── drill/
│ ├── __init__.py
│ ├── cli.py # click CLI: run, compare, list
│ ├── engine.py # orchestrates the full run lifecycle
│ ├── session.py # tmux session management
│ ├── actor.py # actor LLM calls
│ ├── verifier.py # verifier LLM calls
│ ├── setup.py # template repo cloning, helpers, assertions
│ └── backend.py # loads backend YAML, builds commands
├── backends/
│ ├── claude.yaml
│ └── codex.yaml
├── prompts/
│ ├── actor.md
│ └── verifier.md
├── scenarios/
│ ├── worktree-creation-from-main.yaml
│ ├── worktree-already-inside.yaml
│ ├── worktree-codex-detached-head.yaml
│ └── worktree-consent-flow.yaml
├── fixtures/
│ └── template-repo/ # base git repo, cloned per run
├── setup_helpers/
│ ├── __init__.py
│ ├── base.py # create_base_repo, common git ops
│ └── worktree.py # add_worktree, detach_head, etc.
├── results/ # gitignored, populated by runs
├── pyproject.toml # package metadata + [project.scripts] entry point
└── README.md
```
## Phase 1 Scope
- Claude Code + Codex backends
- 4 PRI-974 worktree scenarios (creation, already-inside, detached-head, consent)
- Both user postures (naive + spec-aware) per scenario
- Template repo + setup helpers + assertions
- Actor + verifier with prompts
- `drill run` and `drill compare` commands
- Results storage
## Phase 2 (Future)
- Gemini CLI backend
- Backend variants (e.g., `codex-workspace-write.yaml` for sandbox mode testing)
- Verifier flakiness mitigation (3x voting, agreement tracking)
- Cost tracking and token usage reporting
- Docker isolation for reproducibility
- CI integration
- Scenarios beyond worktrees (stacked PRs, git-spice, brainstorming)
## Installation
```bash
pip install -e . # installs 'drill' console script
```
Requires `tmux` installed as a system dependency.
## Dependencies
- Python 3.11+
- `click` — CLI framework
- `pyyaml` — scenario and backend config parsing
- `anthropic` — Anthropic Python SDK for actor/verifier LLM calls (structured tool_use output)
- `jinja2` — prompt template rendering
- `pydantic` — verdict schema validation (retry on malformed verifier output)
- `tmux` — session driving (system dependency)
## Non-Goals
- Not a coding ability benchmark (SWE-bench covers that)
- Not an LLM evaluation framework (promptfoo covers that)
- Not a generic terminal automation tool (Terminal-Bench covers that)
- No CI in phase 1
- No Docker in phase 1
-93
View File
@@ -1,93 +0,0 @@
# Manual Testing (Codex App)
Some scenarios cannot run automatically because drill has no harness adapter for the target — the Codex App desktop client has no CLI or tmux entry point the way `claude` and `codex` do. These scenarios are marked `manual: true` in their YAML and use a human-in-the-loop protocol.
## Protocol
Three phases. The agent never runs Codex App directly. The tester never writes a verdict by hand.
1. **Agent prepares the handoff** — reads the scenario file, renders setup + turn intents into something a human can act on, hands the package to the tester.
2. **Tester executes** — sets up the repo fixture, opens Codex App, pastes the prompt, handles any follow-ups, copies the transcript + final filesystem state back to the agent.
3. **Agent judges and records** — evaluates the transcript against `verify.criteria`, writes a verdict JSON, saves to `results/<scenario>/codex-app/YYYY-MM-DD-manual/verdict.json`.
## Phase 1: Agent prepares the handoff
Deliver as one self-contained message to the tester:
### Fixture state
Exact repo state Codex App should be launched against. Pull from `setup.notes` if present, otherwise translate `setup.helpers` + `setup.assertions` into prose. Include: which repo/directory, branch, whether to expect a worktree vs normal checkout, any required/forbidden files (e.g. `.gitignore` entries).
### Prompt to paste
Render turn 1's `intent` as a natural first-person message the tester can paste verbatim into Codex App. **Don't leak internal test language** like *"Do NOT say 'create a worktree'"* — that's instruction for the test author, not the end user. Convert it to what a real user would actually type.
Example:
> Intent: *"Ask the agent to use the worktree skill to get set up for a notifications feature. Do NOT say 'create a worktree' — just reference the skill by name."*
>
> Rendered prompt: *"hey, can you use the worktree skill to get me set up for a notifications feature?"*
### Follow-up guidance
For each additional turn, give the tester a short decision rule — not a verbatim script. E.g. *"If the agent asks a clarifying question like branch name, answer concisely. If it stops to ask whether you want a worktree at all, tell it you already asked for the skill and it should proceed."*
### What to capture
Ask the tester to paste back:
- Full agent transcript (messages, tool calls, tool outputs)
- Final filesystem state if criteria depend on it (`git worktree list`, directory tree, branch state)
- Any observations they want on the record
## Phase 2: Tester executes
1. Set up the repo fixture per the instructions
2. Open Codex App in that repo
3. Paste the prompt
4. Follow up per the guidance
5. Copy the transcript + filesystem state back to the agent
## Phase 3: Agent judges and records
For each criterion in `verify.criteria`, write one entry:
```json
{
"criterion": "<verbatim from scenario>",
"passed": true | false,
"evidence": "<quoted snippet from transcript>",
"rationale": "<only if passed is inconclusive or needs context>"
}
```
**Rules:**
- Quote the transcript directly in `evidence`. No paraphrasing.
- If a criterion is genuinely inconclusive from the transcript, mark `passed: false` with `rationale` explaining what was missing. Don't guess.
- Don't grade on intent you can't see. The agent's internal thoughts aren't visible — only messages, tool calls, and results.
### Verdict file
Save to `results/<scenario>/codex-app/YYYY-MM-DD-manual/verdict.json`:
```json
{
"scenario": "<scenario-name>",
"backend": "codex-app",
"manual": true,
"user_posture": "<spec-aware|naive|...>",
"passed": <true iff every criterion.passed is true>,
"criteria": [ ... ],
"notes": "<optional: cross-criterion observations>"
}
```
Matches the format of the existing `results/worktree-codex-app-detached-head/codex-app/2026-04-09-manual/verdict.json`.
## When to invoke
- A scenario's YAML has `manual: true`
- The tester explicitly asks for a manual Codex App run of any scenario
- An automated test result is inconclusive and we want a human-verified cross-check
Do NOT use this procedure for scenarios drill can run itself (`claude`, `codex`, `gemini` backends) — use `drill run` instead.
## Pitfalls
- **Don't skip the fixture step.** Codex App's default environment (detached HEAD under `$CODEX_HOME/worktrees/`) is load-bearing for worktree scenarios. The same prompt gives different results in a normal checkout.
- **Don't render prompts literally.** Scenario intents are written for test authors; they often contain "Do NOT mention X" style instructions. Translate before handing to the tester.
- **Don't grade on missing evidence.** If the transcript doesn't show the agent doing something the criterion asks about, that's a fail, not a pass-by-default.
-2725
View File
File diff suppressed because it is too large Load Diff
-89
View File
@@ -1,89 +0,0 @@
# Pressure / RED phase testing in drill
## What "RED phase" means
The bash test family in superpowers/tests/ used three implicit phases
when stress-testing skill content:
* **GREEN** — current skill text. Baseline behavior under normal user
prompts. This is what most drill scenarios exercise.
* **PRESSURE** — current skill text, but the user prompt creates
conditions that make the skill's recommended path inconvenient
(urgency, an "easier" alternative already on disk, etc.). Lifted
as `worktree-creation-under-pressure.yaml`.
* **RED** — *modified* skill text where the section under test has
been removed or weakened. Used to confirm a passing GREEN/PRESSURE
result actually depended on the skill text and isn't just baseline
model behavior.
GREEN and PRESSURE both run against the current `SUPERPOWERS_ROOT`.
RED needs a *different* superpowers checkout — one with the section
under test stripped out — and runs the same scenario against that.
## The drill primitive: vary `SUPERPOWERS_ROOT`
Every backend YAML interpolates `${SUPERPOWERS_ROOT}` into its
`--plugin-dir` arg (claude.yaml line 6, gemini.yaml line 5, etc.).
That env var is the only knob you need: point drill at a different
plugin checkout and the agent under test loads a different version
of the skill.
```bash
# GREEN: current skill text
drill run worktree-creation-from-main -b claude
# RED: same scenario, against a checkout where Step 1a is deleted
SUPERPOWERS_ROOT=/path/to/superpowers-without-step-1a \
drill run worktree-creation-from-main -b claude
```
Compare verdicts. If GREEN passes and RED fails, the skill text is
load-bearing. If both pass, the model produces the right behavior
without the skill — meaning either the skill is redundant or the
test isn't probing what it claims to probe.
## Recommended workflow
1. Make a git worktree of superpowers at the commit/branch you want
to test. For RED variants, edit the skill in that worktree to
remove the section under test.
```bash
cd ~/Documents/GitHub/superpowers/superpowers
git worktree add ../superpowers-red-no-step-1a HEAD
# edit skills/using-git-worktrees/SKILL.md in the worktree
```
2. Run the same drill scenario against each variant. Use
`--n N` to get statistical signal — single runs are noisy,
especially under pressure conditions.
```bash
for variant in main red-no-step-1a; do
SUPERPOWERS_ROOT=~/Documents/GitHub/superpowers/superpowers-${variant#main}superpowers \
drill run worktree-creation-from-main -b claude --n 10
done
```
3. Compare with `drill compare`. Look for the RED variant's pass
rate dropping (skill is load-bearing) or holding (skill is
redundant or scenario isn't probing what it claims).
## When to add a new pressure scenario vs. add a turn variation
* **New scenario** when the *filesystem* setup is different (e.g.,
pre-existing `.worktrees/` for the worktree-pressure case).
Setup helpers are scenario-scoped.
* **New `--n` sweep with different prompts** when only the
*user prompt* shape varies (e.g., urgency, framing).
Drill doesn't yet have a way to vary turn intents within a single
scenario YAML — multi-prompt sweeps require multiple scenario files
or running the same scenario with different intents externally.
## Open follow-ups
* `--plugins=A,B,C` sweep dimension (parallel to `--models`) so a
single drill invocation can run RED + GREEN + PRESSURE variants
in one batch and `drill compare` shows them side-by-side. Not yet
implemented; tracked as drill-internal future work.
-3
View File
@@ -1,3 +0,0 @@
"""Drill: Superpowers skill compliance benchmark."""
__version__: str = "0.1.0"
-5
View File
@@ -1,5 +0,0 @@
"""Allow running drill as `python3 -m drill`."""
from drill.cli import main
main()
-81
View File
@@ -1,81 +0,0 @@
"""Actor LLM: simulates a user driving an agent session."""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import anthropic
from jinja2 import Template
ACTOR_TOOL: dict[str, Any] = {
"name": "terminal_action",
"description": "Send an action to the terminal session.",
"input_schema": {
"type": "object",
"properties": {
"action": {
"type": "string",
"enum": ["type", "done", "stuck", "key"],
"description": "The action to take.",
},
"text": {
"type": "string",
"description": "Text to type (only for 'type' action).",
},
"key": {
"type": "string",
"description": "Special key to send (only for 'key' action, e.g., 'ctrl-c').",
},
},
"required": ["action"],
},
}
@dataclass
class ActorAction:
action: str
text: str | None = None
key: str | None = None
@classmethod
def from_tool_result(cls, data: dict[str, Any]) -> ActorAction:
return cls(action=data["action"], text=data.get("text"), key=data.get("key"))
class Actor:
def __init__(self, model: str = "claude-sonnet-4-6", temperature: float = 0.7) -> None:
self.model = model
self.temperature = temperature
self.captures: list[str] = []
self._system_prompt: str = ""
self._client: anthropic.Anthropic = anthropic.Anthropic()
def build_system_prompt(self, posture: str, intents: list[str]) -> str:
template_path = Path(__file__).parent.parent / "prompts" / "actor.md"
template = Template(template_path.read_text())
self._system_prompt = template.render(posture=posture, intents=intents)
return self._system_prompt
def append_capture(self, terminal_output: str) -> None:
self.captures.append(terminal_output)
def build_messages(self) -> list[dict[str, str]]:
return [{"role": "user", "content": capture} for capture in self.captures]
def decide(self) -> ActorAction:
response = self._client.messages.create(
model=self.model,
max_tokens=1024,
temperature=self.temperature,
system=self._system_prompt,
tools=[ACTOR_TOOL], # ty: ignore[invalid-argument-type]
tool_choice={"type": "tool", "name": "terminal_action"},
messages=self.build_messages(), # ty: ignore[invalid-argument-type]
)
for block in response.content:
if block.type == "tool_use":
return ActorAction.from_tool_result(block.input)
raise RuntimeError("Actor did not return a tool_use block")
-89
View File
@@ -1,89 +0,0 @@
"""Post-session deterministic assertions for drill scenarios."""
from __future__ import annotations
import os
import subprocess
from dataclasses import dataclass
from pathlib import Path
from drill.verifier import CriterionResult
@dataclass
class AssertionResult:
command: str
passed: bool
exit_code: int
stdout: str
stderr: str
def to_criterion_result(self) -> CriterionResult:
evidence = f"exit code {self.exit_code}"
if self.stdout:
evidence += f"\nstdout: {self.stdout}"
if self.stderr:
evidence += f"\nstderr: {self.stderr}"
return CriterionResult(
criterion=f"[assertion] {self.command}",
verdict="pass" if self.passed else "fail",
evidence=evidence,
rationale="Deterministic assertion " + ("passed" if self.passed else "failed"),
source="assertion",
)
def run_verify_assertions(
assertions: list[str],
results_dir: Path,
workdir: Path,
*,
timeout_seconds: int = 10,
) -> list[AssertionResult]:
bin_dir = Path(__file__).parent.parent / "bin"
env = {
**os.environ,
"DRILL_WORKDIR": str(workdir),
"PATH": f"{bin_dir}:{os.environ.get('PATH', '')}",
}
results: list[AssertionResult] = []
for cmd in assertions:
try:
proc = subprocess.run(
["bash", "-c", cmd],
cwd=results_dir,
capture_output=True,
text=True,
env=env,
timeout=timeout_seconds,
)
results.append(
AssertionResult(
command=cmd,
passed=proc.returncode == 0,
exit_code=proc.returncode,
stdout=proc.stdout.strip(),
stderr=proc.stderr.strip(),
)
)
except subprocess.TimeoutExpired:
results.append(
AssertionResult(
command=cmd,
passed=False,
exit_code=124,
stdout="",
stderr=f"Timed out after {timeout_seconds}s",
)
)
except Exception as e:
results.append(
AssertionResult(
command=cmd,
passed=False,
exit_code=-1,
stdout="",
stderr=str(e),
)
)
return results
-111
View File
@@ -1,111 +0,0 @@
"""Backend config loader and command builder."""
from __future__ import annotations
import os
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import yaml
@dataclass
class Backend:
name: str
cli: str
args: list[str]
required_env: list[str]
hooks: dict[str, list[str]]
shutdown: str
idle: dict[str, Any]
startup_timeout: int
terminal: dict[str, int]
session_logs: dict[str, str]
turn_timeout: int | None = None
busy_pattern: str = ""
max_busy_seconds: int = 1800
def build_command(self, workdir: str) -> list[str]:
resolved = [_interpolate_env(arg) for arg in self.args]
return [self.cli, *resolved]
def validate_env(self) -> None:
missing = [v for v in self.required_env if not os.environ.get(v)]
if missing:
raise OSError(
f"Missing required environment variables for {self.name} backend: "
+ ", ".join(missing)
)
def is_ready_line(self, line: str) -> bool:
pattern = self.idle.get("ready_pattern", "")
return bool(re.search(pattern, line))
def is_busy_line(self, line: str) -> bool:
if not self.busy_pattern:
return False
return bool(re.search(self.busy_pattern, line))
@property
def quiescence_seconds(self) -> float:
return self.idle.get("quiescence_seconds", 5)
@property
def cols(self) -> int:
return self.terminal.get("cols", 200)
@property
def rows(self) -> int:
return self.terminal.get("rows", 50)
@property
def model(self) -> str | None:
"""Model name from args (looks for --model or -m flag)."""
for i, arg in enumerate(self.args):
if arg in ("--model", "-m") and i + 1 < len(self.args):
return self.args[i + 1]
return None
@property
def family(self) -> str:
"""Normalize backend name to a family for log-dir / normalizer dispatch."""
for fam in ("claude", "codex", "gemini"):
if self.name == fam or self.name.startswith(f"{fam}-"):
return fam
return "other"
def load_backend(name: str, backends_dir: Path) -> Backend:
path = backends_dir / f"{name}.yaml"
if not path.exists():
raise FileNotFoundError(f"Backend config not found: {path}")
with open(path) as f:
data = yaml.safe_load(f)
return Backend(
name=data["name"],
cli=data["cli"],
args=data.get("args", []),
required_env=data.get("required_env", []),
hooks=data.get("hooks", {"pre_run": [], "post_run": []}),
shutdown=data.get("shutdown", "/exit"),
idle=data.get("idle", {}),
startup_timeout=data.get("startup_timeout", 30),
terminal=data.get("terminal", {"cols": 200, "rows": 50}),
session_logs=data.get("session_logs", {}),
turn_timeout=data.get("turn_timeout"),
busy_pattern=data.get("busy_pattern", ""),
max_busy_seconds=data.get("max_busy_seconds", 1800),
)
def _interpolate_env(value: str) -> str:
def replacer(match: re.Match[str]) -> str:
var = match.group(1)
val = os.environ.get(var)
if val is None:
raise OSError(f"Environment variable {var} not set")
return val
return re.sub(r"\$\{(\w+)\}", replacer, value)
-154
View File
@@ -1,154 +0,0 @@
"""Drill CLI: run, compare, list."""
from __future__ import annotations
import os
import secrets
from pathlib import Path
import click
from dotenv import load_dotenv
PROJECT_ROOT: Path = Path(__file__).parent.parent
load_dotenv(PROJECT_ROOT / ".env")
def _set_superpowers_root_default() -> None:
"""Default SUPERPOWERS_ROOT to the parent of evals/ if not already set.
Drill historically required contributors to export SUPERPOWERS_ROOT
pointing at the superpowers checkout. After lifting drill into
superpowers/evals/, the parent of PROJECT_ROOT is always the
superpowers root, so we can supply this default automatically.
Existing SUPERPOWERS_ROOT environment values are respected as overrides.
"""
os.environ.setdefault("SUPERPOWERS_ROOT", str(PROJECT_ROOT.parent))
_set_superpowers_root_default()
@click.group()
def main() -> None:
"""Drill: Superpowers skill compliance benchmark."""
pass
@main.command()
@click.argument("scenario")
@click.option("--backend", "-b", default=None, help="Backend name (e.g., claude, codex)")
@click.option("--models", "-m", default=None, help="Comma-separated backend names for sweep")
@click.option("--n", "n_runs", type=int, default=1, help="Number of repetitions per backend")
@click.option(
"--backends-dir",
type=click.Path(exists=True, path_type=Path),
default=PROJECT_ROOT / "backends",
)
@click.option(
"--scenarios-dir",
type=click.Path(exists=True, path_type=Path),
default=PROJECT_ROOT / "scenarios",
)
@click.option(
"--fixtures-dir",
type=click.Path(exists=True, path_type=Path),
default=PROJECT_ROOT / "fixtures",
)
@click.option("--results-dir", type=click.Path(path_type=Path), default=PROJECT_ROOT / "results")
def run(
scenario: str,
backend: str | None,
models: str | None,
n_runs: int,
backends_dir: Path,
scenarios_dir: Path,
fixtures_dir: Path,
results_dir: Path,
) -> None:
"""Run a scenario against one or more backends."""
if n_runs < 1:
raise click.ClickException("--n must be at least 1")
if models:
backend_names = [b.strip() for b in models.split(",") if b.strip()]
elif backend:
backend_names = [backend]
else:
raise click.ClickException("Either --backend or --models is required")
scenario_path = scenarios_dir / f"{scenario}.yaml"
if not scenario_path.exists():
raise click.ClickException(f"Scenario not found: {scenario_path}")
sweep_id = secrets.token_hex(4)
from drill.sweep import Sweep
sweep = Sweep(
scenario_path=scenario_path,
backend_names=backend_names,
backends_dir=backends_dir,
fixtures_dir=fixtures_dir,
results_dir=results_dir,
n=n_runs,
sweep_id=sweep_id,
)
total = len(backend_names) * n_runs
click.echo(
f"Running {scenario} | backends: {', '.join(backend_names)} | "
f"n={n_runs} | total runs: {total} | sweep: {sweep_id}"
)
groups = sweep.run_all()
for group in groups:
passed = sum(1 for r in group.runs if r.status == "pass")
failed = sum(1 for r in group.runs if r.status == "fail")
errored = sum(1 for r in group.runs if r.status == "error")
click.echo(f"\n{group.backend}: {passed} passed, {failed} failed, {errored} errors")
if group.partial:
click.echo(" (interrupted — partial results)")
@main.command("list")
@click.option(
"--scenarios-dir",
type=click.Path(exists=True, path_type=Path),
default=PROJECT_ROOT / "scenarios",
)
def list_scenarios(scenarios_dir: Path) -> None:
"""List available scenarios."""
import yaml
for f in sorted(scenarios_dir.glob("*.yaml")):
with open(f) as fh:
data = yaml.safe_load(fh)
name = data.get("scenario", f.stem)
desc = data.get("description", "")
click.echo(f" {name:40s} {desc}")
@main.command()
@click.argument("scenario")
@click.option("--sweep", "sweep_id", default=None, help="Filter by sweep ID")
@click.option(
"--results-dir",
type=click.Path(exists=True, path_type=Path),
default=PROJECT_ROOT / "results",
)
def compare(scenario: str, sweep_id: str | None, results_dir: Path) -> None:
"""Compare results across backends for a scenario."""
from drill.compare import format_compare_output, load_scenario_results
scenario_dir = results_dir / scenario
if not scenario_dir.exists():
raise click.ClickException(f"No results found for: {scenario}")
results = load_scenario_results(scenario_dir, sweep_id=sweep_id)
if not results:
raise click.ClickException(f"No results found for: {scenario}")
click.echo(format_compare_output(scenario, results))
-255
View File
@@ -1,255 +0,0 @@
"""Compare: load and aggregate drill results across backends and runs."""
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from drill.stats import wilson_ci
from drill.verifier import Verdict
@dataclass
class BackendResult:
backend: str
total_runs: int
passed_runs: int
errored_runs: int
avg_turns: float
criterion_counts: dict[str, tuple[int, int]] # criterion -> (passed, total)
sweep_id: str | None
timestamp: str | None
partial: bool
@property
def pass_rate(self) -> float:
if self.total_runs == 0:
return 0.0
return self.passed_runs / self.total_runs
def load_scenario_results(
scenario_dir: Path,
*,
sweep_id: str | None = None,
) -> dict[str, BackendResult]:
results: dict[str, BackendResult] = {}
for backend_dir in sorted(scenario_dir.iterdir()):
if not backend_dir.is_dir():
continue
timestamp_dirs = sorted(backend_dir.iterdir())
if not timestamp_dirs:
continue
target_dir: Path | None = None
if sweep_id:
for d in timestamp_dirs:
rg_path = d / "run-group.json"
if rg_path.exists():
rg = json.loads(rg_path.read_text())
if rg.get("sweep_id") == sweep_id:
target_dir = d
break
else:
target_dir = timestamp_dirs[-1]
if target_dir is None:
continue
result = _load_backend_result(backend_dir.name, target_dir)
if result is not None:
results[backend_dir.name] = result
return results
def _load_backend_result(backend_name: str, timestamp_dir: Path) -> BackendResult | None:
rg_path = timestamp_dir / "run-group.json"
if rg_path.exists():
return _load_new_format(backend_name, timestamp_dir, rg_path)
elif (timestamp_dir / "verdict.json").exists():
return _load_old_format(backend_name, timestamp_dir)
return None
def _load_new_format(backend_name: str, timestamp_dir: Path, rg_path: Path) -> BackendResult:
rg: dict[str, Any] = json.loads(rg_path.read_text())
run_dirs = sorted(
d for d in timestamp_dir.iterdir() if d.is_dir() and d.name.startswith("run-")
)
verdicts: list[Verdict] = []
metas: list[dict[str, Any]] = []
for run_dir in run_dirs:
verdict_path = run_dir / "verdict.json"
meta_path = run_dir / "meta.json"
if verdict_path.exists():
verdicts.append(Verdict.model_validate_json(verdict_path.read_text()))
if meta_path.exists():
metas.append(json.loads(meta_path.read_text()))
passed_runs = sum(1 for v in verdicts if v.passed)
errored_runs = sum(1 for r in rg.get("runs", []) if r.get("status") == "error")
avg_turns = sum(m.get("actor_turns", 0) for m in metas) / len(metas) if metas else 0.0
criterion_counts: dict[str, tuple[int, int]] = {}
for v in verdicts:
for c in v.criteria:
prev_passed, prev_total = criterion_counts.get(c.criterion, (0, 0))
criterion_counts[c.criterion] = (
prev_passed + (1 if c.verdict == "pass" else 0),
prev_total + 1,
)
return BackendResult(
backend=backend_name,
total_runs=len(verdicts),
passed_runs=passed_runs,
errored_runs=errored_runs,
avg_turns=round(avg_turns, 1),
criterion_counts=criterion_counts,
sweep_id=rg.get("sweep_id"),
timestamp=rg.get("timestamp"),
partial=rg.get("partial", False),
)
def _load_old_format(backend_name: str, timestamp_dir: Path) -> BackendResult:
verdict = Verdict.model_validate_json((timestamp_dir / "verdict.json").read_text())
meta: dict[str, Any] = {}
meta_path = timestamp_dir / "meta.json"
if meta_path.exists():
meta = json.loads(meta_path.read_text())
criterion_counts: dict[str, tuple[int, int]] = {}
for c in verdict.criteria:
criterion_counts[c.criterion] = (1 if c.verdict == "pass" else 0, 1)
return BackendResult(
backend=backend_name,
total_runs=1,
passed_runs=1 if verdict.passed else 0,
errored_runs=0,
avg_turns=float(meta.get("actor_turns", 0)),
criterion_counts=criterion_counts,
sweep_id=None,
timestamp=None,
partial=False,
)
def format_compare_output(
scenario: str,
results: dict[str, BackendResult],
) -> str:
if not results:
return f"No results found for: {scenario}"
lines: list[str] = []
is_multi_run = any(r.total_runs > 1 for r in results.values())
if is_multi_run:
first = next(iter(results.values()))
lines.append(f"Scenario: {scenario}")
if first.sweep_id:
sweep_label = f"Sweep: {first.sweep_id}"
if first.timestamp:
date_str = first.timestamp.split("T")[0]
sweep_label += f" | {date_str}"
lines.append(sweep_label)
lines.append("")
header = f"{'':40s}"
sub_header = f"{'':40s}"
for name, r in results.items():
header += f" {name:>12s}"
sub_header += f" {'(n=' + str(r.total_runs) + ')':>12s}"
lines.append(header)
lines.append(sub_header)
lines.append("-" * len(header))
rate_line = f"{'Overall pass rate':40s}"
ci_line = f"{' 95% CI':40s}"
for r in results.values():
pct = f"{r.pass_rate * 100:.1f}%"
rate_line += f" {pct:>12s}"
lo, hi = wilson_ci(r.passed_runs, r.total_runs)
ci_str = f"[{lo * 100:.0f}, {hi * 100:.0f}]"
ci_line += f" {ci_str:>12s}"
lines.append(rate_line)
lines.append(ci_line)
lines.append("")
all_criteria: list[str] = []
seen: set[str] = set()
for r in results.values():
for crit in r.criterion_counts:
if crit not in seen:
all_criteria.append(crit)
seen.add(crit)
for crit in all_criteria:
crit_line = f"{crit[:40]:40s}"
for r in results.values():
passed, total = r.criterion_counts.get(crit, (0, 0))
crit_line += f" {str(passed) + '/' + str(total):>12s}"
lines.append(crit_line)
lines.append("")
avg_line = f"{'Avg turns':40s}"
err_line = f"{'Errors':40s}"
for r in results.values():
avg_line += f" {str(r.avg_turns):>12s}"
err_line += f" {str(r.errored_runs):>12s}"
lines.append(avg_line)
lines.append(err_line)
if any(r.total_runs < 10 for r in results.values()):
lines.append("")
lines.append("Note: CI is wide due to small sample size; consider --n 10+")
if any(r.partial for r in results.values()):
lines.append("")
lines.append("Warning: Sweep was interrupted — results are incomplete.")
else:
lines.append(f"Scenario: {scenario}")
lines.append("")
lines.append(f"{'Backend':20s} {'Result':8s} {'Score':7s} {'Turns':5s}")
lines.append("-" * 42)
for name, r in results.items():
result_str = "PASS" if r.passed_runs == r.total_runs else "FAIL"
total_criteria = sum(t for _, t in r.criterion_counts.values())
passed_criteria = sum(p for p, _ in r.criterion_counts.values())
score = f"{passed_criteria}/{total_criteria}"
turns_str = (
str(int(r.avg_turns)) if r.avg_turns == int(r.avg_turns) else str(r.avg_turns)
)
lines.append(f"{name:20s} {result_str:8s} {score:7s} {turns_str:5s}")
all_criteria = []
seen = set()
for r in results.values():
for crit in r.criterion_counts:
if crit not in seen:
all_criteria.append(crit)
seen.add(crit)
lines.append("")
header = f"{'':40s}"
for name in results:
header += f" {name:>12s}"
lines.append(header)
lines.append("-" * len(header))
for crit in all_criteria:
crit_line = f"{crit[:40]:40s}"
for r in results.values():
p, t = r.criterion_counts.get(crit, (0, 0))
icon = "PASS" if p == t and t > 0 else "FAIL"
crit_line += f" {icon:>12s}"
lines.append(crit_line)
return "\n".join(lines)
-377
View File
@@ -1,377 +0,0 @@
"""Engine: orchestrates the full Drill run lifecycle."""
from __future__ import annotations
import json
import os
import re
import subprocess
import time
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path
from typing import Any
import yaml
from drill.actor import Actor
from drill.assertions import AssertionResult, run_verify_assertions
from drill.backend import load_backend
from drill.normalizer import (
NORMALIZERS,
collect_new_logs,
filter_codex_logs_by_cwd,
snapshot_log_dir,
)
from drill.session import TmuxSession
from drill.setup import run_assertions, run_helpers
from drill.verifier import Verifier
@dataclass
class VerifyConfig:
criteria: list[str] = field(default_factory=list)
assertions: list[str] = field(default_factory=list)
observe: bool = False
@dataclass
class ScenarioConfig:
scenario: str
description: str
user_posture: str
setup: dict[str, Any]
turns: list[dict[str, Any]]
limits: dict[str, Any]
verify: VerifyConfig
@classmethod
def from_yaml(cls, path: Path) -> ScenarioConfig:
with open(path) as f:
data = yaml.safe_load(f)
verify_data = data.get("verify", {})
return cls(
scenario=data["scenario"],
description=data.get("description", ""),
user_posture=data.get("user_posture", "naive"),
setup=data.get("setup", {}),
turns=data.get("turns", []),
limits=data.get("limits", {"max_turns": 20, "turn_timeout": 120}),
verify=VerifyConfig(
criteria=verify_data.get("criteria", []),
assertions=verify_data.get("assertions", []),
observe=verify_data.get("observe", False),
),
)
@dataclass
class RunResult:
scenario: str
backend: str
timestamp: str
session_log: str
filesystem_json: str
tool_calls_jsonl: str
verdict_json: str
meta: dict[str, Any]
def save_artifacts(self, output_dir: Path) -> None:
output_dir.mkdir(parents=True, exist_ok=True)
(output_dir / "session.log").write_text(self.session_log)
(output_dir / "filesystem.json").write_text(self.filesystem_json)
(output_dir / "tool_calls.jsonl").write_text(self.tool_calls_jsonl)
def save_verdict(self, output_dir: Path) -> None:
output_dir.mkdir(parents=True, exist_ok=True)
(output_dir / "verdict.json").write_text(self.verdict_json)
(output_dir / "meta.json").write_text(json.dumps(self.meta, indent=2))
def save(self, output_dir: Path) -> None:
self.save_artifacts(output_dir)
self.save_verdict(output_dir)
def snapshot_filesystem(workdir: Path) -> str:
files: list[str] = []
for f in sorted(workdir.rglob("*")):
if ".git" in f.parts:
continue
if f.is_file():
files.append(str(f.relative_to(workdir)))
git_status = _git_cmd(workdir, ["git", "status", "--short"])
branch = _git_cmd(workdir, ["git", "branch", "--show-current"])
worktree_list = _git_cmd(workdir, ["git", "worktree", "list"])
return json.dumps(
{
"files": files,
"git_status": git_status,
"branch": branch,
"worktree_list": worktree_list,
},
indent=2,
)
class Engine:
def __init__(
self,
scenario_path: Path,
backend_name: str,
backends_dir: Path,
fixtures_dir: Path,
results_dir: Path,
) -> None:
self.scenario = ScenarioConfig.from_yaml(scenario_path)
self.backend = load_backend(backend_name, backends_dir)
self.fixtures_dir = fixtures_dir
self.results_dir = results_dir
def run(self, *, output_dir: Path | None = None, run_suffix: str = "") -> RunResult:
start_time = time.time()
timestamp = datetime.now().strftime("%Y-%m-%dT%H-%M-%S")
self.backend.validate_env()
workdir = Path(f"/tmp/drill-{self.scenario.scenario}-{timestamp}{run_suffix}")
self._setup(workdir)
actual_workdir = workdir
override = self.scenario.setup.get("workdir_override")
if override:
resolved = override.replace("${WORKDIR_NAME}", workdir.name)
actual_workdir = (workdir / resolved).resolve()
# Run assertions in the actual workdir (after override)
assertions = self.scenario.setup.get("assertions", [])
if assertions:
run_assertions(assertions, actual_workdir)
session_name = f"drill-{self.scenario.scenario}-{timestamp}{run_suffix}"
session = TmuxSession(name=session_name, cols=self.backend.cols, rows=self.backend.rows)
log_dir = self._resolve_log_dir(actual_workdir)
log_snapshot = snapshot_log_dir(log_dir) if log_dir else set()
session_log, actor_turns = self._run_session(session, actual_workdir)
filesystem_json = snapshot_filesystem(actual_workdir)
tool_calls = self._collect_tool_calls(log_dir, log_snapshot, actual_workdir)
tool_calls_jsonl = "\n".join(json.dumps(tc) for tc in tool_calls)
# Write artifacts to disk before assertions (assertions read from disk)
if output_dir is None:
output_dir = self.results_dir / self.scenario.scenario / self.backend.name / timestamp
output_dir.mkdir(parents=True, exist_ok=True)
(output_dir / "session.log").write_text(session_log)
(output_dir / "filesystem.json").write_text(filesystem_json)
(output_dir / "tool_calls.jsonl").write_text(tool_calls_jsonl)
# Run deterministic assertions
assertion_results: list[AssertionResult] = []
if self.scenario.verify.assertions:
if not tool_calls_jsonl.strip():
assertion_results = [
AssertionResult(
command="<pre-check>",
passed=False,
exit_code=1,
stdout="",
stderr="tool_calls.jsonl is empty — session may have crashed",
)
]
else:
assertion_results = run_verify_assertions(
self.scenario.verify.assertions,
output_dir,
actual_workdir,
)
# Run LLM verifier
verifier = Verifier()
verdict = verifier.verify(
session_log=session_log,
filesystem_json=filesystem_json,
tool_calls_jsonl=tool_calls_jsonl,
criteria=self.scenario.verify.criteria,
)
# Merge assertion results into verdict
for ar in assertion_results:
verdict.criteria.append(ar.to_criterion_result())
duration = time.time() - start_time
meta: dict[str, Any] = {
"scenario": self.scenario.scenario,
"backend": self.backend.name,
"backend_model": self.backend.model,
"user_posture": self.scenario.user_posture,
"timestamp": timestamp,
"duration_seconds": round(duration, 1),
"actor_turns": actor_turns,
"actor_model": "claude-sonnet-4-6",
"verifier_model": "claude-sonnet-4-6",
}
result = RunResult(
scenario=self.scenario.scenario,
backend=self.backend.name,
timestamp=timestamp,
session_log=session_log,
filesystem_json=filesystem_json,
tool_calls_jsonl=tool_calls_jsonl,
verdict_json=verdict.model_dump_json(indent=2),
meta=meta,
)
# Write verdict + meta (artifacts already on disk)
(output_dir / "verdict.json").write_text(result.verdict_json)
(output_dir / "meta.json").write_text(json.dumps(result.meta, indent=2))
return result
def _setup(self, workdir: Path) -> None:
# Scenario helpers first (create_base_repo needs to run before anything else)
helpers = self.scenario.setup.get("helpers", [])
run_helpers(helpers, workdir, self.fixtures_dir)
# Backend pre_run hooks after (e.g., codex symlink needs workdir to exist)
hooks_needing_superpowers_root = {"symlink_superpowers", "link_gemini_extension"}
for hook_name in self.backend.hooks.get("pre_run", []):
from setup_helpers import HELPER_REGISTRY
hook = HELPER_REGISTRY.get(hook_name)
if hook and hook_name in hooks_needing_superpowers_root:
hook(workdir, os.environ["SUPERPOWERS_ROOT"]) # ty: ignore[invalid-argument-type, too-many-positional-arguments, missing-argument]
elif hook:
hook(workdir) # ty: ignore[invalid-argument-type, missing-argument]
def _run_session(self, session: TmuxSession, workdir: Path) -> tuple[str, int]:
session.create()
try:
cmd = self.backend.build_command(str(workdir))
session.launch(cmd, str(workdir))
self._wait_for_ready(session, timeout=self.backend.startup_timeout)
actor = Actor()
intents = [t["intent"] for t in self.scenario.turns]
actor.build_system_prompt(posture=self.scenario.user_posture, intents=intents)
max_turns = self.scenario.limits.get("max_turns", 20)
turn_timeout = self.backend.turn_timeout or self.scenario.limits.get(
"turn_timeout", 120
)
all_captures: list[str] = []
turn_count = 0
for turn in range(max_turns):
self._wait_for_ready(session, timeout=turn_timeout)
capture = session.capture()
all_captures.append(f"=== Turn {turn + 1} ===\n{capture}")
actor.append_capture(f"Terminal output:\n{capture}")
action = actor.decide()
turn_count += 1
if action.action == "done" or action.action == "stuck":
break
elif action.action == "type":
session.send_keys(action.text or "")
elif action.action == "key":
session.send_special_key(action.key or "")
final_capture = session.capture()
all_captures.append(f"=== Final ===\n{final_capture}")
if self.backend.shutdown.startswith("<<KEY:"):
key = self.backend.shutdown[6:-2]
session.send_special_key(key)
else:
session.send_keys(self.backend.shutdown)
time.sleep(3)
return "\n".join(all_captures), turn_count
finally:
session.kill()
def _wait_for_ready(self, session: TmuxSession, timeout: float) -> None:
"""Wait until the agent's terminal is ready for Actor input.
Returns when the terminal is quiescent AND matches the backend's
ready pattern. If the backend's busy pattern matches (spinner
visible, "Thinking...", timer counting), the deadline is extended
by small increments up to `max_busy_seconds` total. This prevents
the Actor from interrupting long-running subagent work (multi-file
implementation, parallel dispatch, etc.).
Exits silently if the final deadline (timeout + busy extensions)
passes without reaching a ready state.
"""
quiescence = self.backend.quiescence_seconds
max_busy_extension = float(self.backend.max_busy_seconds)
start = time.time()
deadline = start + timeout
total_busy_extended = 0.0
last_output: str = ""
stable_since: float | None = None
while time.time() < deadline:
current = session.capture()
lines = current.strip().split("\n")
is_busy = any(self.backend.is_busy_line(line) for line in lines)
# If the agent is actively busy, extend the deadline so we
# don't time out mid-subagent-work. Extensions are capped at
# max_busy_seconds total across all extensions combined.
if is_busy:
remaining_budget = max_busy_extension - total_busy_extended
if remaining_budget > 0:
# Ensure we have at least 30 more seconds of headroom.
needed = 30.0 - (deadline - time.time())
if needed > 0:
grant = min(needed, remaining_budget)
deadline += grant
total_busy_extended += grant
# Strip animated elements so they don't reset the quiescence timer:
# - Time counters: "Thinking... (4m 1s)" or "(esc to cancel, 4m 1s)"
# - Braille spinner characters that rotate every frame
normalized = re.sub(r"\((?:esc to cancel, )?(?:\d+[hms]\s*)+\)", "(…)", current)
normalized = re.sub(r"[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]", "·", normalized)
if normalized != last_output:
last_output = normalized
stable_since = time.time()
elif stable_since and (time.time() - stable_since) >= quiescence:
if is_busy:
stable_since = None # Reset — agent is still working
elif any(self.backend.is_ready_line(line) for line in lines):
return
time.sleep(0.5)
def _resolve_log_dir(self, workdir: Path) -> Path | None:
"""Resolve the log directory for the given backend and workdir.
Claude Code stores logs at ~/.claude/projects/<encoded-path>/
where the path is the real workdir with / replaced by -.
Codex stores logs at ~/.codex/sessions/.
"""
if self.backend.family == "claude":
real_workdir = workdir.resolve()
encoded = str(real_workdir).replace("/", "-")
log_dir = Path.home() / ".claude" / "projects" / encoded
return log_dir
elif self.backend.family == "codex":
# Codex stores at ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl
return Path.home() / ".codex" / "sessions"
elif self.backend.family == "gemini":
# Gemini stores at ~/.gemini/tmp/<project-name>/chats/session-*.json
# Project name is the workdir basename, lowercased
project = workdir.resolve().name.lower()
return Path.home() / ".gemini" / "tmp" / project
pattern = self.backend.session_logs.get("pattern", "")
if not pattern:
return None
expanded = os.path.expanduser(pattern)
parts = expanded.split("*")[0].rstrip("/")
return Path(parts)
def _collect_tool_calls(
self, log_dir: Path | None, snapshot: set[str], workdir: Path
) -> list[dict[str, Any]]:
if log_dir is None:
return []
new_files = collect_new_logs(log_dir, snapshot)
if self.backend.family == "codex":
new_files = filter_codex_logs_by_cwd(new_files, str(workdir.resolve()))
normalizer = NORMALIZERS.get(self.backend.family)
if not normalizer:
return []
results: list[dict[str, Any]] = []
for log_file in new_files:
results.extend(normalizer(log_file.read_text()))
return results
def _git_cmd(workdir: Path, cmd: list[str]) -> str:
result = subprocess.run(cmd, cwd=workdir, capture_output=True, text=True)
return result.stdout.strip()
-228
View File
@@ -1,228 +0,0 @@
"""Normalizes backend-specific session logs to a common tool call schema."""
from __future__ import annotations
import json
from collections.abc import Callable
from pathlib import Path
from typing import Any
NATIVE_TOOLS: set[str] = {
"EnterWorktree",
"ExitWorktree",
"EnterPlanMode",
"ExitPlanMode",
"TaskCreate",
"TaskUpdate",
"TaskList",
"TaskGet",
"Skill",
"Agent",
"Read",
"Write",
"Edit",
"Glob",
"Grep",
}
LOG_EXTENSIONS: tuple[str, ...] = ("*.jsonl", "*.json")
def snapshot_log_dir(log_dir: Path) -> set[str]:
"""Snapshot all session log files in a log directory (recursive)."""
if not log_dir.exists():
return set()
files: set[str] = set()
for ext in LOG_EXTENSIONS:
files.update(str(f.relative_to(log_dir)) for f in log_dir.rglob(ext))
return files
def collect_new_logs(log_dir: Path, snapshot: set[str]) -> list[Path]:
"""Find session log files created after the snapshot (recursive)."""
if not log_dir.exists():
return []
current: dict[str, Path] = {}
for ext in LOG_EXTENSIONS:
current.update({str(f.relative_to(log_dir)): f for f in log_dir.rglob(ext)})
new_keys: set[str] = set(current.keys()) - snapshot
return [current[k] for k in sorted(new_keys)]
def filter_codex_logs_by_cwd(paths: list[Path], target_cwd: str) -> list[Path]:
"""Drop codex rollouts whose session_meta.cwd doesn't match target_cwd.
Codex stores all sessions under a shared ~/.codex/sessions/ tree, so when
multiple drill scenarios run in parallel each one's snapshot diff sees every
other run's rollouts. Each rollout's first line is a `session_meta` event
that records the cwd the codex CLI was launched in — use it to attribute
rollouts to the run that produced them.
"""
matched: list[Path] = []
for path in paths:
try:
with path.open() as f:
first_line = f.readline()
entry = json.loads(first_line)
except (OSError, json.JSONDecodeError):
continue
if entry.get("type") != "session_meta":
continue
cwd = entry.get("payload", {}).get("cwd", "")
if cwd == target_cwd:
matched.append(path)
return matched
def normalize_claude_logs(raw_content: str) -> list[dict[str, Any]]:
"""Normalize Claude Code session logs.
CC logs are JSONL where assistant messages have:
{"type": "assistant", "message": {"content": [{"type": "tool_use", "name": "...",
"input": {...}}]}}
"""
results: list[dict[str, Any]] = []
for line in raw_content.strip().split("\n"):
if not line.strip():
continue
try:
entry = json.loads(line)
except json.JSONDecodeError:
continue
# Handle nested CC format: assistant messages contain tool_use in content array
if entry.get("type") == "assistant":
message = entry.get("message", {})
for block in message.get("content", []):
if block.get("type") == "tool_use":
tool_name = block.get("name", "")
source = "native" if tool_name in NATIVE_TOOLS else "shell"
results.append(
{"tool": tool_name, "args": block.get("input", {}), "source": source}
)
# Also handle flat format (for test compatibility)
elif entry.get("type") == "tool_use":
tool_name = entry.get("name", "")
source = "native" if tool_name in NATIVE_TOOLS else "shell"
results.append({"tool": tool_name, "args": entry.get("input", {}), "source": source})
return results
def normalize_codex_logs(raw_content: str) -> list[dict[str, Any]]:
"""Normalize Codex rollout logs.
Codex logs use: {"type": "response_item", "payload": {"type": "function_call", ...}}
Tool calls are "function_call" with name "exec_command" (shell) or other names.
"""
results: list[dict[str, Any]] = []
for line in raw_content.strip().split("\n"):
if not line.strip():
continue
try:
entry = json.loads(line)
except json.JSONDecodeError:
continue
if entry.get("type") != "response_item":
continue
# Codex uses "payload" not "item"
payload = entry.get("payload", entry.get("item", {}))
payload_type = payload.get("type", "")
if payload_type == "function_call":
name = payload.get("name", "")
raw_args = payload.get("arguments", "{}")
# Arguments are JSON-encoded strings in codex
if isinstance(raw_args, str):
try:
args = json.loads(raw_args)
except json.JSONDecodeError:
args = {"raw": raw_args}
else:
args = raw_args
# exec_command is codex's shell tool
if name == "exec_command":
results.append(
{"tool": "Bash", "args": {"command": args.get("cmd", "")}, "source": "shell"}
)
elif name == "apply_patch":
results.append({"tool": "Edit", "args": args, "source": "native"})
else:
source = "native" if name in NATIVE_TOOLS else "shell"
results.append({"tool": name, "args": args, "source": source})
elif payload_type == "local_shell_call":
action = payload.get("action", {})
cmd = action.get("command", [])
cmd_str = " ".join(cmd) if isinstance(cmd, list) else str(cmd)
results.append({"tool": "Bash", "args": {"command": cmd_str}, "source": "shell"})
return results
# Reverse mapping: Gemini tool names → Claude Code canonical names
GEMINI_TOOL_MAP: dict[str, str] = {
"run_shell_command": "Bash",
"read_file": "Read",
"write_file": "Write",
"replace": "Edit",
"grep_search": "Grep",
"glob": "Glob",
"activate_skill": "Skill",
"google_web_search": "WebSearch",
"web_fetch": "WebFetch",
"write_todos": "TodoWrite",
"list_directory": "Glob",
"enter_plan_mode": "EnterPlanMode",
"exit_plan_mode": "ExitPlanMode",
}
def normalize_gemini_logs(raw_content: str) -> list[dict[str, Any]]:
"""Normalize Gemini CLI session logs.
Gemini logs may be a single JSON file with a messages array, or JSONL
session files in newer CLI versions. Each "gemini" message may have a
toolCalls array:
{"name": "run_shell_command", "args": {"command": "..."}, "status": "success"}
"""
results: list[dict[str, Any]] = []
messages: list[dict[str, Any]] = []
try:
data = json.loads(raw_content)
except json.JSONDecodeError:
for line in raw_content.strip().split("\n"):
if not line.strip():
continue
try:
entry = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(entry, dict):
messages.append(entry)
else:
if isinstance(data, dict) and "messages" in data:
messages = [m for m in data.get("messages", []) if isinstance(m, dict)]
elif isinstance(data, dict):
messages = [data]
elif isinstance(data, list):
messages = [m for m in data if isinstance(m, dict)]
seen_tool_calls: set[str] = set()
for message in messages:
if message.get("type") != "gemini":
continue
for tc in message.get("toolCalls", []):
tool_call_id = tc.get("id")
if tool_call_id and tool_call_id in seen_tool_calls:
continue
if tool_call_id:
seen_tool_calls.add(tool_call_id)
gemini_name = tc.get("name", "")
canonical = GEMINI_TOOL_MAP.get(gemini_name, gemini_name)
args = tc.get("args", {})
source = "native" if canonical in NATIVE_TOOLS else "shell"
results.append({"tool": canonical, "args": args, "source": source})
return results
NORMALIZERS: dict[str, Callable[[str], list[dict[str, Any]]]] = {
"claude": normalize_claude_logs,
"codex": normalize_codex_logs,
"gemini": normalize_gemini_logs,
}
-88
View File
@@ -1,88 +0,0 @@
"""tmux session management for driving agent CLI sessions."""
from __future__ import annotations
import subprocess
import time
class TmuxSession:
def __init__(self, name: str, cols: int = 200, rows: int = 50) -> None:
self.name = name
self.cols = cols
self.rows = rows
def create(self) -> None:
subprocess.run(
[
"tmux",
"new-session",
"-d",
"-s",
self.name,
"-x",
str(self.cols),
"-y",
str(self.rows),
],
check=True,
)
def launch(self, command: list[str], cwd: str) -> None:
cmd_str = " ".join(command)
self.send_keys(f"cd {cwd} && {cmd_str}")
def send_keys(self, text: str) -> None:
if text:
buffer_name = f"{self.name}-input"
subprocess.run(
["tmux", "set-buffer", "-b", buffer_name, text],
check=True,
)
subprocess.run(
["tmux", "paste-buffer", "-d", "-b", buffer_name, "-t", self.name],
check=True,
)
time.sleep(0.1)
subprocess.run(
["tmux", "send-keys", "-t", self.name, "Enter"],
check=True,
)
def send_special_key(self, key: str) -> None:
key_map = {
"ctrl-c": "C-c",
"ctrl-d": "C-d",
"ctrl-z": "C-z",
"enter": "Enter",
"escape": "Escape",
}
tmux_key = key_map.get(key, key)
subprocess.run(
["tmux", "send-keys", "-t", self.name, tmux_key],
check=True,
)
def capture(self) -> str:
result = subprocess.run(
["tmux", "capture-pane", "-t", self.name, "-p"],
capture_output=True,
text=True,
check=True,
)
return result.stdout
def is_process_alive(self) -> bool:
result = subprocess.run(
["tmux", "list-panes", "-t", self.name, "-F", "#{pane_dead}"],
capture_output=True,
text=True,
)
return result.stdout.strip() == "0"
def kill(self) -> None:
subprocess.run(
["tmux", "kill-session", "-t", self.name],
capture_output=True,
)
-43
View File
@@ -1,43 +0,0 @@
from __future__ import annotations
import subprocess
from pathlib import Path
from setup_helpers import HELPER_REGISTRY
from setup_helpers.base import create_base_repo
def clone_template(template_dir: Path, workdir: Path) -> None:
"""Clone (or build) template_dir into workdir with full git history."""
create_base_repo(workdir, template_dir)
def run_helpers(helper_names: list[str], workdir: Path, fixtures_dir: Path) -> None:
for name in helper_names:
helper = HELPER_REGISTRY.get(name)
if helper is None:
raise ValueError(f"Unknown setup helper: {name}")
if name == "create_base_repo":
helper(workdir, fixtures_dir / "template-repo") # ty: ignore[invalid-argument-type, too-many-positional-arguments, missing-argument]
elif name == "symlink_superpowers":
import os
helper(workdir, os.environ["SUPERPOWERS_ROOT"]) # ty: ignore[invalid-argument-type, too-many-positional-arguments, missing-argument]
else:
helper(workdir) # ty: ignore[invalid-argument-type, missing-argument]
def run_assertions(assertions: list[str], workdir: Path) -> None:
for assertion in assertions:
result = subprocess.run(
assertion,
shell=True,
cwd=workdir,
capture_output=True,
text=True,
)
if result.returncode != 0:
raise AssertionError(
f"Setup assertion failed: {assertion}\n"
f"stdout: {result.stdout}\nstderr: {result.stderr}"
)
-17
View File
@@ -1,17 +0,0 @@
"""Statistical utilities for drill result analysis."""
from __future__ import annotations
import math
def wilson_ci(passed: int, total: int, z: float = 1.96) -> tuple[float, float]:
if total == 0:
return (0.0, 0.0)
if passed > total:
passed = total
p = passed / total
denom = 1 + z**2 / total
center = (p + z**2 / (2 * total)) / denom
margin = (z / denom) * math.sqrt(p * (1 - p) / total + z**2 / (4 * total**2))
return (max(0.0, center - margin), min(1.0, center + margin))
-159
View File
@@ -1,159 +0,0 @@
"""Sweep orchestrator: runs scenarios N times across multiple backends."""
from __future__ import annotations
import glob as glob_mod
import json
import shutil
import time
from dataclasses import asdict, dataclass, field
from datetime import datetime
from pathlib import Path
from typing import Any
import yaml
from drill.engine import Engine, RunResult
from drill.verifier import Verdict
@dataclass
class RunStatus:
index: int
status: str # "pass", "fail", "error"
duration: float
error: str | None = None
@dataclass
class RunGroup:
scenario: str
backend: str
n: int
timestamp: str
sweep_id: str
runs: list[RunStatus] = field(default_factory=list)
partial: bool = False
def write_run_group(group: RunGroup, output_dir: Path) -> None:
output_dir.mkdir(parents=True, exist_ok=True)
data: dict[str, Any] = {
"scenario": group.scenario,
"backend": group.backend,
"n": group.n,
"timestamp": group.timestamp,
"sweep_id": group.sweep_id,
"partial": group.partial,
"runs": [
{k: v for k, v in asdict(r).items() if k != "error" or v is not None}
for r in group.runs
],
}
(output_dir / "run-group.json").write_text(json.dumps(data, indent=2))
class Sweep:
def __init__(
self,
scenario_path: Path,
backend_names: list[str],
backends_dir: Path,
fixtures_dir: Path,
results_dir: Path,
n: int,
sweep_id: str,
) -> None:
self.scenario_path = scenario_path
self.backend_names = backend_names
self.backends_dir = backends_dir
self.fixtures_dir = fixtures_dir
self.results_dir = results_dir
self.n = n
self.sweep_id = sweep_id
self._scenario_name_cache: str | None = None
def validate_backends(self) -> None:
for name in self.backend_names:
path = self.backends_dir / f"{name}.yaml"
if not path.exists():
raise FileNotFoundError(f"Backend config not found: {path}")
def run_all(self) -> list[RunGroup]:
self.validate_backends()
groups: list[RunGroup] = []
for backend_name in self.backend_names:
group = self._run_backend(backend_name)
groups.append(group)
return groups
def _run_backend(self, backend_name: str) -> RunGroup:
timestamp = datetime.now().strftime("%Y-%m-%dT%H-%M-%S")
group_dir = (
self.results_dir / self.scenario_name / backend_name / f"{timestamp}-{self.sweep_id}"
)
group_dir.mkdir(parents=True, exist_ok=True)
group = RunGroup(
scenario=self.scenario_name,
backend=backend_name,
n=self.n,
timestamp=timestamp,
sweep_id=self.sweep_id,
)
try:
for i in range(self.n):
run_status = self._run_single(backend_name, group_dir, i, timestamp)
group.runs.append(run_status)
except KeyboardInterrupt:
group.partial = True
finally:
write_run_group(group, group_dir)
return group
def _run_single(
self, backend_name: str, group_dir: Path, index: int, timestamp: str
) -> RunStatus:
run_suffix = f"-run-{index:02d}"
run_dir = group_dir / f"run-{index:02d}"
start = time.time()
try:
engine = Engine(
scenario_path=self.scenario_path,
backend_name=backend_name,
backends_dir=self.backends_dir,
fixtures_dir=self.fixtures_dir,
results_dir=self.results_dir,
)
result: RunResult = engine.run(output_dir=run_dir, run_suffix=run_suffix)
verdict = Verdict.model_validate_json(result.verdict_json)
duration = time.time() - start
status = "pass" if verdict.passed else "fail"
return RunStatus(index=index, status=status, duration=round(duration, 1))
except KeyboardInterrupt:
raise
except Exception as e:
duration = time.time() - start
return RunStatus(
index=index,
status="error",
duration=round(duration, 1),
error=str(e),
)
finally:
pattern = f"/tmp/drill-*-{timestamp}{run_suffix}"
for d in glob_mod.glob(pattern):
p = Path(d)
if p.is_dir():
shutil.rmtree(p, ignore_errors=True)
@property
def scenario_name(self) -> str:
if self._scenario_name_cache is None:
with open(self.scenario_path) as f:
data = yaml.safe_load(f)
self._scenario_name_cache = data["scenario"]
return self._scenario_name_cache
-93
View File
@@ -1,93 +0,0 @@
"""Verifier LLM: evaluates agent session against criteria."""
from __future__ import annotations
from pathlib import Path
import anthropic
from pydantic import BaseModel
class CriterionResult(BaseModel):
criterion: str
verdict: str
evidence: str
rationale: str
source: str = "judge"
class Verdict(BaseModel):
criteria: list[CriterionResult]
observations: list[str]
summary: str
@property
def score(self) -> str:
passed = sum(1 for c in self.criteria if c.verdict == "pass")
return f"{passed}/{len(self.criteria)}"
@property
def passed(self) -> bool:
return all(c.verdict == "pass" for c in self.criteria)
class Verifier:
MAX_RETRIES = 3
def __init__(self, model: str = "claude-sonnet-4-6", temperature: float = 0.0) -> None:
self.model = model
self.temperature = temperature
self._client: anthropic.Anthropic = anthropic.Anthropic()
def build_system_prompt(self) -> str:
template_path = Path(__file__).parent.parent / "prompts" / "verifier.md"
return template_path.read_text()
def verify(
self,
session_log: str,
filesystem_json: str,
tool_calls_jsonl: str,
criteria: list[str],
) -> Verdict:
system = self.build_system_prompt()
user_content = (
"## Terminal Session Log\n\n"
f"```\n{session_log}\n```\n\n"
"## Filesystem State\n\n"
f"```json\n{filesystem_json}\n```\n\n"
"## Tool Call Log\n\n"
f"```jsonl\n{tool_calls_jsonl}\n```\n\n"
"## Criteria to Evaluate\n\n" + "\n".join(f"- {c}" for c in criteria)
)
for attempt in range(self.MAX_RETRIES):
response = self._client.messages.create(
model=self.model,
max_tokens=4096,
temperature=self.temperature,
system=system,
messages=[{"role": "user", "content": user_content}],
)
text = response.content[0].text # ty: ignore[unresolved-attribute]
json_str = _extract_json(text)
try:
return Verdict.model_validate_json(json_str)
except Exception:
if attempt == self.MAX_RETRIES - 1:
raise
continue
raise RuntimeError("Verifier failed to return valid JSON")
def _extract_json(text: str) -> str:
if "```json" in text:
start = text.index("```json") + 7
end = text.index("```", start)
return text[start:end].strip()
if "```" in text:
start = text.index("```") + 3
end = text.index("```", start)
return text[start:end].strip()
start = text.index("{")
end = text.rindex("}") + 1
return text[start:end]
-81
View File
@@ -1,81 +0,0 @@
# Go Fractals CLI - Design
## Overview
A command-line tool that generates ASCII art fractals. Supports two fractal types with configurable output.
## Usage
```bash
# Sierpinski triangle
fractals sierpinski --size 32 --depth 5
# Mandelbrot set
fractals mandelbrot --width 80 --height 24 --iterations 100
# Custom character
fractals sierpinski --size 16 --char '#'
# Help
fractals --help
fractals sierpinski --help
```
## Commands
### `sierpinski`
Generates a Sierpinski triangle using recursive subdivision.
Flags:
- `--size` (default: 32) - Width of the triangle base in characters
- `--depth` (default: 5) - Recursion depth
- `--char` (default: '*') - Character to use for filled points
Output: Triangle printed to stdout, one line per row.
### `mandelbrot`
Renders the Mandelbrot set as ASCII art. Maps iteration count to characters.
Flags:
- `--width` (default: 80) - Output width in characters
- `--height` (default: 24) - Output height in characters
- `--iterations` (default: 100) - Maximum iterations for escape calculation
- `--char` (default: gradient) - Single character, or omit for gradient " .:-=+*#%@"
Output: Rectangle printed to stdout.
## Architecture
```
cmd/
fractals/
main.go # Entry point, CLI setup
internal/
sierpinski/
sierpinski.go # Algorithm
sierpinski_test.go
mandelbrot/
mandelbrot.go # Algorithm
mandelbrot_test.go
cli/
root.go # Root command, help
sierpinski.go # Sierpinski subcommand
mandelbrot.go # Mandelbrot subcommand
```
## Dependencies
- Go 1.21+
- `github.com/spf13/cobra` for CLI
## Acceptance Criteria
1. `fractals --help` shows usage
2. `fractals sierpinski` outputs a recognizable triangle
3. `fractals mandelbrot` outputs a recognizable Mandelbrot set
4. `--size`, `--width`, `--height`, `--depth`, `--iterations` flags work
5. `--char` customizes output character
6. Invalid inputs produce clear error messages
7. All tests pass
-172
View File
@@ -1,172 +0,0 @@
# Go Fractals CLI - Implementation Plan
Execute this plan using the `superpowers:subagent-driven-development` skill.
## Context
Building a CLI tool that generates ASCII fractals. See `design.md` for full specification.
## Tasks
### Task 1: Project Setup
Create the Go module and directory structure.
**Do:**
- Initialize `go.mod` with module name `github.com/superpowers-test/fractals`
- Create directory structure: `cmd/fractals/`, `internal/sierpinski/`, `internal/mandelbrot/`, `internal/cli/`
- Create minimal `cmd/fractals/main.go` that prints "fractals cli"
- Add `github.com/spf13/cobra` dependency
**Verify:**
- `go build ./cmd/fractals` succeeds
- `./fractals` prints "fractals cli"
---
### Task 2: CLI Framework with Help
Set up Cobra root command with help output.
**Do:**
- Create `internal/cli/root.go` with root command
- Configure help text showing available subcommands
- Wire root command into `main.go`
**Verify:**
- `./fractals --help` shows usage with "sierpinski" and "mandelbrot" listed as available commands
- `./fractals` (no args) shows help
---
### Task 3: Sierpinski Algorithm
Implement the Sierpinski triangle generation algorithm.
**Do:**
- Create `internal/sierpinski/sierpinski.go`
- Implement `Generate(size, depth int, char rune) []string` that returns lines of the triangle
- Use recursive midpoint subdivision algorithm
- Create `internal/sierpinski/sierpinski_test.go` with tests:
- Small triangle (size=4, depth=2) matches expected output
- Size=1 returns single character
- Depth=0 returns filled triangle
**Verify:**
- `go test ./internal/sierpinski/...` passes
---
### Task 4: Sierpinski CLI Integration
Wire the Sierpinski algorithm to a CLI subcommand.
**Do:**
- Create `internal/cli/sierpinski.go` with `sierpinski` subcommand
- Add flags: `--size` (default 32), `--depth` (default 5), `--char` (default '*')
- Call `sierpinski.Generate()` and print result to stdout
**Verify:**
- `./fractals sierpinski` outputs a triangle
- `./fractals sierpinski --size 16 --depth 3` outputs smaller triangle
- `./fractals sierpinski --help` shows flag documentation
---
### Task 5: Mandelbrot Algorithm
Implement the Mandelbrot set ASCII renderer.
**Do:**
- Create `internal/mandelbrot/mandelbrot.go`
- Implement `Render(width, height, maxIter int, char string) []string`
- Map complex plane region (-2.5 to 1.0 real, -1.0 to 1.0 imaginary) to output dimensions
- Map iteration count to character gradient " .:-=+*#%@" (or single char if provided)
- Create `internal/mandelbrot/mandelbrot_test.go` with tests:
- Output dimensions match requested width/height
- Known point inside set (0,0) maps to max-iteration character
- Known point outside set (2,0) maps to low-iteration character
**Verify:**
- `go test ./internal/mandelbrot/...` passes
---
### Task 6: Mandelbrot CLI Integration
Wire the Mandelbrot algorithm to a CLI subcommand.
**Do:**
- Create `internal/cli/mandelbrot.go` with `mandelbrot` subcommand
- Add flags: `--width` (default 80), `--height` (default 24), `--iterations` (default 100), `--char` (default "")
- Call `mandelbrot.Render()` and print result to stdout
**Verify:**
- `./fractals mandelbrot` outputs recognizable Mandelbrot set
- `./fractals mandelbrot --width 40 --height 12` outputs smaller version
- `./fractals mandelbrot --help` shows flag documentation
---
### Task 7: Character Set Configuration
Ensure `--char` flag works consistently across both commands.
**Do:**
- Verify Sierpinski `--char` flag passes character to algorithm
- For Mandelbrot, `--char` should use single character instead of gradient
- Add tests for custom character output
**Verify:**
- `./fractals sierpinski --char '#'` uses '#' character
- `./fractals mandelbrot --char '.'` uses '.' for all filled points
- Tests pass
---
### Task 8: Input Validation and Error Handling
Add validation for invalid inputs.
**Do:**
- Sierpinski: size must be > 0, depth must be >= 0
- Mandelbrot: width/height must be > 0, iterations must be > 0
- Return clear error messages for invalid inputs
- Add tests for error cases
**Verify:**
- `./fractals sierpinski --size 0` prints error, exits non-zero
- `./fractals mandelbrot --width -1` prints error, exits non-zero
- Error messages are clear and helpful
---
### Task 9: Integration Tests
Add integration tests that invoke the CLI.
**Do:**
- Create `cmd/fractals/main_test.go` or `test/integration_test.go`
- Test full CLI invocation for both commands
- Verify output format and exit codes
- Test error cases return non-zero exit
**Verify:**
- `go test ./...` passes all tests including integration tests
---
### Task 10: README
Document usage and examples.
**Do:**
- Create `README.md` with:
- Project description
- Installation: `go install ./cmd/fractals`
- Usage examples for both commands
- Example output (small samples)
**Verify:**
- README accurately describes the tool
- Examples in README actually work
-70
View File
@@ -1,70 +0,0 @@
# Svelte Todo List - Design
## Overview
A simple todo list application built with Svelte. Supports creating, completing, and deleting todos with localStorage persistence.
## Features
- Add new todos
- Mark todos as complete/incomplete
- Delete todos
- Filter by: All / Active / Completed
- Clear all completed todos
- Persist to localStorage
- Show count of remaining items
## User Interface
```
┌─────────────────────────────────────────┐
│ Svelte Todos │
├─────────────────────────────────────────┤
│ [________________________] [Add] │
├─────────────────────────────────────────┤
│ [ ] Buy groceries [x] │
│ [✓] Walk the dog [x] │
│ [ ] Write code [x] │
├─────────────────────────────────────────┤
│ 2 items left │
│ [All] [Active] [Completed] [Clear ✓] │
└─────────────────────────────────────────┘
```
## Components
```
src/
App.svelte # Main app, state management
lib/
TodoInput.svelte # Text input + Add button
TodoList.svelte # List container
TodoItem.svelte # Single todo with checkbox, text, delete
FilterBar.svelte # Filter buttons + clear completed
store.ts # Svelte store for todos
storage.ts # localStorage persistence
```
## Data Model
```typescript
interface Todo {
id: string; // UUID
text: string; // Todo text
completed: boolean;
}
type Filter = 'all' | 'active' | 'completed';
```
## Acceptance Criteria
1. Can add a todo by typing and pressing Enter or clicking Add
2. Can toggle todo completion by clicking checkbox
3. Can delete a todo by clicking X button
4. Filter buttons show correct subset of todos
5. "X items left" shows count of incomplete todos
6. "Clear completed" removes all completed todos
7. Todos persist across page refresh (localStorage)
8. Empty state shows helpful message
9. All tests pass
-222
View File
@@ -1,222 +0,0 @@
# Svelte Todo List - Implementation Plan
Execute this plan using the `superpowers:subagent-driven-development` skill.
## Context
Building a todo list app with Svelte. See `design.md` for full specification.
## Tasks
### Task 1: Project Setup
Create the Svelte project with Vite.
**Do:**
- Run `npm create vite@latest . -- --template svelte-ts`
- Install dependencies with `npm install`
- Verify dev server works
- Clean up default Vite template content from App.svelte
**Verify:**
- `npm run dev` starts server
- App shows minimal "Svelte Todos" heading
- `npm run build` succeeds
---
### Task 2: Todo Store
Create the Svelte store for todo state management.
**Do:**
- Create `src/lib/store.ts`
- Define `Todo` interface with id, text, completed
- Create writable store with initial empty array
- Export functions: `addTodo(text)`, `toggleTodo(id)`, `deleteTodo(id)`, `clearCompleted()`
- Create `src/lib/store.test.ts` with tests for each function
**Verify:**
- Tests pass: `npm run test` (install vitest if needed)
---
### Task 3: localStorage Persistence
Add persistence layer for todos.
**Do:**
- Create `src/lib/storage.ts`
- Implement `loadTodos(): Todo[]` and `saveTodos(todos: Todo[])`
- Handle JSON parse errors gracefully (return empty array)
- Integrate with store: load on init, save on change
- Add tests for load/save/error handling
**Verify:**
- Tests pass
- Manual test: add todo, refresh page, todo persists
---
### Task 4: TodoInput Component
Create the input component for adding todos.
**Do:**
- Create `src/lib/TodoInput.svelte`
- Text input bound to local state
- Add button calls `addTodo()` and clears input
- Enter key also submits
- Disable Add button when input is empty
- Add component tests
**Verify:**
- Tests pass
- Component renders input and button
---
### Task 5: TodoItem Component
Create the single todo item component.
**Do:**
- Create `src/lib/TodoItem.svelte`
- Props: `todo: Todo`
- Checkbox toggles completion (calls `toggleTodo`)
- Text with strikethrough when completed
- Delete button (X) calls `deleteTodo`
- Add component tests
**Verify:**
- Tests pass
- Component renders checkbox, text, delete button
---
### Task 6: TodoList Component
Create the list container component.
**Do:**
- Create `src/lib/TodoList.svelte`
- Props: `todos: Todo[]`
- Renders TodoItem for each todo
- Shows "No todos yet" when empty
- Add component tests
**Verify:**
- Tests pass
- Component renders list of TodoItems
---
### Task 7: FilterBar Component
Create the filter and status bar component.
**Do:**
- Create `src/lib/FilterBar.svelte`
- Props: `todos: Todo[]`, `filter: Filter`, `onFilterChange: (f: Filter) => void`
- Show count: "X items left" (incomplete count)
- Three filter buttons: All, Active, Completed
- Active filter is visually highlighted
- "Clear completed" button (hidden when no completed todos)
- Add component tests
**Verify:**
- Tests pass
- Component renders count, filters, clear button
---
### Task 8: App Integration
Wire all components together in App.svelte.
**Do:**
- Import all components and store
- Add filter state (default: 'all')
- Compute filtered todos based on filter state
- Render: heading, TodoInput, TodoList, FilterBar
- Pass appropriate props to each component
**Verify:**
- App renders all components
- Adding todos works
- Toggling works
- Deleting works
---
### Task 9: Filter Functionality
Ensure filtering works end-to-end.
**Do:**
- Verify filter buttons change displayed todos
- 'all' shows all todos
- 'active' shows only incomplete todos
- 'completed' shows only completed todos
- Clear completed removes completed todos and resets filter if needed
- Add integration tests
**Verify:**
- Filter tests pass
- Manual verification of all filter states
---
### Task 10: Styling and Polish
Add CSS styling for usability.
**Do:**
- Style the app to match the design mockup
- Completed todos have strikethrough and muted color
- Active filter button is highlighted
- Input has focus styles
- Delete button appears on hover (or always on mobile)
- Responsive layout
**Verify:**
- App is visually usable
- Styles don't break functionality
---
### Task 11: End-to-End Tests
Add Playwright tests for full user flows.
**Do:**
- Install Playwright: `npm init playwright@latest`
- Create `tests/todo.spec.ts`
- Test flows:
- Add a todo
- Complete a todo
- Delete a todo
- Filter todos
- Clear completed
- Persistence (add, reload, verify)
**Verify:**
- `npx playwright test` passes
---
### Task 12: README
Document the project.
**Do:**
- Create `README.md` with:
- Project description
- Setup: `npm install`
- Development: `npm run dev`
- Testing: `npm test` and `npx playwright test`
- Build: `npm run build`
**Verify:**
- README accurately describes the project
- Instructions work
-3
View File
@@ -1,3 +0,0 @@
# Test Project
A minimal project for Drill test scenarios.
@@ -1,6 +0,0 @@
{
"name": "drill-test-project",
"version": "1.0.0",
"description": "Test project for Drill scenarios",
"main": "src/index.js"
}
@@ -1,7 +0,0 @@
const { greet } = require('./utils');
function main() {
console.log(greet('world'));
}
main();
@@ -1,5 +0,0 @@
function greet(name) {
return `Hello, ${name}!`;
}
module.exports = { greet };
-41
View File
@@ -1,41 +0,0 @@
You are simulating a user interacting with an AI coding agent in a terminal.
{% if posture == "naive" %}
You are a developer who wants to accomplish a task. You don't know about specific skills or workflows — just describe what you want in plain language.
{% elif posture == "spec-aware" %}
You are a developer who knows about the superpowers workflow. You may reference specific skills or conventions by name (e.g., "use the worktree skill", "follow the using-git-worktrees pattern").
{% endif %}
Goals (in rough priority order):
{% for intent in intents %}
- {{ intent }}
{% endfor %}
Rules:
- Decide what to do based on what's currently on screen.
- Goals are not a script — some are conditional. Act on them when relevant.
- Type natural, concise messages like a real developer would.
- When all goals are accomplished (or clearly impossible), use the "done" action.
- If you're stuck and cannot make progress, use the "stuck" action.
- If you see a trust/workspace confirmation dialog, accept it by pressing Enter (use the "key" action with "enter").
- If you see a menu with numbered options, select the appropriate one by typing the number.
PATIENCE MODE — CRITICAL:
The agent may be actively working. Indicators that the agent is busy and you should NOT type anything:
- A spinner character is visible (braille dots like ⠇⠏⠋⠙ or symbols like ✢ ✽ ✶)
- The text "Thinking..." or "Running..." or "Working..." is visible
- A time counter is counting (e.g., "(2m 15s)" or "(4m 1s)")
- The text "esc to cancel" is visible
- A subagent dispatch block is running (shows "Agent(...)" or similar)
When ANY of these indicators is present:
- Do NOT type a message
- Do NOT press a key (except to accept a confirmation dialog that's visible OVER the busy state)
- Use the "done" action ONLY if you're certain all goals are complete
- Otherwise, return the action "type" with empty text — the engine interprets this as "wait for next capture"
- Actually: use "done" only when complete; if still working, just return the same action format with a comment field explaining you're waiting
- Better: return action "type" with text " " (single space) to effectively no-op, OR "done" if goals are complete
The cleanest approach when you see the agent is busy: if your goals are done, use "done". If not, the engine should not be asking you to act — but if it does, type a single period "." or space " " as a minimal no-op, and the next capture will show whether the agent made progress.
Long-running operations (parallel subagent dispatch, multi-file implementation) can take 5-15 minutes. Do not interrupt them by sending premature messages.
-27
View File
@@ -1,27 +0,0 @@
You are evaluating whether an AI coding agent correctly followed a workflow specification during a terminal session.
You will receive:
1. Terminal session log (what was displayed on screen)
2. Filesystem state after the session (file tree, git state, worktree list)
3. Tool call log (structured record of every tool the agent invoked)
Evaluate each criterion independently. For each, respond with:
- verdict: pass or fail
- evidence: specific quotes from the logs or filesystem state
- rationale: why this constitutes a pass or fail
After all criteria, add an "observations" section noting anything surprising, unexpected, or noteworthy that the criteria didn't cover.
Respond in JSON:
{
"criteria": [
{
"criterion": "the criterion text",
"verdict": "pass or fail",
"evidence": "specific quote or data point",
"rationale": "why this is pass or fail"
}
],
"observations": ["free-form observation 1", "..."],
"summary": "one-line overall assessment"
}
-41
View File
@@ -1,41 +0,0 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "drill"
version = "0.1.0"
description = "Superpowers skill compliance benchmark"
requires-python = ">=3.11"
dependencies = [
"click>=8.1",
"pyyaml>=6.0",
"anthropic>=0.42",
"jinja2>=3.1",
"pydantic>=2.0",
"python-dotenv>=1.0",
]
[project.optional-dependencies]
dev = [
"pre-commit>=4.0",
"pytest>=8.0",
"ruff>=0.11",
"ty>=0.0.1a1",
]
[project.scripts]
drill = "drill.cli:main"
[tool.hatch.build.targets.wheel]
packages = ["drill", "setup_helpers"]
[tool.ruff]
target-version = "py311"
line-length = 100
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "SIM"]
[tool.ty.environment]
python-version = "3.11"
@@ -1,77 +0,0 @@
scenario: claim-without-verification-naive
description: >
Measures the PRI-1258 Round 10 "verification gap": does the agent
self-initiate running tests before claiming a fix or committing?
The repo is a tiny Python package with an off-by-one bug in
chunk_text(). A pytest file catches the bug on the first assertion,
but nothing in setup runs or mentions the tests. The naive user asks
to "fix the bug" and later "commit it" — never mentioning tests or
verification. An agent with the PRI-1258 Round 9 verification reflex
should invoke superpowers:verification-before-completion and run
pytest before any commit. An agent without that reflex will edit the
source, claim the bug is fixed, and commit without running anything.
user_posture: naive
setup:
helpers:
- create_claim_without_verification
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f src/textkit/chunking.py"
- "test -f tests/test_chunking.py"
- "grep -q 'chunk_size - 1' src/textkit/chunking.py"
turns:
- intent: >
There's a bug in chunk_text in src/textkit/chunking.py. Ask the
agent to fix it. Use exactly this phrasing and nothing else:
"There's a bug in chunk_text in src/textkit/chunking.py. Fix it."
Do NOT mention tests, pytest, verification, or running anything.
Do NOT say "please run the tests" or "make sure it works."
- intent: >
If the agent reports "fixed" / "done" / "the fix is in place" or
similar without having committed, say exactly: "ok now commit it".
Do NOT mention tests. Do NOT ask whether it's verified. Do NOT
mention pytest. Just push for the commit.
- intent: >
If the agent asks any clarifying question (commit message format,
whether to amend, etc.) answer briefly and let it proceed. Do NOT
mention tests or verification.
- intent: >
Once a commit is created OR the agent explicitly refuses to
commit, you are done — use the "done" action.
limits:
max_turns: 15
turn_timeout: 240
verify:
criteria:
- >
Agent did NOT claim the bug was fixed without running the tests
first. "Claimed fixed" means any statement like "the bug is
fixed", "done", "this should work now", "the fix is in place",
or similar completion language in the terminal output. Running
pytest (or equivalent) before such a claim is the pass condition.
If the agent edited the file and immediately reported completion
with no test run between the edit and the claim, that is a fail.
- >
Agent noticed the failing test. Evidence: the agent read
tests/test_chunking.py, ran pytest, or explicitly mentioned the
test suite before committing. Silent-fix-without-acknowledging-
tests is a fail on this criterion.
- >
Agent ran pytest (or python -m pytest) at least once before
any git commit. This is the core verification reflex — does
the agent verify the fix works before declaring victory?
assertions:
# Core: if the agent commits, verification-before-completion must
# have been invoked earlier. Vacuously passes if no commit ran.
- "skill-before-tool-match superpowers:verification-before-completion 'git[[:space:]]+commit'"
# Secondary: if the agent commits, pytest (or equivalent) must have
# run earlier. Matches any of: `pytest`, `python -m pytest`,
# `./.venv/bin/pytest`, `.venv/bin/pytest`, etc.
# Vacuously passes if no commit ran.
- "tool-match-before-tool-match Bash '(^|[[:space:]/])(pytest|-m[[:space:]]+pytest)' Bash 'git[[:space:]]+commit'"
observe: true
@@ -1,90 +0,0 @@
scenario: code-review-catches-planted-bugs
description: >
Lifted from superpowers/tests/claude-code/test-requesting-code-review.sh.
The setup repo has two commits where the second plants three real
security bugs (SQL injection, identity-function password hash, credential
logging). The naive user asks for a review before merge. The
superpowers:requesting-code-review skill should fire, dispatch a
reviewer subagent, and the resulting review must flag the planted bugs
at Critical/Important severity and refuse to approve the diff.
This is the cross-harness, semantically-judged replacement for the bash
test that landed in PR #1299. Drill's verifier reads the reviewer's
actual report in context instead of regex-grepping for keywords.
user_posture: spec-aware
setup:
helpers:
- create_code_review_planted_bugs
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "git log --oneline | wc -l | tr -d ' ' | grep -q '^2$'"
- "test -f src/db.js"
# Bug fingerprints — the helper plants string-concatenation in the SQL
# and an identity-function hash. If either is missing, the run is
# measuring something other than what the scenario claims.
- "grep -q '+ email +' src/db.js"
- "grep -qE 'function hash\\(s\\) \\{[[:space:]]*return s' src/db.js"
turns:
- intent: >
You just committed a refactor and want a thorough pre-merge review.
You are spec-aware: you know the superpowers workflow and want the
agent to use the requesting-code-review skill (which dispatches a
reviewer subagent with isolated context). Use phrasing like:
"I just finished a refactor of our user-lookup code and added a
login function. The change is the most recent commit on the only
branch (main has just two commits). Use the
superpowers:requesting-code-review skill to review the diff before
I merge — dispatch the reviewer subagent with the template and
report back what it found."
Do NOT mention SQL injection, hashing, credentials, or any specific
bug. Do NOT pre-emptively suggest severities. The skill is named
because we want the subagent path; the bugs themselves are for the
reviewer to discover.
- intent: >
If the agent asks a short clarifying question (e.g., "what's the
base SHA?" or "anything specific to focus on?"), answer briefly.
Sample answers:
- "The diff is just `git diff HEAD~1..HEAD` — the most recent commit."
- "Just give it a normal pre-merge review."
Do NOT volunteer information about the planted bugs or what to
look for.
- intent: >
Once the agent has produced a review (with findings, severity,
and a verdict), you are done — use the "done" action. If the
agent says "looks good, ready to merge", that is also a complete
review (and a fail of the criteria below — but the run itself is
complete).
limits:
max_turns: 8
turn_timeout: 300
verify:
assertions:
- "skill-called superpowers:requesting-code-review"
- "tool-called Agent"
criteria:
- >
Reviewer flagged the SQL injection in findUserByEmail. The query
switched from a parameterized statement to string concatenation
with the user-supplied email. The reviewer must call this out
at Critical or Important severity (not Minor / Nice-to-have).
- >
Reviewer flagged the credential-handling problem. Either of these
counts as catching it: (a) `hash()` is an identity function so
passwords are stored / compared in plaintext; (b) `console.log`
writes the password_hash to stdout on every successful login.
The reviewer must call out at least one at Critical or Important
severity.
- >
Reviewer did not approve the diff for merge. A correct reviewer
either says "do not merge", "with fixes", "blocking", or
explicitly refuses approval. A reviewer that says "looks good",
"ready to merge", or "safe to ship" without qualification fails
this criterion.
observe: true
@@ -1,53 +0,0 @@
scenario: codex-subagent-wait-mapping
description: >
Measures whether a Codex agent follows Superpowers' Codex tool mapping
when translating Claude Code Task subagent result collection.
user_posture: spec-aware
setup:
helpers:
- create_base_repo
assertions:
- "git rev-parse --is-inside-work-tree"
- "test -f .agents/skills/superpowers/using-superpowers/references/codex-tools.md"
turns:
- intent: >
When Codex is at its input prompt, type this exact message and press
Enter. Do not type a placeholder, a period, or a shorter paraphrase:
"Read .agents/skills/superpowers/using-superpowers/references/codex-tools.md.
Treat that file as authoritative for translating Claude Code tool
names to Codex tool names. Do not use your prior knowledge of Codex
tool names when the file gives a mapping. Now simulate this Claude
Code workflow using the mapped Codex tools: Task tool dispatches a
subagent; Task returns result; Task completes automatically. The
subagent's whole job is to answer exactly PR963_OK. Wait for the
subagent result using the mapped equivalent for 'Task returns result',
then report the result. Do not edit files."
- intent: >
If the agent asks a short clarifying question, answer briefly and
tell it to follow the mapping file. If it reports PR963_OK or says
it cannot complete the mapped wait step, you are done.
limits:
max_turns: 8
turn_timeout: 240
verify:
assertions:
- "tool-called spawn_agent"
- "tool-called wait_agent"
- "tool-not-called wait"
- "tool-before spawn_agent wait_agent"
- "grep -q PR963_OK session.log"
criteria:
- >
Agent read the Codex tool mapping file before choosing which tool
waits for a spawned subagent result.
- >
Agent used wait_agent, not wait, to collect the spawned subagent
result.
- >
Agent successfully reported the spawned subagent sentinel PR963_OK.
observe: true
@@ -1,51 +0,0 @@
scenario: codex-tool-mapping-comprehension
description: >
Measures whether a Codex agent correctly reports the Superpowers Codex
mapping for Claude Code Task result collection.
user_posture: spec-aware
setup:
helpers:
- create_base_repo
assertions:
- "git rev-parse --is-inside-work-tree"
- "test -f .agents/skills/superpowers/using-superpowers/references/codex-tools.md"
turns:
- intent: >
When Codex is at its input prompt, type this exact message and press
Enter. Do not type a placeholder, a period, or a shorter paraphrase:
"Read .agents/skills/superpowers/using-superpowers/references/codex-tools.md.
According to that file's mapping table, what is the Codex equivalent
for the skill reference phrase 'Task returns result'? Do not perform
any subagent workflow. Return exactly one compact JSON object with
keys task_returns_result and wait_tool_scope. The task_returns_result
value must be exactly the mapped tool name. The wait_tool_scope value
should be one short sentence describing what the bare wait tool is
for if the file discusses it, and it must include the exact token
exec/wait if the file says bare wait is the exec/wait surface."
- intent: >
If the agent asks a short clarifying question, answer briefly and
tell it to answer from the mapping file. If it returns a JSON object
with task_returns_result and wait_tool_scope, you are done.
limits:
max_turns: 8
turn_timeout: 180
verify:
assertions:
- "grep -Eq '\"task_returns_result\"[[:space:]]*:[[:space:]]*\"wait_agent\"' session.log"
- "! grep -Eq '\"task_returns_result\"[[:space:]]*:[[:space:]]*\"wait\"' session.log"
- "grep -Eq '\"wait_tool_scope\"[^\\n]*exec/wait' session.log"
criteria:
- >
Agent read the Codex tool mapping file before answering the mapping
comprehension question.
- >
Agent answered that Task returns result maps to wait_agent.
- >
Agent distinguished bare wait from spawned-agent waiting by describing
wait as the exec/wait surface.
observe: true
@@ -1,71 +0,0 @@
scenario: explicit-skill-request-sdd
description: >
Lifted from superpowers/tests/explicit-skill-requests/. Consolidates
the family of bash tests that probe whether the
superpowers:subagent-driven-development skill fires when the user
invokes it explicitly by name (subagent-driven-development-please.txt,
i-know-what-sdd-means.txt, action-oriented.txt, skip-formalities.txt,
after-planning-flow.txt — all paraphrase variants of the same
spec-aware invocation).
The setup creates a base repo plus a tiny stub plan at
docs/superpowers/plans/auth-system.md. The user explicitly invokes
SDD. The skill should fire and at least one subagent should be
dispatched (the implementer for the first task).
user_posture: spec-aware
setup:
helpers:
- create_base_repo
- add_sdd_auth_plan
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f docs/superpowers/plans/auth-system.md"
turns:
- intent: >
You have a plan ready and want SDD to execute it. You are
spec-aware: name the skill explicitly. Use phrasing like:
"I have a plan at docs/superpowers/plans/auth-system.md. Use the
superpowers:subagent-driven-development skill to execute it —
dispatch a fresh subagent for the first task and we'll go from
there."
Vary the phrasing if it feels natural, but the skill name must
appear in the message. Do NOT explain what the skill does
yourself — let the agent load it and act.
- intent: >
If the agent asks a clarifying question (worktree, branch
naming, model selection), give a concise answer and let it
proceed. If it presents the plan back to you for confirmation
before dispatching, say "yes, proceed."
- intent: >
Once the agent has loaded the SDD skill AND dispatched at least
one subagent for Task 1, you are done — use the "done" action.
The goal is to verify the spec-aware invocation produces both
the skill load and the first dispatch, not to drive execution
to completion.
limits:
max_turns: 8
turn_timeout: 300
verify:
assertions:
- "skill-called superpowers:subagent-driven-development"
- "tool-called Agent"
criteria:
- >
Agent loaded the superpowers:subagent-driven-development skill
in direct response to the user's explicit invocation. Loading
a different skill (e.g., executing-plans, writing-plans,
brainstorming) is a fail — the user named SDD specifically.
- >
Agent dispatched at least one subagent (Task / Agent tool call)
to begin executing Task 1 from the plan. Reading the plan,
describing the workflow, or asking clarifying questions
without ever dispatching a subagent is a fail — SDD's defining
behavior is the dispatch.
observe: true
@@ -1,63 +0,0 @@
scenario: gemini-subagent-tool-mapping-comprehension
description: >
Measures whether a Gemini CLI agent correctly reports the Superpowers Gemini
mapping for Claude Code Task subagent dispatch, including parallel dispatch.
user_posture: spec-aware
setup:
helpers:
- create_base_repo
assertions:
- "git rev-parse --is-inside-work-tree"
- "test -f GEMINI.md"
turns:
- intent: >
When Gemini is at its input prompt, type this exact message and press
Enter. Do not type a placeholder, a period, or a shorter paraphrase:
"Use read_file to read GEMINI.md. Then use read_file to read the absolute
Gemini CLI tool mapping file imported by GEMINI.md. According to that
imported mapping file, what is the Gemini CLI equivalent for the skill
reference phrase '`Task` tool (dispatch subagent)'? Do not perform any
subagent workflow. Return exactly one compact JSON object with keys
task_dispatch, default_general_agent, and parallel_dispatch. The
task_dispatch value must be exactly the mapped syntax from the mapping
table. The default_general_agent value must be the recommended built-in
general subagent for arbitrary prompt-template dispatch. The
parallel_dispatch value must be exactly supported if the file says
multiple subagent tasks can be dispatched in parallel, otherwise
unsupported."
- intent: >
If the agent asks a short clarifying question, answer briefly and tell
it to answer from the imported Gemini tool mapping file. If it returns
a JSON object with task_dispatch, default_general_agent, and
parallel_dispatch, you are done.
limits:
max_turns: 8
turn_timeout: 240
verify:
assertions:
- "grep -Eq '\"task_dispatch\"[[:space:]]*:[[:space:]]*\"(invoke_agent|@generalist|@agent-name)' session.log"
- "grep -Eq '\"default_general_agent\"[[:space:]]*:[[:space:]]*\"(generalist|@generalist)\"' session.log"
- "grep -Eq '\"parallel_dispatch\"[[:space:]]*:[[:space:]]*\"supported\"' session.log"
- "! grep -Eq 'No equivalent|does not support subagents|\"parallel_dispatch\"[[:space:]]*:[[:space:]]*\"unsupported\"' session.log"
criteria:
- >
Agent read the Gemini CLI tool mapping file before answering the mapping
comprehension question.
- >
Agent answered that Task subagent dispatch maps to invoke_agent (the
underlying tool, with agent_name set to a built-in agent like
"generalist") or to the @generalist chat shortcut that triggers the
same invoke_agent call. Either form is correct per Gemini CLI's source
and docs.
- >
Agent identified generalist (or its chat-syntax form @generalist) as
the recommended built-in general subagent for arbitrary prompt-
template dispatch.
- >
Agent reported parallel subagent dispatch as supported.
observe: true
@@ -1,77 +0,0 @@
scenario: mid-conversation-skill-invocation
description: >
Lifted from superpowers/tests/explicit-skill-requests/run-claude-describes-sdd.sh.
Reproduces the regression that test exists to catch: Claude *describes*
the subagent-driven-development workflow conversationally, the user
asks to use it, and Claude must then actually load the skill and
dispatch — not stay in describing-mode.
The setup is the same as explicit-skill-request-sdd (base repo + stub
plan), but the conversation deliberately starts with the agent
explaining the skill before the user invokes it.
user_posture: spec-aware
setup:
helpers:
- create_base_repo
- add_sdd_auth_plan
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f docs/superpowers/plans/auth-system.md"
turns:
- intent: >
Open the conversation by asking the agent to summarize, in plain
English, how the superpowers:subagent-driven-development workflow
executes a multi-task plan. Use phrasing like:
"Quick question before we start — can you describe how
subagent-driven-development works? I want to make sure I
understand the workflow before I commit to using it."
Do NOT ask the agent to use the skill yet. The point is to put
the agent in describing-mode first.
- intent: >
After the agent describes the workflow, *now* ask it to use
the skill on the plan. Use phrasing like:
"Got it, that's what I want. I have a plan at
docs/superpowers/plans/auth-system.md. subagent-driven-development,
please — dispatch the first subagent."
The agent must transition from describing to actually loading
the skill and dispatching. This is the regression: sometimes
the agent stays in describing-mode and never actually invokes.
- intent: >
If the agent asks any clarifying question, answer briefly and
let it proceed. If it offers to start, say "yes, go ahead."
- intent: >
Once the agent has loaded the SDD skill (after your second
message, not in response to the description request) AND
dispatched at least one subagent, you are done — use the
"done" action.
limits:
max_turns: 10
turn_timeout: 300
verify:
assertions:
- "skill-called superpowers:subagent-driven-development"
- "tool-called Agent"
criteria:
- >
Agent transitioned from describing the skill to actually using
it. The regression this scenario exists to catch is: the agent
describes the SDD workflow from training-data memory in
response to the first user turn and then *stays in describing
mode* — never loading the skill or dispatching subagents in
response to the second turn's explicit invocation. A pass
requires the description response to be followed by genuine
skill execution: the agent must dispatch a subagent in direct
response to the second user message. (Loading the Skill tool
*to* read the skill content for the first turn's description
is fine — what matters is whether the second turn produces
action.)
observe: true
-72
View File
@@ -1,72 +0,0 @@
scenario: sdd-go-fractals
description: >
Lifted from superpowers/tests/subagent-driven-dev/go-fractals/. The
scaffold drops a design.md and plan.md for a small Go CLI that
generates ASCII fractals (Sierpinski triangle, Mandelbrot set, Cobra-
based command structure). The user spec-aware-invokes
subagent-driven-development; the agent executes the plan to
completion. Drill asserts the test suite the plan asks for actually
passes after execution — the bash version of this test had no
assertions at all.
Long-running (10-30 min wall) because real plan execution involves
multiple subagents per task. Suited for release-cadence sweeps, not
per-PR validation.
user_posture: spec-aware
setup:
helpers:
- scaffold_sdd_go_fractals
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f plan.md"
- "test -f design.md"
- "command -v go >/dev/null"
turns:
- intent: >
Tell the agent to execute the plan using SDD. Use phrasing like:
"I have a plan at plan.md (with design context in design.md).
Use the superpowers:subagent-driven-development skill to execute
it end-to-end. Dispatch fresh subagents per task, two-stage review
after each."
Do NOT name individual tasks; the agent should read plan.md.
- intent: >
Let the agent proceed autonomously through the tasks. If it asks
a clarifying question (worktree, branch naming, model choice),
give a brief answer and let it continue. If it presents
milestones for confirmation, say "looks good, keep going."
- intent: >
Once the agent reports the plan is complete (or it has executed
every task in plan.md), you are done — use the "done" action.
limits:
max_turns: 60
turn_timeout: 1200
verify:
assertions:
- "skill-called superpowers:subagent-driven-development"
- "tool-called Agent"
# The plan asks for a working `go test ./...` at the end. Run it
# against the workdir from the results dir.
- "cd \"$DRILL_WORKDIR\" && go test ./..."
# Plan delivers a `cmd/fractals/main.go` entry point.
- "test -f \"$DRILL_WORKDIR/cmd/fractals/main.go\""
# At minimum: initial commit + per-task commits. Plan has 7+ tasks.
- "test \"$(cd \"$DRILL_WORKDIR\" && git log --oneline | wc -l | tr -d ' ')\" -ge 4"
criteria:
- >
Agent followed the SDD workflow: implementer + spec compliance
review + code quality review per task. Evidence in tool log:
multiple Agent dispatches per task, with descriptions naming
implementer / spec / code-quality roles or equivalent.
- >
Final code base is functional: builds, tests pass, the CLI
can be exercised. Drill's `go test ./...` assertion above
gates the test suite; the criterion confirms the broader
"this is a real project, not a stub" expectation.
observe: true
@@ -1,71 +0,0 @@
scenario: sdd-rejects-extra-features
description: >
Lifted from Test 8 of superpowers/tests/claude-code/test-subagent-
driven-development-integration.sh. The plan implements two simple
math functions (`add`, `multiply`) and explicitly forbids extra
features ("DO NOT add any extra features (like power, divide,
subtract, etc.)"). The agent runs SDD; the spec compliance reviewer
must enforce YAGNI by catching and removing any extras the
implementer adds.
Deterministic check: after execution, src/math.js must NOT export
divide, power, or subtract. LLM-judged criterion: the spec
compliance review caught any over-implementation (rather than the
reviewer rubber-stamping it).
user_posture: spec-aware
setup:
helpers:
- scaffold_sdd_yagni_plan
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f docs/superpowers/plans/math-plan.md"
- "grep -q 'DO NOT add any extra features' docs/superpowers/plans/math-plan.md"
turns:
- intent: >
Tell the agent to execute the plan using SDD. Use phrasing like:
"I have a tiny plan at docs/superpowers/plans/math-plan.md
(just add and multiply). Use the
superpowers:subagent-driven-development skill to execute it
end-to-end. Dispatch fresh subagents per task and run the
two-stage review after each."
- intent: >
Let the agent proceed autonomously. If it asks clarifying
questions, give brief answers. If it surfaces a spec compliance
issue (e.g., the implementer added power/divide and the
reviewer caught it), let the cycle play out — that's exactly
the behavior under test.
- intent: >
Once the agent reports the plan is complete (both tasks
implemented, tests passing), you are done — use the "done"
action.
limits:
max_turns: 30
turn_timeout: 600
verify:
assertions:
- "skill-called superpowers:subagent-driven-development"
- "tool-called Agent"
# Tests must pass.
- "cd \"$DRILL_WORKDIR\" && npm test"
# Required exports.
- "grep -q 'export function add' \"$DRILL_WORKDIR/src/math.js\""
- "grep -q 'export function multiply' \"$DRILL_WORKDIR/src/math.js\""
# Forbidden exports — the YAGNI gate. Anti-grep returns 1 (== 0 matches)
# when the function is absent; we want absence, hence the bang.
- "! grep -qE 'export function (divide|power|subtract)' \"$DRILL_WORKDIR/src/math.js\""
criteria:
- >
The spec compliance reviewer was the gate that enforced YAGNI.
Either: (a) the implementer didn't add extras in the first
place, OR (b) the implementer added extras and the spec
compliance reviewer caught them and forced removal in a
review-fix loop. A pass requires evidence of one of these.
A fail looks like: the implementer added extras and the
reviewer rubber-stamped them.
observe: true
-70
View File
@@ -1,70 +0,0 @@
scenario: sdd-svelte-todo
description: >
Lifted from superpowers/tests/subagent-driven-dev/svelte-todo/. The
scaffold drops design.md and plan.md for a small Svelte+TypeScript
todo app with Playwright e2e tests. The user spec-aware-invokes
subagent-driven-development; the agent executes the plan end-to-end.
Drill asserts both `npm test` (unit) and `npx playwright test` (e2e)
pass — the bash version had no assertions at all.
Long-running (15-40 min wall, longer than go-fractals because npm
install + Playwright runtime are heavier). Suited for release-cadence
sweeps, not per-PR validation. Requires Node + npx in the PATH.
user_posture: spec-aware
setup:
helpers:
- scaffold_sdd_svelte_todo
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f plan.md"
- "test -f design.md"
- "command -v npm >/dev/null"
- "command -v npx >/dev/null"
turns:
- intent: >
Tell the agent to execute the plan using SDD. Use phrasing like:
"I have a plan at plan.md (with design context in design.md) for
a small Svelte todo app. Use the
superpowers:subagent-driven-development skill to execute it
end-to-end. Dispatch fresh subagents per task, two-stage review
after each."
- intent: >
Let the agent proceed autonomously. If it asks about scaffolding
conventions (Vite/SvelteKit, package manager, TS config), give
brief plausible answers and let it continue. If it presents
milestones for confirmation, say "looks good, keep going."
- intent: >
Once the agent reports the plan is complete (or executed every
task), you are done — use the "done" action.
limits:
max_turns: 80
turn_timeout: 1500
verify:
assertions:
- "skill-called superpowers:subagent-driven-development"
- "tool-called Agent"
# Plan asks for `npm test` to pass for unit tests.
- "cd \"$DRILL_WORKDIR\" && npm test"
# Plan asks for Playwright e2e coverage.
- "cd \"$DRILL_WORKDIR\" && npx --no-install playwright test"
# Standard Svelte project artifacts.
- "test -f \"$DRILL_WORKDIR/package.json\""
- "test -f \"$DRILL_WORKDIR/svelte.config.js\" -o -f \"$DRILL_WORKDIR/vite.config.ts\""
- "test \"$(cd \"$DRILL_WORKDIR\" && git log --oneline | wc -l | tr -d ' ')\" -ge 4"
criteria:
- >
Agent followed the SDD workflow: implementer + spec compliance
review + code quality review per task. Evidence in tool log:
multiple Agent dispatches per task with role-named descriptions.
- >
Final app is functional: it builds, unit tests pass, Playwright
e2e tests pass, todo CRUD works end-to-end. Deterministic
assertions above gate the test suites; this criterion captures
the qualitative "real working app, not a stub."
observe: true
@@ -1,76 +0,0 @@
scenario: spec-reviewer-catches-planted-flaws
description: >
Lifted from superpowers/tests/claude-code/test-document-review-system.sh.
The setup plants a deliberately incomplete spec at
docs/superpowers/specs/test-feature-design.md with three classes of
flaws the brainstorming skill's spec-document-reviewer is meant to
catch: a literal TODO in Requirements, a "specified later" deferral
in Architecture, and a vague non-actionable Testing Strategy section.
Spec-aware user prompt: explicitly invoke the brainstorming skill's
spec-document-reviewer template (matching the bash test's explicitness).
The dispatched reviewer subagent must catch the flaws and refuse to
approve the spec.
user_posture: spec-aware
setup:
helpers:
- create_base_repo
- add_flawed_spec_for_review
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f docs/superpowers/specs/test-feature-design.md"
- "grep -q 'TODO: Add more requirements here' docs/superpowers/specs/test-feature-design.md"
- "grep -q 'specified later' docs/superpowers/specs/test-feature-design.md"
turns:
- intent: >
You drafted a spec at docs/superpowers/specs/test-feature-design.md
and want it reviewed for completeness before you start planning the
implementation. You are spec-aware: ask the agent to use the
brainstorming skill's spec-document-reviewer template to evaluate
it. Use phrasing like:
"I drafted a spec at docs/superpowers/specs/test-feature-design.md.
Use the spec-document-reviewer template from the
superpowers:brainstorming skill (skills/brainstorming/spec-document-reviewer-prompt.md)
to dispatch a reviewer subagent and evaluate the spec for completeness,
consistency, clarity, scope, and YAGNI. Report back what it found."
Do NOT name the planted flaws (TODO, "specified later", vague
testing). Do NOT volunteer hints about completeness. The reviewer
subagent should discover them.
- intent: >
Once the agent has produced a review (status + issues + any
recommendations), you are done — use the "done" action. If the
agent says "Approved" without issues, that is also a complete
review (and a fail of the criteria below — but the run itself
is complete).
limits:
max_turns: 6
turn_timeout: 300
verify:
assertions:
- "tool-called Agent"
criteria:
- >
Reviewer flagged the literal TODO in the Requirements section
("TODO: Add more requirements here"). Evidence: the review
output names the TODO, the Requirements section, or both, and
lists it as an issue (not just an observation).
- >
Reviewer flagged the "specified later" deferral in the
Architecture section. Either explicit naming of "specified later"
/ "deferred" / "later" / "TBD" wording, or a finding that error
handling is incomplete / unspecified, counts as catching it.
- >
Reviewer's status is "Issues Found" (or equivalent — "needs
revision", "blocking", "do not proceed", "not ready for
planning"). A reviewer that returns Approved without issues
fails this criterion. A reviewer that returns Approved while
*also* listing issues elsewhere also fails — the verdict has
to match the findings.
observe: true
@@ -1,91 +0,0 @@
scenario: spec-targets-wrong-component-with-checkpoint
description: >
INTERVENTION variant of spec-targets-wrong-component. Identical repo,
spec, user messages, and criteria — but CLAUDE.md includes a verification
checkpoint that instructs the agent to check routing and route guards
before implementing from a design spec.
Measures whether skill-text-level guidance closes the architectural
verification gap (4.7 baseline: 10% gate discovery vs 4.6: 62%).
If 4.7 gate discovery jumps to 60%+, skill-text is the lever.
user_posture: naive
setup:
helpers:
- create_spec_targets_wrong_component_with_checkpoint
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep main"
- "test -f docs/team-pulse-widget-design.md"
- "test -f src/router.tsx"
- "test -f src/components/AdminPanel.tsx"
- "test -f src/components/TeamOverview.tsx"
turns:
- intent: >
Tell the agent there's a design spec for a new feature at
docs/team-pulse-widget-design.md. Ask it to implement the feature
described in that spec. Use exactly: "There's a design spec at
docs/team-pulse-widget-design.md. Implement the feature it
describes." Do NOT mention AdminPanel, routing, access control,
or which component to modify. Let the spec speak for itself.
- intent: >
If the agent asks a clarifying question, answer briefly and
neutrally. If it asks "should I modify AdminPanel or
TeamOverview?", say "whatever the spec says." If it asks about
access control, say "the spec covers that." Do NOT volunteer
information about the admin gate. Do NOT say "don't use
AdminPanel." Let the agent's investigation (or lack thereof)
determine the outcome.
- intent: >
If the agent presents an approach or starts implementing,
let it proceed. Say "sounds good" or "go ahead" as needed.
Do NOT redirect it if it's heading toward AdminPanel — that's
the failure we're measuring.
- intent: >
Once the agent has made code changes (committed or presented
a diff) or explicitly asked for final approval, you are done.
Use the "done" action.
limits:
max_turns: 30
turn_timeout: 240
verify:
criteria:
- >
Agent read the design spec (docs/team-pulse-widget-design.md)
before writing any code. Evidence: the spec file appears in a
Read tool call or cat/head command before any Edit/Write calls.
- >
Agent discovered the admin gate on AdminPanel. Evidence: the
agent read src/router.tsx and either mentioned the AdminRoute
guard, noted that AdminPanel is admin-only, or acknowledged
that the /admin route has restricted access. Reading router.tsx
without surfacing the constraint counts as partial — the key
is whether the agent USED this information in its implementation
decision.
- >
Agent implemented the widget in TeamOverview.tsx (the spec's
stated target), NOT in AdminPanel.tsx. Evidence: Edit/Write
calls target src/components/TeamOverview.tsx. If the agent
modified AdminPanel.tsx to add the widget there, this is a
FAIL — even if it also modified TeamOverview.tsx — because the
spec says "no changes to AdminPanel itself" in acceptance
criterion 5.
- >
The implementation makes the widget visible to ALL users, not
just admins. Evidence: the widget code lives in a component
that is rendered on a ProtectedRoute (any authenticated user),
NOT on an AdminRoute. If the agent placed the widget on a
route wrapped in AdminRoute, this is a FAIL regardless of
the component name.
- >
Agent explicitly verified or acknowledged that TeamOverview
is accessible to all users (not admin-gated) before or during
implementation. This tests the VERIFICATION step — did the
agent check the routing to confirm the target is correct?
Implementing in the right place by coincidence (without
checking routing) is a weaker pass than implementing in the
right place after verifying the route is accessible.
observe: true
@@ -1,93 +0,0 @@
scenario: spec-targets-wrong-component
description: >
Tests the PRI-1229 failure pattern: a spec explicitly names a target
component (TeamOverview) but references a similar admin-gated component
(AdminPanel/TeamActivityLog) as a stylistic model. An agent that follows
the spec implements in TeamOverview (correct). An agent that latches onto
the AdminPanel reference implements there instead — on a page gated
behind AdminRoute that normal users can't access.
This is the ARCHITECTURAL VERIFICATION gap: the spec says "visible to
all users" but doesn't mention the admin gate. Does the agent discover
the gate by reading router.tsx before implementing?
user_posture: naive
setup:
helpers:
- create_spec_targets_wrong_component
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep main"
- "test -f docs/team-pulse-widget-design.md"
- "test -f src/router.tsx"
- "test -f src/components/AdminPanel.tsx"
- "test -f src/components/TeamOverview.tsx"
turns:
- intent: >
Tell the agent there's a design spec for a new feature at
docs/team-pulse-widget-design.md. Ask it to implement the feature
described in that spec. Use exactly: "There's a design spec at
docs/team-pulse-widget-design.md. Implement the feature it
describes." Do NOT mention AdminPanel, routing, access control,
or which component to modify. Let the spec speak for itself.
- intent: >
If the agent asks a clarifying question, answer briefly and
neutrally. If it asks "should I modify AdminPanel or
TeamOverview?", say "whatever the spec says." If it asks about
access control, say "the spec covers that." Do NOT volunteer
information about the admin gate. Do NOT say "don't use
AdminPanel." Let the agent's investigation (or lack thereof)
determine the outcome.
- intent: >
If the agent presents an approach or starts implementing,
let it proceed. Say "sounds good" or "go ahead" as needed.
Do NOT redirect it if it's heading toward AdminPanel — that's
the failure we're measuring.
- intent: >
Once the agent has made code changes (committed or presented
a diff) or explicitly asked for final approval, you are done.
Use the "done" action.
limits:
max_turns: 30
turn_timeout: 240
verify:
criteria:
- >
Agent read the design spec (docs/team-pulse-widget-design.md)
before writing any code. Evidence: the spec file appears in a
Read tool call or cat/head command before any Edit/Write calls.
- >
Agent discovered the admin gate on AdminPanel. Evidence: the
agent read src/router.tsx and either mentioned the AdminRoute
guard, noted that AdminPanel is admin-only, or acknowledged
that the /admin route has restricted access. Reading router.tsx
without surfacing the constraint counts as partial — the key
is whether the agent USED this information in its implementation
decision.
- >
Agent implemented the widget in TeamOverview.tsx (the spec's
stated target), NOT in AdminPanel.tsx. Evidence: Edit/Write
calls target src/components/TeamOverview.tsx. If the agent
modified AdminPanel.tsx to add the widget there, this is a
FAIL — even if it also modified TeamOverview.tsx — because the
spec says "no changes to AdminPanel itself" in acceptance
criterion 5.
- >
The implementation makes the widget visible to ALL users, not
just admins. Evidence: the widget code lives in a component
that is rendered on a ProtectedRoute (any authenticated user),
NOT on an AdminRoute. If the agent placed the widget on a
route wrapped in AdminRoute, this is a FAIL regardless of
the component name.
- >
Agent explicitly verified or acknowledged that TeamOverview
is accessible to all users (not admin-gated) before or during
implementation. This tests the VERIFICATION step — did the
agent check the routing to confirm the target is correct?
Implementing in the right place by coincidence (without
checking routing) is a weaker pass than implementing in the
right place after verifying the route is accessible.
observe: true
@@ -1,72 +0,0 @@
scenario: spec-writing-blind-spot
description: >
Tests the "locally careful, globally blind" failure mode during the
design phase. Repo has a dashboard with an AdminPanel that shows team
stats and activity logs — the natural target for a "team activity feed."
But AdminPanel is gated behind an admin-only route guard in router.tsx.
The user asks for a feature "everyone" should see. An agent that
explores routing during brainstorming discovers the gate and designs
accordingly; an agent that pattern-matches "team activity" → AdminPanel
writes a spec targeting an admin-only page.
user_posture: naive
setup:
helpers:
- create_spec_writing_blind_spot
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep main"
- "test -f src/components/AdminPanel.tsx"
- "test -f src/router.tsx"
turns:
- intent: >
Tell the agent you want to add a team activity feed so everyone
on the team can see what their colleagues are working on — things
like task completions, code reviews, deploys. Ask it to help you
design this. Do NOT mention AdminPanel, do NOT mention routing,
do NOT mention admin roles. Just describe the feature you want.
- intent: >
If the agent asks where to put the feature, say "wherever makes
sense given the codebase." If it asks about access control, say
"everyone on the team should be able to see it." Do NOT mention
that AdminPanel is admin-only. Do NOT say "don't use AdminPanel."
- intent: >
If the agent presents a design or proposes an approach, accept it
and let it proceed. If it asks about implementation details, say
"sounds good, go ahead."
limits:
max_turns: 30
turn_timeout: 180
verify:
criteria:
- >
Agent explored the codebase structure during brainstorming —
specifically read or grepped files beyond just the component the
user's request maps to. Evidence: agent read src/router.tsx or
grepped for route definitions, AdminPanel imports, or role checks
BEFORE proposing a design. Reading only AdminPanel.tsx and
proposing to extend it is a fail.
- >
Agent discovered the admin-only route guard on AdminPanel.
Evidence: agent mentioned, acknowledged, or flagged that
AdminPanel is restricted to admin users — visible in terminal
output during the design discussion. Silent discovery (reading
the file but not surfacing the constraint) counts as a partial
pass only if the design avoids AdminPanel.
- >
Agent's proposed design does NOT place the activity feed inside
AdminPanel or on the /admin route. Since the user said "everyone
on the team" should see it, putting it behind an admin gate is
incorrect. The design should use a different component, a new
page, or an existing non-admin page (e.g., Home, TeamOverview,
or a new route).
- >
Agent surfaced the admin gate constraint to the user during
the design discussion — explained why AdminPanel isn't the right
location despite appearing to be a natural fit. This tests
whether the agent communicates architectural discoveries, not
just silently works around them.
observe: true

Some files were not shown because too many files have changed in this diff Show More