Verification & Review (quality)

> Provenance: .agents/skills/ws-plan-verify/SKILL.md, .agents/skills/ws-code-review/SKILL.md, .agents/skills/ws-testing/SKILL.md, .agents/skills/ws-shared/runtime/scripts/scan_stack_invariants.cjs, .agents/skills/ws-fix-pr/SKILL.md, .agents/skills/ws-goal-fix-pr/SKILL.md, .agents/skills/ws-implement-tasks/SKILL.md, .agents/skills/ws-implement-tasks/scripts/check_test_adequacy.cjs, living synthesis of specs 0004, 0005, 0015, 0028, 0031, 0038, 0044, 0046, 0050, 0067, 0082, 0084, 0089, 0095, 0098-us-353, 0108-code-review-findings-fixes, 0109-code-review-round-2-fixes, 0113-us-380, 0133-us-430, 0147-us-446, 0156-per-task-test-adequacy-review, 0160-explicit-knowledge-chain.

Feature [edit]

Quality is a left-shifted, fail-closed chain embedded in the Spec-to-PR orchestrators. ws-plan-verify scores spec compliance and advances only when the derived ledger reaches defaults.minVerifyScore (default 9), never from self-reported scores. ws-code-review runs two-phase adversarial review with fix and re-review loops capped at three rounds. ws-testing executes unit, integration, end-to-end, and coverage gates with an opt-in mutation substep. ws-preview offers a user-invoked CI-shaped dry-run that publishes nothing. ws-fable-judge adversarial audits and ws-secrets-leak-review scans gate shipping. Stack-invariant profiles block pull requests when reviewer-aligned defects appear. ws-fix-pr and ws-goal-fix-pr converge PR threads with proactive same-class sweeps before resolution. ws-implement-tasks runs a per-task Test Adequacy review after its TDD cycle: every task acceptance criterion binds to a covering test with file-and-line evidence, every mapped test proves its key assertion fails under a wrong implementation, and no test ships without a backing requirement.

How it works [edit]

Verify scores derive exclusively from ac-ledger.json produced by ac_ledger.cjs. A score of 9 with known defects is unreachable by design. Uncovered negative scenarios cap the score at 8. Critical stack-invariant violations cap Step 5 below minVerifyScore and force scoreAndRefine back into implementation. When fable.enabled and auditVerdictsBlockShip include refuted, a REFUTED verdict caps below the bar and halts ws-ship-pr. Secrets scan failure halts push and PR creation. Adversarial audits also check source-chain compliance: every audited claim must cite the knowledge-chain link that grounds it, an unsourced claim without the UNCERTAIN marker fails the sourcing check, inventing an API, path, number, version, or tool output with no chain reference fails the anti-fabrication check, and UNCERTAIN (no chain source grounds the claim) is never interchangeable with UNVERIFIABLE (a verification that cannot be re-run).

A dirty baseline does not cost the score. An alias result sets knownDefect only when it declares productFailure: true or its failingPaths intersect workflow filesTouched; a bare non-zero exit is not enough. ws-plan-verify enumerates failing paths before linking a format or build alias as a defect, and links untouched paths as skipReason: baseline-dirty with the real exit code. To make a first-pass 10 reachable, ws-implement-tasks runs every scoring alias (backendFormat, backendBuild, backendTest, frontendTest) before handoff and formats only its own files when the format check fails. When ws-plan-interview closes a decision that contradicts an acceptance criterion, it rewrites that sentence in the spec of record and the registered step-00 copy in the same step, so verify never reads a stale criterion.

Stack invariants are non-negotiable. Step 6 requires two-phase review proof covering Evidence, Failure, Missing-Protection, and Discards categories with the stack rule pack loaded from configured profiles. The optional localReviewCommand dry-run stays read-only and never replaces remote CI, and Step 6 resolves that runner only from config.json.verification.localReviewCommand (or preview.localReviewCommand) so no branded reviewer path is hardcoded; stack anti-patterns are detected by running scan_stack_invariants.cjs against {skillsRoot}/ws-shared/runtime/stacks/ rather than restating cheat sheets in step bodies. Mutation testing runs only after green build, unit, integration, and coverage steps and before the Step 7 verdict in the standard pipeline; below-threshold mutation scores fail Step 7 into fix mode, and the tester role never edits product code directly. skipQualityGates bypasses orchestration gates only: build, tests, and leak scans still run, and mutation or sabotage failures stay fail-closed.

Proactive class sweep after validating a review thread requires naming the defect class, searching code, MEMORY, PR context, and pattern files, fixing small local siblings immediately, and recording skips with path and reason. Resolving after an anchor-only fix while unrecorded siblings remain is forbidden. Fix-PR convergence follows the surgical-commit contract on a dirty worktree: preExistingDirty paths stay unstaged, staged WIP is separated from fix hunks per tools.md commit-code (git diff HEAD, git diff --cached, and git status --porcelain on the path, git apply --cached, git restore --staged when the index holds unrelated hunks), and a score 6–10 thread whose anchor cannot be separated stays open or is escalated with path + reason instead of being resolved without a landed commit. Verification keys matching /^_/ are never required aliases; a valid skipReason counts as observed without defect caps; genuinely missing required aliases still fail closed. Configured aliases with an empty command are skipped, never executed (verify.cjs).

The goal loop is hardened against wedged loop sessions. Dispatch prompts order the model to continue with tool calls in the same response and never end the turn after the VerboseMode preview, update_state dispatch examples always show the state-path form because a bare workflow id fails, and batch-worker results delivered back to the loop arrive bounded as summaries plus artifact pointers so an oversized payload cannot wedge the follow-up turn. The parent watches loop liveness through round-log freshness and session-log silence, and a suspected wedge triggers a fresh provider list-threads plus check-pr-status read before the resume-takeover procedure dispatches a fresh loop with the full handoff brief or drives the remaining batches inline on the parent. The workflow-side watch is now one deterministic read-only verdict from worker_liveness_check.cjs (no-baseline → alive → pending → wedged, exit 2 only for wedged), backed by the batch worker's first liveness marker: durable progress outranks a missing marker, only a proven wedge authorizes takeover, a batch that is not proven wedged never gets a second worker, and a ping outcome is recorded as watchdog evidence instead of silence (pipeline details).

Fix-PR convergence runs under orchestrator dispatch: the ws-goal-fix-pr session owns the goal loop inline while each Act round batch executes in a fresh worker running one ordered fixPrPlan → fixPrExec pair. Workers resolve fixPrPlan through reviewerModel and fixPrExec through executionModel along the stepModels → preset role → top-level role → captured session model chain, retry under the session model on host rejection while recording configured versus actual, and emit ordered dispatch telemetry without ever finishing the outer Step 9. Guards stay intact across the dispatch boundary: revision-guarded updates, blocked verdicts only after three identical consecutive rounds, $RUNTIME_DIR under {us-dir}/.runtime, zero-mutation dry-run, and the post-round learning rule that forbids Learning: N/A when the round fixed a valid reviewer or CI defect (the batch worker owns the per-round MEMORY write; the session records only the reported Learning: titles).

Coverage is pushed toward completeness as a quality gate rather than a report alone. The suite grows until near-total line and branch coverage holds across runtime resolvers, guards, and provider intents, with the verification manifest recording per-suite evidence so regressions in dial-tone helpers surface before they reach review. Review findings close in audited rounds: each round carries its own plan gate, fix pass, re-review, and testing report, and a round ships only when its review threads resolve with landed commits rather than anchor-only fixes.

Preview via /ws-preview or /pipeline-review is user-only, never model-invoked, always --dry-run, includes uncommitted changes, and stops after the summary without shipping. The same configured dry-run command is additionally reused as the optional non-blocking Step 4b gate inside ws-ship-pr, where it runs verbatim after push and before Create PR, reports findings on the Prepare-to-PR board, and always continues shipping regardless of outcome.

Test adequacy is judged where tests are written, not at scoring time. After the TDD cycle for each task, the implement worker records an adequacy record asserting a three-way binding: each task acceptance criterion maps to at least one covering test with file-and-line evidence whose name occurs inside the declared range, each mapped test passes a non-shallow litmus showing its key assertion fails under a wrong implementation (one targeted inversion or documented wrong-code run per test, recording test name and exit code), and every test the task added maps back to a task criterion or a spec negative scenario. Tests that pass on unmodified code are rejected as inadequate without a litmus run; orphan tests are removed or remapped before handoff. A deterministic helper validates each record before handoff, tasks failing adequacy return to the TDD cycle within the existing refine and fix bounds, and the records ride step-output evidence plus ledger links so Step 5 scores observed adequacy instead of asserted coverage. Fix mode applies the same review to anti-regression tests.

Backend [edit]

Ledgers and artifacts include ac_ledger.cjs with alias filtering, skipReason handling, known-defect cap 8, and missing-evidence cap 9. Adequacy records attach per acceptance criterion via link --adequacy-file (multi-task histories accumulate; the latest record governs scoring), and an inadequate adequacy status fails closed through the known-defect cap while criteria without adequacy score as before. Testing emits step-07-*.testing.{plan,report}.md with Mutation status passed|failed|skipped. Audit logs use {us-dir}/audit-{slug}-{timestamp}.log.md. Benchmark evolution stores under benchmarks/results/ with telemetry in metrics.json.

Configuration spans verification.mutationTest, verification.mutationThreshold (default 80), defaults.skipMutationTesting (default true), fable.enabled, fable.autoAudit, fable.autoDetectDomain, fable.auditVerdictsBlockShip, optional localReviewCommand, and stack profiles under ws-shared/stacks/ consumed by scan_stack_invariants.cjs. Review fix reports carry defectClass, sourcesConsulted, proactiveFixed, and proactiveSkipped fields for convergence auditing.