Diagnostics & Benchmarks (harness)

> Provenance: .agents/skills/ws-doctor/SKILL.md, .agents/skills/ws-check-harness/SKILL.md, .agents/skills/ws-check-workflows/SKILL.md, .agents/skills/ws-benchmarks/SKILL.md, .agents/skills/ws-monitor/SKILL.md, living synthesis of specs 0003, 0012, 0014, 0019, 0020, 0026, 0030, 0036, 0039, 0041, 0056, 0059, 0064, 0088, 0096-us-348, 0100-us-356, 0107-us-369, 0116-us-385, 0121-us-395, 0125-us-412-413, 0128-us-412-418, 0129-us-419, 0136-skill-loader-host-capability, 0138-workflow-parallel-writer-compat, 0144-us-440, 0146-us-436, 0149-us-455, 0151-us-458, 0158-benchmark-publish-fixed-models, 0161-us-478, 0162-us-477, 0163-us-476, 0166-us-473, 0168-us-464, 0171-us-497-link-integrity.

Feature [edit]

Harness health is observable without running a full delivery. ws-doctor provides a read-only diagnostic inspector with optional --skill scoping and a machine-parseable --json stdout contract. ws-check-harness runs Phases 0 through 5c as an integrity auditor over routing, links, portability, digests, duplication, and topology. ws-check-workflows simulates FSM step continuity, state isolation, and provider dispatch. The audit starts with a deterministic Phase 0 detector that classifies install mode (upstream | consumer | none) and install scope (upstream | project | hybrid | global | none) with coexistence evidence when a machine-global install sits beside the upstream source tree, and the upstream tree holds a clean-audit invariant where a deterministic run at the package root reports zero findings. ws-benchmarks manages frozen-fixture evolution reports on explicit invoke only. Cleanup pairs two skills: ws-spec-archive harvests shipped plan folders into {specsDir}/index.PRD Archive before proposing their removal, and ws-cleanup lists disposable workflow leftovers (telemetry, .runtime, shipped plan dirs, untracked orphans) behind a user-gate, deleting only user-approved untracked paths. An opt-in runtime audit wrapper observes deliveries and proposes upstream issues without mutating consumer trees. Host capability binding probes the active host once per workflow and caches portable tool aliases.

How it works [edit]

ws-doctor edits nothing. It runs path, script, and reference checks even when project config is missing, reporting config-unavailable status and recommending ws-configure-project. It does not replace ws-check-harness phases or ws-show-harness inventory. The doctor.js --json flag emits exactly one JSON object on stdout; warnings and usage text go to stderr. A skill-local ESM marker keeps copied installs loadable without modifying root package.json. The path scanner mirrors real hub/installer resolution (spec 0167-us-469): repo-root-relative citations resolve against the project root before the citing file, Markdown links are validated by their href rather than backticked display text, fenced blocks and prose placeholders/shell redirects/home paths are skipped, {skillsRoot} citations honor a {globalSkillsRoot} alternative, a hub file citing its own directory expands once, archived run/example trees are excluded, and only unresolved install citations are reported so genuine broken paths stay visible. ws-version prints the install snapshot — scope, loaded skill directory, canonical package version, stored path tokens — read-only with no writes or network calls. ws-check-workflows asserts autoMode ON (unattended internal checkpoints plus the canonical stop list) and OFF (per-step or gateGranularity) behavior.

Duplicated normative blocks fail ws-check-harness. Path recipes resolve the consumer hub from --repo-root or cwd probe, never from __file__ inside a managed copy. Token expansion keeps {skillsRoot} local-first and {sharedDir} independent. Effective defaults.autoload is false unless explicitly true; when true, a consumer root AGENTS.md must exist and delegate to autoload.md, otherwise the harness fails closed. The installer never creates root agent index files.

Install-mode coexistence is evidence rather than a finding, because the global tree stays installer-managed while the hub invoke-versus-edit rule governs body reads. The link and path gate rejects broken internal links, author-machine absolute paths, declared tokens inside markdown link targets, bare ws-shared/ shorthand outside rule text, and unrouted ws-* skills, and the clean-audit test runs the detector plus the mechanical gates and integrity check together, printing a zero-findings proof that CI also runs non-blocking on main with a published report artifact.

Link-target classification is scope-aware, so the gate fails only on links that are genuinely broken in the layout it audits. Both hub directories come from the resolved consumer context (the relocated project hub and the resolved skills install's ws-shared hub) and are never assumed to be the literal .ws path; the install scope and hub presence are consumed as Phase 0 evidence, and the gate adds no detection of its own. While the resolved scope is global and the project hub is absent, a target that resolves to a file directly inside a resolved hub directory and names a hub binding or routing file (AGENTS.md, autoload.md, config.json, STACK.md) is recorded in an informational installLayoutNotes bucket with a ws-configure-project remediation pointer, and the absent hub is recorded in warnings. Neither bucket counts toward the five finding buckets that drive total and ok, so a notes-only audit exits 0 while the notes stay visible on the human path in both outcomes. The tolerance is deliberately narrow and never a prefix rule on .ws/... or ws-shared/... link text: a target below the hub binding level keeps failing even when its basename matches a binding file, and a runtime contract sibling the installer left unrewritten is a genuine broken link in every layout. Benchmarks never auto-run. ws-benchmarks acts only on explicit invoke behind a six-action gate where View Evolution Report is recommended. Static benchmark runs are fully scripted with no product writes; live runs stay sandboxed with dryRun and no push or PR. Regressions fail comparison against frozen tables. The audit wrapper adds zero overhead when defaults.enableAuditing is false; end-of-run findings propose an upstream issue through user-gate and never auto-file or block completion alone. Host binding runs one probe per workflow, caches results in gitignored host-capabilities.json, and keeps contracts brand-neutral with zero modal gates in autoMode. Capability detection goes further than a raw tool list: common native tools (file read, file write and update, shell exec, subagent dispatch) are detected once at session or workflow start, mapped to a portable capability-token vocabulary, and cached per host shape so later steps reuse the cache instead of re-probing. Agents query the cached capabilities before choosing how to act, which keeps them on native tools instead of reinventing shell equivalents and avoids whole classes of quoting and escaping failures. The repository ships pre-mapped model and tool names for the most common hosts, including dispatch-agent variants, so new hosts resolve fast, and probe counts over multi-step runs prove the cache holds. An eighth token, skillLoader, covers loading skill bodies: when bound to a host-native skill loader the agent uses it (including its already-loaded query), and when it resolves to none the agent reads {skillsRoot}/ws-<id>/SKILL.md through {readFile} under local-first resolution. One canonical procedure in host-capability-tokens.md replaces every divergent skill-load wording across shipped bodies and hubs.

Phase 5a carries two further fail-closed gates. check_skill_load.cjs rejects restated raw "Read the SKILL.md" recipes that bypass {skillLoader}, while plain Markdown cross-links stay allowed. The gate deliberately exempts prose that states what to load rather than how, so in-session body-load sites were converted by hand to the token-centered shape: each calls {skillLoader} and links the canonical procedure instead of restating it, covering the lifecycle, spec-manager, spec-write, code-review, ship-pr, and goal-fix-pr bodies while dispatch-agent orchestration (the subagent loads its own body) and script-level delegation (no body read) stay unchanged. check_git_ownership.cjs scans fenced command blocks and skill scripts tree-wide for broad staging (git add -A, git add ., bare git add -u, directory adds) and destructive whole-tree verbs (git reset --hard, git clean -fd, whole-tree git stash, force-push), reporting them as critical, so consumer audits surface parallel-writer hazards and not only the upstream test suite.

ws-monitor correlates live runs with host transcripts through built-in adapters for the known agent hosts, all behind the existing opt-in discovery flag so default runs never touch host stores. Each workflow snapshot reports its transcript source (adapter name plus location class) or transcript-unavailable with a reason, sessions correlate to workflows by workspace directory, branch, and run window without hand-supplied ids, and reads stay strictly read-only with immutable opens, bounded to a recent activity window plus an error-pattern scan. Tokens, prompt content, host-private paths, and credentials are sanitized before anything reaches a report. Worker-session liveness is a first-class signal: a worker idle while the queue shows in_progress reads as stall evidence, and transcript error parts feed the existing subagent-error and stall classifications. Expected artifacts are pipeline-aware: the monitor reads the workflowType discriminator and applies the lite step map (spec, plan, implement, review, ship, fix-pr) to lite runs instead of the standard map, so healthy lite runs no longer collect false critical missing-artifact findings for interview, DAG, verify, and testing files they never owe. Terminal-state hygiene is a monitor finding family as well: runs whose steps are all terminal but whose status never closed, batch children that finished while the parent row stayed in_progress, and queues holding stale or superseded rows all surface in read-only snapshots instead of polling forever as live signals. Artifact expectations follow step membership. expectedArtifacts requires a file only when that step is in completedSteps or stepStatus: completed. The currentStep >= N watermark is only a fallback when step lists are absent. A fix-PR-only shape (completedSteps 8 and 9, empty plan dir) yields zero critical missing-artifact findings. Terminal runs with companion-evidence drift (a refined plan present, no separate interview file) downgrade instead of going critical. A live run that advanced past verification without artifacts is still flagged.

Transcript discovery reads the correlated session inside one tick. Enumeration correlates on the path before filling a name-sorted per-root slice, so a matching file that sorts last is still scanned and capped stays honest when the bound truncates. The scan filter and resolveTranscriptSource share one correlation window, so a file the scan counts resolves as available. A dispatched run carries an agentTranscripts marker: known host paths, or an explicit absent reason (discovery-disabled, no-matching-session, or scan-capped). When --open-issue is set, stall and defect findings can open a tracker item through the provider create-issue intent. The body stays anonymized.

Monitor liveness findings are scoped to keep normal orchestration windows quiet. Discovery selects runs on the state-derived slug independently of plan folder or layout, with --workflow-id authoritative and an unmatched slug returning zero. Context-mismatch branch severity keys off the literal run status (completed, cancelled, failed, superseded, stopped): terminal runs report info naming the status while every other status stays critical, and shape-derived terminal handling remains informational only. The session-source stopwatch re-reads the correlated session file each tick, prefers the state-recorded driver session over an unrelated --session-id root, and suppresses the worker-session-stall finding to info while the state or telemetry clock advances, marking stale references and reporting the resolved session reference. Stale-parent-row detection carries a child-terminal-age grace defaulting to --stall-window: a fresh child close inside the grace reads as info propagation-pending while the run advances, and only an older unpropagated close warns, with both measured ages carried on the finding. Issue-proposal bodies are anonymized by construction: session identifiers are redacted, the reproduction command shows a --session-id <redacted> placeholder, the summary carries units, and the checklist includes a truthful body-anonymized row.

Fixed-model comparisons are published, never executed, through publish_comparison.cjs. The publisher validates a run directory holding frozen prd.md, a binary-check judge.json, and a run-manifest.json: both frozen bytes must match their manifest sha256 pins, every column needs at least 3 samples, models stay uniform per role, results stay strictly binary, and comparator defaults stay pinned unless a protocolExceptions entry records the deviation. Passing runs write the report plus JSON twin under benchmarks/results/ and append to the ## Fixed-Model Comparison Runs comparison section.

Backend [edit]

Diagnostic tooling centers on doctor.js with resolveCitedPath, explicit python|node|bash recipe checks, and syntax probes via py_compile, Node parsing, or bash -n. Harness auditing adds detect_install_mode.cjs for the Phase 0 install-mode and scope classification and check_harness_links.cjs for the link, path, shorthand, and routing gate, with test-harness-clean.js enforcing the upstream zero-findings invariant. Workflow integrity uses validate_state.cjs. Benchmark management runs through benchmarks_manager.cjs with flags such as --check, --list, --evolution, --update-comparison, --compare, and --snapshot. Audit drafts emit through audit_log.js. Monitor expectations are pinned by pipeline-specific suites (test-ws-monitor-us385.js for lite artifacts, test-ws-monitor-us388.js for child-state expectations, test-ws-monitor-us395.js plus test-terminal-close-us395.js for terminal retirement, test-ws-monitor-us464.js for slug-scoped discovery, test-ws-monitor-us473.js for literal-status severity, test-ws-monitor-us476.js for the propagation grace, test-ws-monitor-us478.js for proposal sanitization) so the standard map, the lite map, and the multi-spec finding family stay consistent with the snapshot builder. Fixed-model comparison publishing is pinned by publish_comparison.cjs --run <runDir> gate coverage.

Reports land in benchmarks/results/BENCHMARK_EVOLUTION.md plus versioned table-<version>.md files with score, time, tokens, model, and verdict columns governed by report.schema.json dimensions. Per-run audit logs use {us-dir}/audit-{slug}-{timestamp}.log.md. Configuration keys include defaults.enableAuditing, defaults.autoload, model preset bundles, defaults.hostAdapter.mode, context budget limits, plan.index.json, and run.json.