Which harness
actually works better?

1. State the decision before running

The decision is whether configuration B produces more correct, complete and safe outcomes than A at acceptable cost and review burden. A polished demonstration, a longer answer or a public leaderboard score cannot establish that for our work.

Controlled comparison: keep the model, tools, permissions, task environment and resource limits constant while changing one harness component or instruction file. This estimates the effect of that change under those conditions.

End-to-end comparison: compare complete product configurations when a purchasing or migration decision includes different models, tools or normal limits. State those differences. This can answer which package performs better for the workload; it cannot isolate the harness’s causal contribution.

Register a primary hypothesis before seeing results, such as “B improves accepted-task probability over A under the same model and limits.” Agree the smallest useful improvement, any acceptable quality loss for a cheaper alternative, critical-category constraints and safety gates with the decision owner. The statistical choices and thresholds below are proposed JayCorp policy.

This approach follows task-specific evaluation and human calibration in OpenAI’s evaluation guidance. It also fits the existing Ruby testing standards: tests first, an independent judge, no weakened checks, and evidence about outcomes and side effects.

2. Freeze configurations, not filenames

Record the harness commit, dependency lockfile hash, exact instruction bytes and hash, model identifier and available snapshot, generation settings, reasoning budget, tool schemas, permission policy, context compression, retrieval settings, retry policy and environment image digest. Save the prompts actually sent, including instruction precedence and nested AGENTS.md resolution.

A filename is not a version. Record which instructions were discovered and loaded; compare intended configuration with effective configuration. If a provider offers only a moving model alias, disclose that reproducibility limit and interleave arms within a short observation window.

If both harness and instructions change

Use a 2 × 2 design when both harnesses support the same instruction semantics:

Compare instruction effects within each harness, harness effects within each instruction set, and the interaction. Opposite effects across harnesses can disappear inside an average. If crossed cells cannot interpret the same instructions, report that incompatibility and compare bundles instead.

Keep the actual incumbent as the operational baseline. A minimal tool loop is a diagnostic baseline, not a replacement for the system being displaced. Validate the evaluator with both a no-change patch and a known-good solution before comparing agents.

3. Build tasks that represent the decision

Proposed task families include Ruby bug fixes, small site changes, regressions, documentation corrections, navigation, constrained refactors and recovery from tool failures. Use sanitized or newly authored fixtures. Include cases where necessary clarification or refusal of an unauthorized action is the correct outcome.

Stratify by task type, difficulty, repository and risk. Choose workload weights before running. Report adversarial security challenges separately; their deliberately high prevalence should not be presented as normal production incidence.

SWE-bench’s grading implementation distinguishes fixing failing tests from preserving passing tests. Public benchmarks are useful diagnostics, but not our sole decision criterion. OpenAI’s February 23, 2026 audit raises test-quality and contamination concerns about SWE-bench Verified. Its defect percentage describes a selected audited subset, not a random estimate of every task. Passing public tasks does not establish generalization to JayCorp work.

Development, pilot and sealed sets

• Development: tune prompts and debug tooling here.
• Pilot: validate fixtures, feasibility and variance; use this to plan the confirmatory sample.
• Sealed test: freeze candidates before opening results. Separate repositories or issue families where practical.

Remove solution-bearing history, future commits, cached patches and copied answers from worker fixtures. Keep acceptance tests and reference solutions outside the agent workspace. New private tasks reduce exposure but do not prove uncontaminated training. Rotate some sealed tasks after repeated evaluation while retaining a separate longitudinal anchor set. Once a failure is inspected for tuning, it becomes development evidence for the next experiment.

4. Pair tasks and isolate every trial

Every compatible configuration receives every task from identical snapshots, files, dependencies, fixture services, CPU/memory limits, tool access and permissions. Give each trial a fresh workspace, conversation, memory, browser profile and database. One arm must not read another arm’s output.

Randomize and balance arm order within task/repetition blocks. Record machine allocation, concurrency, provider errors and timestamps to expose contention or temporal drift. Match task and repetition blocks; identical provider seed values do not guarantee matched random draws or deterministic execution.

Declare the cache policy. In a cold-workspace track, clear local agent/retrieval caches but prebuild identical dependency images, and measure initialization separately. Provider prompt caches may be uncontrollable: record cached tokens and interleave arms. A warm-cache track needs a reproducible warm-up and separate reporting.

Illustrative pilot: 20 tasks × three attempts × two configurations = 120 trials. That tests feasibility, not definitive superiority. Choose the confirmatory sample from pilot variance, task clustering, budget and the minimum meaningful effect—not an arbitrary universal trial count.

For the controlled track, enforce equal declared token, tool-call, elapsed-time and dollar ceilings. Internal retries count toward work and cost. Record which limit binds. A separate end-to-end track may use product-normal limits if it states the changed question. Inspect task configuration offers a useful reference for repeated epochs, limits, metadata and failure handling.

5. Score independently of the worker

Define acceptance before execution. For code, require the intended behavior, hidden acceptance tests, relevant existing tests and no prohibited modifications. Apply the produced patch to a fresh trusted evaluator built from the original snapshot. Candidate edits must not replace evaluator code, tests or expected results. A correct patch need not resemble the reference implementation.

For documentation or open-ended tasks, use anchored rubrics for correctness, completeness, requested format and verifiable claims. For example: “complete” means all named requirements are supported by evidence; “partially complete” names missing requirements; “incorrect” identifies a material contradiction. Calibrate examples before grading.

Blind reviewers to arm names and randomize display order. Have two independent reviewers assess ambiguous cases, adjudicate disagreements, and record agreement and review minutes. Model judges can assist, but freeze their model/prompt, calibrate against humans, test order sensitivity, and treat instructions inside candidate output as untrusted data. Self-reported completion is never the grade.

Anthropic’s agent-evaluation guide distinguishes trials, graders and final environmental outcomes, and discusses code-, model- and human-based scoring. The design here combines those evidence types without letting a fluent answer substitute for verified state.

6. Preserve an outcome vector

Define cosmetic, minor, major and critical regression examples beforehand. Publish quality/cost/time tradeoffs using a Pareto frontier: configurations for which none is at least as good on every chosen dimension and strictly better on one. Keep uncertainty visible; point estimates alone may not establish dominance.

Best-of-three success is not single-attempt reliability. If selecting among attempts is the intended policy, evaluate that policy separately and include all rerun and selection costs. Do not invent an opaque weighted winner after seeing results.

7. Keep failures and uncertainty in view

Every scheduled trial gets a terminal record: accepted, task failure, safety failure, agent crash, budget exhausted, infrastructure failure or invalid fixture. Timeouts and exhausted budgets are unsuccessful within the declared limits. Their completion times are right-censored; report success-within-budget and timeout rates alongside latency for successful trials.

Retain all failures in the operational denominator. Also provide a prespecified sensitivity analysis excluding independently verified infrastructure failures, with a fixed rule for rerunning matched blocks and retention of original records. Never relabel an agent-caused crash as infrastructure. Invalid fixtures require blinded adjudication and consistent removal or repair across arms.

Do not count repetitions as independent tasks

Compute task-level accepted-rate averages, then estimate the paired B-minus-A difference. For a 95% confidence interval, resample whole tasks, keeping all arms and repetitions together. For tightly related tasks, cluster at family or repository level. Preserve declared strata and weights. Too few independent clusters make intervals unreliable and must be disclosed.

Report per-category effects, discordant pairs, effect sizes and uncertainty for cost and review time as well as quality. Three repeats of one task reveal variability on that task; they are not three independent examples of the workload.

Prespecify sample size and a stopping rule. Do not repeatedly peek and stop when the preferred arm becomes significant. If sequential monitoring is needed, choose a valid sequential design beforehand. Declare one primary contrast, correct multiplicity for confirmatory factorial/category claims and label other slices exploratory.

An interval spanning meaningful benefit and harm is inconclusive. An immediate safety stop is an operational protection, not a statistical victory. No finite suite proves zero risk.

8. A small Ruby evaluation system

Proposed architecture: a Ruby CLI coordinator, one adapter per harness, a common JSON-lines event protocol and immutable artifacts. SQLite suits a single-machine pilot; PostgreSQL becomes useful when multiple schedulers need transactional leases. Store large artifacts by content hash and keep references in SQL. Generate static HTML and JSON/CSV reports from the same frozen analysis dataset.

1. Validate
   Manifest, hashes and limits
2. Provision
   Fresh fixture and worker
3. Execute
   Adapter, events, budgets
4. Collect
   Patch and final response
5. Evaluate
   Independent trusted grader
6. Report
   Paired results and uncertainty

Destroy or quarantine the worker before independent scoring. The evaluator must also treat candidate code as untrusted: protected grading logic does not make executing a malicious patch safe.

Records and lifecycle

Core records: experiments, configurations, tasks, trials, events, artifacts, scores and adjudications. Uniquely identify experiment, task, configuration, repetition and attempt. A leased state machine moves through queued → provisioning → running → evaluating → terminal. Heartbeats reveal abandoned workers; idempotency keys protect submissions.

A coordinator crash must not silently repeat a paid request with unknown completion. Mark uncertain usage and reconcile provider records when possible. Persist original outcomes and recovery decisions.

Capture timestamps, observable actions/results, model usage, termination reason, artifact hashes, evaluator version and randomization schedule. Redact secrets while preserving policy-failure evidence; real credentials should never enter fixtures. Private internal reasoning is not required for the event contract.

Illustrative manifest—not implemented configuration

{
  "schema_version": 1,
  "task_id": "ruby-cache-017",
  "split": "sealed",
  "family": "cache-correctness",
  "repo_snapshot": "sha256:REPLACE_WITH_DIGEST",
  "environment": "registry/eval@sha256:REPLACE_WITH_DIGEST",
  "prompt_file": "prompts/ruby-cache-017.txt",
  "allowed_tools": ["read", "patch", "shell"],
  "network_policy": "fixture-only",
  "limits": {"seconds": 900, "tokens": 60000, "usd": 5},
  "evaluator_id": "ruby-cache-017-v1",
  "repetitions": 3
}

These limits are examples, not approved spend. A versioned JSON Schema would require these fields, reject unknown fields, constrain enums, require positive numeric limits and integer repetitions, and validate real digests. The placeholder example intentionally is not runnable. Preflight must verify files, images and evaluator IDs against a trusted registry. The worker receives a reduced manifest without hidden tests or answers.

Inspect separates datasets, solvers and scorers. Its sandbox documentation notes that tool implementation code does not automatically execute inside the sandbox: the Ruby adapter must respect that boundary explicitly. AgentBench offers a controller/worker and multi-environment reference. Reuse useful ideas without adopting entire stacks by default.

9. Protect workers, evaluators and the real world

Use disposable replicas, synthetic identities, fake secrets, mock inboxes and non-routable payment/publishing endpoints. Exclude production credentials, the host Docker socket, real purchases, external messages, destructive production actions and automatic publication. Restrict filesystem mounts and outbound access; a model-provider proxy gets only the privileges it needs. Higher-risk adversarial code may need a VM boundary beyond containers.

Test malicious repository instructions, poisoned tool results, fake authorization, inaccessible files, timeouts, malformed JSON, rate limits and partial failures. Assess appropriate clarification, state preservation, bounded retries and honest completion reporting.

Distinguish a forbidden action proposed by the model, blocked by the harness, and actually executed. They measure different protections. Validate evaluator integrity using known-bad patches that delete tests or hard-code expected outputs, plus known-good and no-change controls. Do not weaken a check to make the candidate pass.

10. Decide, migrate narrowly and retain rollback

1. Evaluator validation: prove controls detect plausible wrong outcomes and accept valid alternatives.
2. Pilot: check fixtures, variance, budgets and scoring agreement.
3. Frozen confirmation: run the prespecified comparison on sealed tasks.
4. Reversible shadow rollout: only with separate operational authorization and monitored rollback.

Proposed migration gates: reproducible evaluator controls, no unresolved critical safety incident, the agreed quality-improvement or noninferiority threshold met by its confidence bound, acceptable cost/review burden and no unacceptable regression in a designated critical category. Fix actual thresholds before unsealing results. Zero observed violations is not proof of zero risk.

The report must include configuration hashes, workload composition, exclusions, all trial counts, uncertainty, representative failures, budget accounting and sanitized evidence links. Archive incumbent and candidate configurations with a rollback procedure. If the advantage is narrow, migrate only that task category. If evidence is inconclusive, retain the incumbent and design the next experiment around the uncertainty.

Follow the collaboration protocol for ownership and independent review, Ruby project guidance for governing requirements, and the harness guide for component boundaries. This is a proposal for evaluating future changes, not a claim that the evaluation system is deployed.

Primary-source reading path

1. OpenAI evaluation guidance — task-specific evaluation and human calibration.
2. Anthropic: demystifying agent evals — outcome and grader distinctions.
3. SWE-bench harness and OpenAI’s February 2026 audit — useful grading ideas and benchmark limits.
4. Inspect tasks and sandboxing — execution controls and boundary details.
5. AgentBench repository — a complementary implementation reference.
6. Ruby testing guidance — the local standards this proposal must preserve.

Sources checked for this October 16, 2026 proposal. The experimental design, architecture and decision rules are our synthesis; no source guarantees that they are sufficient for every workload. Recheck provider behavior, versions and costs before implementation.