What a passing score leaves out

Question

What does a grader pass leave out? The optional check examines a test contribution; it is not a measure of coverage or test adequacy.

Method

Model: GPT-6 Astra. The API patch pilot covered 40 attempts: 10 synthetic tasks × 2 settings × 2 repetitions per task-setting. Each attempt used one response, with no model tools or retries. Patches were graded separately in isolation.

The grader recorded 30 passed, 9 failed, and 1 invalid output. These are attempt counts, not counts of distinct tasks.

What the optional check measures

For nine tasks, the check requires a recognized test declaration or assertion added in an expected public test file. The changed hunk must match task-specific concept groups and pass source-inspection restrictions. Candidate tests are replayed with non-test changes reversed to the baseline. The check passes if a replay exits unsuccessfully and its output names an active candidate test. Candidate-suite regression checks are separate mandatory checks. The retry task also applies a type-compatibility patch to the baseline.

Nullable completion times uses a different rule: recognize a test of a nullable job value in changed hunk context, require both candidate public suites to pass, then change the null fixture value in a disposable copy. A named qualifying test must fail under that mutation. These bounded syntax and output checks do not establish arbitrary test quality. The nullable assertion may already exist in unchanged hunk context; a newly added assertion is not required by that rule.

A returned check is either passed or failed. No contribution, irrelevant changes, missing expected files, or no qualifying replay/mutation failure produce a failed check. Some inspection or execution errors also become failed checks; uncaught execution errors instead abort grading. Missing, skipped, and unavailable are not distinct states in this producer’s check output. An absent receipt is not a pass or a measured failure.

Overall success requires no detected tampering and all non-test-behavior assertions to pass. The API wrapper also verifies candidate identity and tampering. This is the recovered API rule, not the historical weighted score rubric.

Interpretation

21 of the 30 grader passes have an optional added-test failure recorded. The denominator is grader passes, not all 40 attempts. The 9 overall failures and 1 invalid output stay in the overall attempt denominator, but outside this conditional count. Failing the optional check does not block a grader pass.

Archived receipts directly record 9 passes on both checks. All 39 gradeable attempts have one explicit optional result: 9 passed and 30 failed; none are missing or skipped. The invalid output has no grader receipt and stays ungraded. The result is not production readiness or an independent assessment of test adequacy.

Limits

• Repeated attempts are not independent tasks. The 40 attempts revisit 10 synthetic tasks, not 40 distinct tasks or a sample of production work.
• No human review has been completed. The counts describe recorded checks, not an independent assessment of patch or test quality.
• This aggregate-only release cannot reproduce individual patches and cannot establish general model reliability.
• The source boundary was recovered and its fingerprint matched the stored snapshot. Archived receipts, candidate digests, and task counts were reconciled to the frozen evidence hash. This verifies existing records; execution was not rerun. The private receipts are not downloadable, which limits independent public reproduction.
• No comparison to the historical agent cohort is supported by this separate single-response study.

Evidence

The download contains the same public projection used by this note: outcome counts and task-level summaries and a reviewed source-boundary binding, not individual patches or transcripts. A source digest identifies code; it does not authenticate individual runs.

Download the exact method, version binding, and state definitions (JSON)

Download the aggregate results (JSON)