Grader Audit

Validation package

The offline package prepares blinded review packets and computes aggregate reports from two reviewers and adjudication. Its frozen review rubric separates patch correctness from test adequacy. Development controls stay outside the held-out study.

Engineering checks use synthetic fixtures only. Authorized held-out case intake, independent human reviewers, and adjudication are pending. No human validation result has been produced. The two tasks reserved by the earlier study are not a completed patch set for this audit.

Download the frozen review rubric (JSON)

Recorded control check

The grader was exercised against 15 constructed patches: 3 acceptable references and 12 faulty references. These are authored controls, not agent-generated submissions.

False accepts

0 of 12 faulty references

False rejects

0 of 3 acceptable references

Independent human reviews

0

This check ran in a trusted local process, not an isolated sandbox. It is not a measure of real-world accuracy, model capability, or independent reviewer agreement. The small fault set and reference patches were developed together; agreement on them does not validate the grader on unseen submissions.

Download the control diagnostics (JSON)

The artifact includes a digest of the frozen 220-file evaluation boundary, aggregate counts, and limitations. It contains no patches, model identities, or private configuration mappings.

What the grader has and has not faced

During grader development, 64 acceptable alternatives and 95 named fault mutations were checked across 10 synthetic tasks. These cases helped build the rubric. They are development diagnostics, not unseen validation or an accuracy estimate.

No independent unseen validation case or human label is available. The documentation checker uses a bounded vocabulary and relation rules, so a valid paraphrase can still be rejected. That unresolved accuracy question remains open until independent adjudication tests representative alternatives and natural failures.

The question

A task score is useful only if the grader catches the failures that matter. This audit will test whether it accepts correct work, rejects faulty work, and gives reasons that agree with the underlying evidence.

Controls and measurement

• Freeze the grader, then keep its development controls out of final validation. Use unseen known-good patches or acceptable alternatives, naturally observed failures where permitted, and safe adversarial controls with seeded defects. Seed one defect at a time: a dropped authorization header, broken cursor handling, contract drift, or incorrect CLI output.
• Include unchanged baselines and adversarial controls for test deletion, grader tampering, and unsupported completion claims. Keep the starting fixture, execution environment, and frozen rubric the same within each comparison.
• Measure false acceptance of faulty patches and false rejection of correct patches. Publish a confusion matrix, denominators, uncertainty, and a breakdown by defect type. Keep grader errors and inconclusive runs separate from valid decisions.
• Repeat grading to check stability. Report decision changes and reason-level disagreement rather than assuming a consistent score means the decision is correct.

False acceptance uses adjudicated incorrect solutions as its denominator; false rejection uses adjudicated correct solutions. Reviewer disagreement uses independently double-reviewed cases. A missing denominator means null, not zero.

Human review, when available

Sample accepted and rejected attempts using a sampling rule recorded before review. The plan calls for blinded review against the same rubric, hiding identities and grader decisions. Report disagreement before adjudication and keep unresolved cases separate.

Keep grader-development cases separate from final validation; freeze the revised grader before validation. Reviewer recruitment, independent labels, and adjudication are pending. Automated controls are diagnostics, not independent human review.

Limits

Seeded defects are useful probes, but they cannot represent every naturally occurring mistake. Expected labels also depend on the quality of the control patches and rubric. This is an audit of the measurement process, not a comparative ranking of systems.

Historical pilot counts are not audit evidence. The recorded exploratory pilot used post-hoc grading and cannot establish the grader’s false acceptance or false rejection rate.

Inspectable protocol and artifacts

View the audit protocol (JSON) · Download

The recorded counts above are a readable artifact view. Future publication will contain aggregates and limitations, without original patches, traces, or identity mappings.

• Agent-submission results artifact: pending.
• Synthetic control diagnostics: recorded separately above.
• Human review artifact: pending.