Gareth PaulEvaluation notes

Independent work / Developer platforms

A closer look at
agent evaluations.

I study changes across APIs, SDKs, CLIs, tests, and documentation—and whether the checks used to evaluate them measure what we think they do.

I have spent twenty years building developer platforms at Google, Apple, Twitter, Twilio, and Zoom. Background and approach

On this page

  1. A finding, examined
  2. Historical model results
  3. API results · GPT-6 Astra
  4. The next experiments

01 / Research note

Exploratory pilot ·

Grader validity / GPT-6 Astra

What a passing score leaves out

21 of the 30 grader passes have an optional added-test failure recorded. The aggregate preserves that distinction.

The recovered rule checks test contributions and a task-specific replay or mutation. These recorded signals do not establish test quality or general model reliability.

Sample
40 attempts / 10 synthetic tasks
Protocol
One response; no tools or retries
Author
Gareth Paul
Read the research note Aggregate data
Figure 0140 attempts · one cohort
21/30

grader passes had an optional-check failure recorded

30 passed the grader. Of those, 21 have a recorded added-test check failure. All 39 gradeable receipts contain an explicit optional-check result; the invalid output is ungraded.

Same 40 attempts. Automated checks, not a human readiness review.

  • Grader + optional check passed9
  • Grader passed; optional failure recorded21
  • Grader failed9
  • Invalid output1
9 of 40 attempts passed the grader and the optional check, verified from archived receipts. Each square is one attempt, grouped by recorded outcome, not run order.
Observation
An optional failure is recorded for 21 of 30 grader passes.
Interpretation
The recovered grader excludes this diagnostic from its pass decision. Archived receipts confirm nine passes on both checks; this does not establish test adequacy.
Not established
Production readiness or reliability on new tasks. Independent human review has not been completed.

Evaluation tools and methods

Can a coding agent ship a developer-platform change? Can the grader tell?

Inspect the harness and grader

Exploratory evaluation tools and methods; prospective results pending, including the two held-out tasks.

Built so far

12 synthetic platform tasks
Each spans an API, generated SDKs, a CLI, tests, and documentation. Ten shaped the grader; two are held out for the prospective study.
120 recorded pilot attempts
Three public model configurations, two reasoning settings each, graded post hoc. Historical exploratory evidence, not a comparative ranking.
64 + 95 acceptable alternatives and seeded faults
Development controls the grader had to accept or reject before it was frozen. Diagnostics, not an accuracy estimate.
0 / 12 · 0 / 3 false accepts · false rejects
Fifteen constructed control patches rerun against the frozen grader: 0 of 12 faulty references accepted and 0 of 3 acceptable references rejected. Trusted-local execution, not containment evidence or validation on unseen submissions.
220 files frozen, digest-published grader boundary
The historical score contract and control diagnostics publish their frozen boundary digest. The separate API download now binds to its recovered source fingerprint and method. These identify source versions, not proof of individual runs.
Every byte of this site is inspected before publish
The build enumerates and scans each exported file, including image chunks, compressed streams, and encoded text, and fails closed.

The recorded control check uses authored controls, not agent-generated submissions. No independent unseen validation case or human label is available.

How I evaluate

These principles guide the planned studies; the historical exploratory pilot was graded post hoc.

  1. Grader validity comes before model comparison.

    A score is only as trustworthy as the thing producing it. I audit the grader with known-good patches and seeded defects before I let it rank anything.

  2. Pre-register, then run.

    Tasks, conditions, repetitions, timeouts, and the primary outcome are frozen and published as JSON before collection. Scored failures stay in the denominator. Pre-agent evaluator or host-infrastructure failures are recorded separately and excluded from scored outcome denominators.

  3. A release gate, not an average.

    The prospective policy calls for test deletion, grader tampering, out-of-scope edits, and unsafe operations to block an attempt. Its successor grader must be frozen and verified before collection; historical scope penalties remain unchanged.

  4. Corrections are versioned, never silent.

    When a grading rule turned out to be wrong, the affected runs were excluded and the change was recorded in a public correction history rather than rescored quietly.

02–03 / Recorded results

Two protocols · not directly comparable

Historical pilot

Three models. The same ten tasks.

10 synthetic tasks. 120 attempts. Exploratory, not a controlled comparison.

  • Ready
  • Needs review
  • Not ready
  • GPT-5.6 Sol
    40 attempts · 2 settings

    26 ready · 8 review · 6 not ready

  • GPT-5.6 Luna
    40 attempts · 2 settings

    15 ready · 6 review · 19 not ready

  • GPT-5.6 Terra
    40 attempts · 2 settings

    19 ready · 6 review · 15 not ready

Exploratory, with post-hoc grading and two repetitions per task and setting. These are recorded outcomes, not a controlled comparison or independent human review. Read the pilot’s limits

Separate API cohort

API patch pilot

10 tasks. 40 attempts. 2 repetitions per task and setting.

Separate cohort, not directly comparable to historical agent runs. One response per attempt; no tools or test execution by the model. Patches are separately graded in a sandbox.

  • Checks passed
  • Checks failed
  • Invalid output
  • GPT-6 Astra
    40 attempts · 2 settings

    Checks passed: 30/40 · Checks failed: 9/40 · Invalid output: 1/40

04 / Research agenda

Planned work, not completed evidence

The next experiments

Prospective study · Results pending

Agent DX Eval

Test whether an agent carries a developer-platform change consistently across the API, SDKs, CLI, tests, and documentation.

Next: freeze task selection and grading before a controlled rerun. The historical pilot is exploratory evidence only.

Planned audit · Results pending

Grader Audit

Test whether the grader distinguishes a correct change from a convincing but broken one, using known-good patches and seeded defects.

Next: audit grading decisions against adversarial controls and independent human review.

Working together

My focus is evaluation design for coding agents and developer platforms. For more about my background or to discuss this work:

Methodology only

GolfBench

A separate next-shot planning example using deterministic checks and Monte Carlo simulation. Results remain withheld pending a controlled rerun; this is not evidence for either study.