Gareth Paul

Why evaluations

Every developer platform I have run had the same failure mode: a change passes its local tests and still breaks a client, an SDK, or a documented contract. Coding agents make that change cheap to attempt and expensive to trust. The interesting question is not whether an agent can pass a test suite. It is whether it can carry one change across every surface a developer touches, and whether the evaluation catches it when it cannot.

That is why this portfolio has two studies, kept deliberately separate. Agent DX Eval measures the agent. Grader Audit measures the measurement. Neither prospective study has empirical results yet, and the site says so on every page, because a protocol download is a promise about method, not proof of execution.

What I built for this work

• A synthetic developer platform with twelve tasks, each spanning an API, generated SDKs, a command-line interface, tests, and documentation, so a single change can be graded on every surface at once.
• A historical grader with a weighted score and a separate release gate. The historical score contract and control diagnostics publish a 220-file boundary digest. The separate API download binds to its recovered source fingerprint and method. These source bindings do not authenticate individual run receipts. The prospective blocking policy must be verified in a frozen successor; it does not replace historical scope penalties or change recorded outcomes.
• Development controls of 64 acceptable alternatives and 95 seeded faults, then a fifteen-control rerun against the frozen grader, reported with denominators and the limits of trusted-local execution.
• Pre-registered protocols, a readable score contract, a sanitized worked example, and a versioned correction history, all downloadable as JSON from the study pages.
• A publication verifier that enumerates and inspects every exported byte of this site before it can be deployed, so private identifiers cannot leak through markup, downloads, or image metadata.

Background

2025 – presentPoe

Product Manager, CreatorsDeveloper experience for the Poe API: onboarding, documentation, key management, and access to more than a hundred models through a chat-completions-compatible API used by popular coding agents.

2023 – 2025Zoom

Product and Research Lead, AI Applied ResearchShipped AI Companion, speech services, and applied machine-learning products to the entire user base. Learned first-hand that AI products live or die by their evaluations.

2020 – 2023Twilio

Head of Developer InterfacesLed developer tooling for millions of developers. Open-sourced the OpenAPI specification and built the pipeline that generates SDKs across the supported languages.

2016 – 2020Foursquare

Senior Director of EngineeringRan the developer platform and its monetization.

2013 – 2016Twitter

Developer platform and ecosystemBuilt the developer ecosystem programs and platform partnerships.

2005 – 2013Google, then Apple

Developer platform foundationsEight years of platform and developer-facing work at web and mobile scale.

What is public

The publication policy is aggregate only; approved public names. The home-page chart uses three explicitly approved display names, checked against official model documentation. Only per-model outcome counts are projected from the historical pilot. Raw model slugs, private identifiers, provider mappings, and unapproved names stay unpublished.

Original results, trajectories, configurations, and repository links are intentionally omitted. Withholding those artifacts limits independent reproduction; public display names do not turn exploratory work into controlled evidence. GolfBench remains a short methodology-only note on the overview.

Reading the status labels

Historical record

Recorded pilot counts, with the original sampling and grading limitations.

Prospective study / Planned audit

A design for collecting evidence. A protocol download is not proof of execution.

Results pending

No empirical model comparison or independent human audit to report.