Gareth PaulEvaluation systemsMain portfolio

PUBLIC EVALUATION PORTFOLIO01

Measure the system.
Not the demo.

I design reproducible evaluations that turn model behavior into launch decisions— with explicit contracts, controlled runs, distribution-aware baselines, and limitations that travel with every result.

Read the case study Inspect the method
  • Research PM
  • Core models
  • Model performance
eval-system.mapv1.0 / public
SanitizedComparable rerun pending
Version the taskControl the runMeasure the tailsName the limits
01 / METHOD

Operating thesis

Evaluation is product infrastructure.

A useful evaluation is not a score generator. It is a decision system: stable enough to detect change, legible enough to diagnose it, and honest enough to stop a launch.

01

Deterministic suites

Seed scenario generation. Version prompts and schemas. Make every grader and retry rule reproducible.

02

Monte Carlo baselines

When outcomes are stochastic, compare distributions—not anecdotes. Inspect expected performance and tail risk.

03

Launch readiness

Pre-register gates, segment failures, and pair every recommendation with uncertainty and known limitations.

02 / LOOP

From question to call

A traceable path from behavior to decision.

Each stage leaves an artifact that can be reviewed without exposing private traces or proprietary identifiers.

StageWorkDecision
01ContractDefine the user task, system boundary, response schema, and grading contract.What counts as success?
02GenerateCreate seeded fixtures, difficulty slices, perturbations, and known edge cases.Is coverage intentional?
03ExecutePin configuration, retry policy, tool access, and trace capture before a run.Can the run be repeated?
04CompareUse deterministic references and distribution-aware simulation where the world is stochastic.How far from acceptable?
05DecideApply launch gates to regressions, tail behavior, feasibility, and critical slices.Ship, hold, or investigate?
06PublishRelease sanitized methodology, coverage, limitations, and provenance.What can others trust?

SANITIZED CASE STUDY01

GolfBench

Strategic planning under uncertainty,
without performance theatre.

Comparable rerun pending

Results withheld pending a controlled rerun using active public pinned model snapshots. The rerun must use identical scenario coverage, versioned configuration, and reproducible evidence before any comparison is published.

01 The question

Can a system make one next-shot decision that balances progress, risk, and constraints?

Golf provides a bounded planning environment with hazards, wind, club constraints, and uncertain outcomes. The task is not golf trivia. It is selecting a feasible action whose expected outcome stays close to the GolfBench baseline planner as difficulty rises.

02 The design

Separate format compliance, decision feasibility, expected cost, and tail behavior.

Seeded scenarios feed a typed decision contract. Hard constraints are checked before a valid action enters repeated simulation. The resulting distribution is compared with the GolfBench baseline planner rather than a single hand-picked outcome.

MEASUREMENT STACK

One number cannot explain planning quality.

01

Regret

Expected cost above the GolfBench baseline planner.

02

Feasibility

Share of decisions that respect hard constraints.

03

Tail

High-percentile downside under repeated simulation.

04

Frontier

Highest difficulty that still clears the gate.

05

AUC

Area under the matched difficulty curve as a coverage-aware summary.

PUBLICATION CONTROL

What is safe to publish now.

Suite definition
Public Sanitized task and metric design
Methodology
Public Deterministic and simulation approach
Raw traces
Held Private outputs remain private
Model identifiers
Held No names published in this case study
Comparative results
Held Awaiting matched coverage
Future evidence identity
Schema Model owner, pinned public model ID, and Access route recorded separately

LIMITATIONS MEMO

What this evaluation cannot establish.

  1. 01A simulator is only as valid as its assumptions about the environment.
  2. 02The GolfBench baseline planner encodes choices that must be documented and challenged.
  3. 03Structured-output validity is necessary, but it is not evidence of task quality.
  4. 04Coverage gaps can make apparently comparable runs non-comparable.
  5. 05A bounded planning domain does not prove performance in open-ended real work.
03 / PRACTICE

Built to extend

A portfolio that grows by adding evidence.

Future studies use the same publication contract, so new work can be compared without erasing its context.

EVAL CARD / .MD

Evaluation contract

  • User task and boundary
  • Dataset and slice versions
  • Baselines and graders
  • Acceptance gates
  • Known limitations

LAUNCH MEMO / .PDF

Decision narrative

  • What changed and why
  • Critical regressions
  • Confidence and uncertainty
  • Ship, hold, or investigate
  • Required follow-up

RUN BUNDLE / .JSON

Reproduction trail

  • Suite and runner version
  • Configuration checksum
  • Seed registry
  • Coverage manifest
  • Sanitized summary only

RESEARCH × PRODUCT × SYSTEMSGP

Make capability claims falsifiable—and launch calls explainable.

I work at the boundary between research questions, product decisions, and the systems needed to keep both honest. The goal is not more evaluation activity. It is better evidence at the moment a team has to decide.

Continue to Gareth’s portfolio