Claude Sonnet 5 (mixed rubric) — results at 100 runs

← Paper tables, tests and figures

Test selection for AI agents: each benchmark task is a test case and each agent the system under test. Built from the per-agent results of the 100-run experiments in the replication package (results/historical_mid_range_100x_all_ten/ and the saved orderings).

This dashboard. Claude Sonnet 5 is the judge of the four LLM-judge strategies, with the coding rubric for the SWE-bench family, USACO and TerminalBench and the general rubric for CORE-Bench, TAU-bench, GAIA and Online-Mind2Web (the mixed-rubric configuration used in the paper). Two of the judgment sets, the five-factor judgments for SWE-bench Verified and Mini, come from an earlier run whose exact model version is not recorded (see the README). Randomized strategies use 100 runs; budgets of 10/20/50% are shares of the tasks, not of the dollar cost; GAIA is evaluated on one common set of 164 tasks for every strategy. The tables, pooled panels and curves of this app rank the fourteen strategies together (as in the original dashboard); Historical Mid-Range, an external baseline, is shown as an extra row. The paper’s family-wise tests, which include Random and Historical Mid-Range, are on the landing page.

Methodology. For randomized techniques, metrics are averaged over 100 runs separately for each target agent. Deterministic techniques contribute one value per agent. Tables and descriptive plots report the median across agents within each benchmark; statistical comparisons use the per-agent values before that median aggregation.

Mean-over-runs (a per-agent number, computed once per agent before anything else) is not the same thing as median-over-agents (what every table cell below displays) — and neither is the same as a statistical mean rank (what the significance panels display: the average rank a strategy earns across matched agents, not an average of the metric's raw value).

Agent-level blocks within one benchmark share that benchmark's task set and are therefore not fully independent replications of "the benchmark" — Friedman/Nemenyi still applies (it only uses within-block ranks), but a large per-benchmark N should not be read as N independent experiments.

Budget selection applies to the table above (for Failure detection / Accuracy estimation) and to the pooled CD diagrams in the Statistical comparison section below; the APFD table and its CD panel have no budget dimension.

USACO note: USACO has no genuine 5-dimension or single-dimension LLM judgment covering all tasks. Cells marked † in the "Judge" rows for USACO are NOT LLM judgments — they are substituted with Metadata Difficulty (derived from the public contest division: bronze/silver/gold/platinum), per friedman_nemenyi.JUDGE_ALIAS / run_cross_benchmark.JUDGE_LABEL. A genuine but partial LLM judgment for USACO also exists in the raw data under the same generic strategy names — it is intentionally excluded from this canonical comparison and is not shown here; see MANIFEST.json's usaco_dual_difficulty_source field for the full trace.

Statistical comparison

Failure detection

Accuracy estimation

APFD (whole order — not budget-dependent)

Area under the Prefix Fault-detection Curve over the entire task ordering. This is a separate statistic from the prefix-based failure-detection comparison above; it is shown here as its own panel and is never substituted for the Overall failure-detection result.

Per-benchmark detail (supplementary)

An independent Friedman + Nemenyi test run separately within each benchmark (one block per matched agent, so N is much smaller than the pooled panels above). This is a diagnostic breakdown, not a replacement for the pooled comparison.

Sensitivity: excluding SWE-bench Mini from the pooled panels

SWE-bench Mini's 50 tasks overlap SWE-bench Verified, so it is not an independent task suite. This reruns every pooled panel above (at the currently selected budget) with Mini's blocks excluded and reports whether the top-ranked strategy changes.

Curves over the whole budget range

Instead of three snapshots at 10%, 20% and 50%, these curves follow every budget from 0 to 100% of a benchmark's tasks. Each curve is the mean over target agents (and over the 100 runs of randomized strategies). Hover a panel to read values; click a panel to enlarge it. The dotted vertical lines mark the paper's 10%, 20% and 50% budgets.

Objective

Curves

Benchmark

Budgets shown are the three RELATIVE budgets actually used by the raw pipeline — 10%, 20%, 50% of each benchmark's own task count — never the absolute-task-count budgets (k10abs/k25abs/k50abs) also present in the raw CSV, which are a different axis and are not shown as a percentage here. Complete-case matching: an agent is only included in a statistical test if it has a valid value under every one of the fourteen strategies being compared. Generated by extension/rebuild_dashboard_data_from_raw.py from results/raw_runs/median_from_raw.csv, apfd_median_from_raw.csv, stats_per_benchmark.csv, and stats_overall.csv; the curves from results/dashboard_100x/sonnet5/dash_data.json (pipeline/historical_mid_range_100x/build_dashboard_data.py).