Full interactive dashboards
This landing page holds the paper's own tables and statistics (family-wise Friedman and Nemenyi tests that include Random and Historical Mid-Range) and the paper figures.
What this is
Each benchmark task is treated as a test case and each agent as the system under test. A selection strategy orders the tasks before the agent runs; only a prefix of that order (10%, 20% or 50% of the tasks) is executed. We measure two things: how far the Success Rate on that prefix is from the Success Rate on the whole benchmark (SccErr, lower is better), and how many of the agent's failures the prefix contains (failures found, higher is better).
Strategy families
How the numbers are formed
- Every table value is the median across target agents. For randomized strategies each agent contributes the mean over its 100 runs; deterministic strategies have a single run.
- Statistics (Friedman, Nemenyi) are computed on the per-agent values, with benchmark–agent pairs as blocks. The Historical Mid-Range baseline orders tasks by how close other agents' success rate is to 50% (leave-one-agent-out) and is tested together with each family and Random.
- The judge-based strategies use the mixed-rubric configuration: coding rubric for the SWE-bench family, USACO and TerminalBench, general rubric for the others (CORE-Bench, TAU-bench, GAIA, Online-Mind2Web). Two of the Sonnet 5 judgment sets, the five-factor judgments for SWE-bench Verified and Mini, come from an earlier run whose exact model version is not recorded (see the README). GAIA uses a common set of 164 tasks.
- Everything on this page comes from
paper_results/andresults/in the replication package; nothing is typed in by hand.
Median (IQR) across agents
Highlighted = best median in that benchmark column (ties at the displayed precision share it). Underlined = better than Random. Dagger rows: SWE-bench Mini is not independent of SWE-bench Verified. The non-judge rows are identical for both judges.
Judge sensitivity: the four judge strategies, side by side
Friedman and Nemenyi tests
Mean rank: 1 = best (ranks are computed within each benchmark–agent block, then averaged). Each family is tested together with Random and Historical Mid-Range. Two strategies whose mean ranks differ by less than the critical difference (CD) are not significantly different. The matrix below lists every pair: red = significant at α = 0.05.
Best group per benchmark (20% budget)
Check mark = statistically indistinguishable from the best strategy in that benchmark (Nemenyi, per benchmark).
Every agent, every strategy
Rows are target agents, sorted by true Success Rate (shown in the first column). Cell shading: greener = better within that agent's row. Values are per-agent means over the 100 runs for randomized strategies.
Cost saving of running only a prefix
Budgets of 10%, 20% and 50% are shares of the tasks executed, not of the dollar cost, so strategies at the same task budget can cost different amounts. Cost saving = 100 × (1 − cost of the selected prefix / cost of the whole benchmark), per agent, summarised by the median over agents. Per-task dollar cost exists only for SWE-bench Verified and Multilingual.