ESTIMATE SUCCESS RATE
—%lower execution cost than a full run
— percentage points median error
MEASURED EXAMPLESWE-bench Verified
100 / 500 tasksA 20% task budget. Two different goals.
FIND FAILURES EARLIER
—×as many failures as Random at the same task budget
Different strategies serve different goals. Retrospective medians; execution costs exclude ordering and judging. Partial evaluation leaves tasks untested.
Change the benchmark or budget ↓10benchmark settings
15selection approaches
100runs per randomized strategy
Inspect the statistical comparisons ↗
THE PAYOFF · EXPLORE THE RECORDED RESULTS
Spend less.
Keep your goal in focus.
See the best observed approach for each goal. Change the budget to compare execution cost, estimation error, and failures found. Results below cover the two benchmarks with per-task cost data.
Task budget20% of tasks evaluated
Costs are median shares of recorded full-suite execution cost; ordering and judging costs are excluded. Performance values are medians across agents, after averaging randomized runs within each agent. Failure uplift is a ratio of these medians, not a mean per-agent improvement. A partial evaluation does not provide full-suite coverage. Winners are retrospective; they are not guaranteed to win on a new benchmark.
Explore the full cost and results tables ↗DOWNLOAD · ALL TEN BENCHMARKS
Go from evidence
to a task list.
Choose any of the 15 approaches, select a 10%, 20%, or 50% budget, and download ordered task IDs with configuration metadata.
Choose what to run
Compare failure detection and SccErr, choose an approach, and download the ordered task IDs. No account required.
Choose & download a subset ↗See who ranks where
Explore the agents in each benchmark. Compare rankings at each budget with the full-suite ranking.
Explore agents & rankings ↗Includes all ten benchmark settings. Cost comparisons above cover only Verified and Multilingual. Historical Mid-Range downloads are target-specific; the library asks which target agent you want to evaluate.
01 / THE RESEARCH
Choose your goal.
Then choose your strategy.
Finding failures and estimating overall performance are different evaluation needs. A useful selection strategy depends on which question you want to answer.
How well does this agent perform?
Estimate the success rate of the complete benchmark from a smaller subset of tasks.
SUCCESS RATE ESTIMATIONLower error is better ↘
Explore how black-box strategies compare with difficulty-based selection and Random.
Compare estimation errors ↗Where does this agent fail?
Prioritize tasks that reveal failures early, when only part of the benchmark can be evaluated.
EARLY FAILURE DETECTIONMore failures found is better ↗
Explore how task-difficulty signals, including LLM judges, change which failures appear first.
Compare failure detection ↗02 / THE METHOD
Three steps to a smaller run.
Tasks are prioritized before evaluating the target agent. The selected prefix determines the evaluation budget.
Choose a benchmark
Find your evaluation setting in the subset library. Compare recorded results before choosing a strategy.
Set your budget
Select 10%, 20%, or 50% of tasks. See the trade-off between estimation error and failure detection.
Download & evaluate
Export task IDs as CSV or JSON, obtain the tasks from the benchmark provider, and run them in your own harness.
03 / BENCHMARK COVERAGE
Beyond a single kind of task.
Ten benchmark settings span coding, terminal use, tool interactions, research, and web-based tasks.
SWE-bench VerifiedIssue resolutionOfficial website ↗SWE-bench Mini50-task Verified subsetParent benchmark ↗SWE-bench MultilingualMultilingual codingOfficial dataset ↗SWE ProSWE-bench ProSoftware engineeringOfficial repository ↗
USACOProgramming problemsOfficial website ↗
TerminalBench 2.0Terminal tasks2.0 leaderboard ↗τ-benchTAU-bench AirlineTool-use dialogueOfficial repository ↗CORECORE-Bench HardResearch reproductionOfficial repository ↗
GAIAGeneral assistant tasksOfficial dataset ↗Mind2WebOnline-Mind2WebWeb navigationOfficial repository ↗Benchmark settings are not all independent task suites: SWE-bench Mini overlaps with SWE-bench Verified.
MAKE YOUR NEXT EVALUATION COUNT
Your next run can be smaller.
Choose your benchmark, compare the approaches, and download the task list that fits your goal.
Choose your task subset ↗