Task subset library
Browse approaches in a table. Preview the selected tasks, then export CSV or JSON. Seed 0 is used for randomized methods.
Choose & download a subset ↗ TEST SELECTION FOR AI AGENTS
Evaluate more.
Run less.
Every task costs time. Make the ones you evaluate count.
We study how to choose benchmark tasks for two different goals: estimating an agent’s success rate and discovering its failures earlier.
An empirical study of budget-aware agent evaluation
THE EVALUATION BUDGETILLUSTRATIVE EXAMPLE
A full suite.
A smaller first step.
20/ 100 tasksChoose a task budget
Evaluate first Remaining tasks
The strategy sets the order. Your budget sets where to stop. This illustration does not show measured savings.
10benchmark settings
2evaluation objectives
100runs per randomized strategy
Inspect the statistical comparisons ↗A SMALLER BUDGET. A CLEARER PURPOSE.
The best strategy.
For the question you ask.
The best strategies for each objective, at your selected task budget. Winners are selected by median performance in the Sonnet mixed-rubric results.
Task budget20% of tasks evaluated
Costs are median shares of recorded full-suite execution cost; ordering and judging costs are excluded. Performance values are medians across agents, after averaging randomized runs within each agent. Failure uplift is a ratio of these medians, not a mean per-agent improvement. A partial evaluation does not provide full-suite coverage. Winners are retrospective; they are not guaranteed to win on a new benchmark.
Explore the full cost and results tables ↗DOWNLOAD · ALL TEN BENCHMARKS
Your benchmark.
Your approach. Your subset.
Choose any of the 15 approaches, select a 10%, 20%, or 50% budget, and download ordered task IDs with configuration metadata.
Agents & leaderboards
Explore the agents in each benchmark. Compare rankings at each budget with the full-suite ranking.
Explore agents & rankings ↗Includes all ten benchmark settings. Cost comparisons above cover only Verified and Multilingual. Historical Mid-Range downloads are target-specific; the library asks which target agent you want to evaluate.
01 / THE RESEARCH
One budget.
Two different questions.
Finding failures and estimating overall performance are different evaluation needs. A useful selection strategy depends on which question you want to answer.
How well does this agent perform?
Estimate the success rate of the complete benchmark from a smaller subset of tasks.
SUCCESS RATE ESTIMATIONLower error is better ↘
Explore how black-box strategies compare with difficulty-based selection and Random.
Compare estimation errors ↗Where does this agent fail?
Prioritize tasks that reveal failures early, when only part of the benchmark can be evaluated.
EARLY FAILURE DETECTIONMore failures found is better ↗
Explore how task-difficulty signals, including LLM judges, change which failures appear first.
Compare failure detection ↗02 / THE METHOD
Order. Select. Evaluate.
Tasks are prioritized before evaluating the target agent. The selected prefix determines the evaluation budget.
Describe the tasks
Use information available before execution: task text, metadata, or estimated difficulty.
Build an ordering
Compare baselines, adapted black-box regression techniques, and task-side difficulty estimation.
Evaluate a prefix
Measure success-rate estimation error and failures found at 10%, 20%, and 50% task budgets.
03 / BENCHMARK COVERAGE
Beyond a single kind of task.
Ten benchmark settings span coding, terminal use, tool interactions, research, and web-based tasks.
01SWE-bench VerifiedIssue resolution
02SWE-bench MiniIssue resolution · Verified subset
03SWE-bench MultilingualMultilingual coding
04SWE-bench ProSoftware engineering
05USACOProgramming problems
06TerminalBench 2.0Terminal tasks
07TAU-bench AirlineTool-use dialogue
08CORE-Bench HardResearch reproduction
09GAIAGeneral assistant tasks
10Online-Mind2WebWeb navigation
Benchmark settings are not all independent task suites: SWE-bench Mini overlaps with SWE-bench Verified.
LOOK BEYOND THE HEADLINE
Explore the evidence.
Compare strategies, change the budget, and inspect results across benchmarks and judge configurations.
Open the research dashboard ↗