BENCHMARK TASK SELECTION FOR AI AGENTS

Run fewer tasks.
Learn what matters.

TRY A SMALLER RUNIllustrative · 100 tasks

20of 100 tasks

Explore subsets at 20% ↗

Find your task subset ↗See the measured savings ↓

MEASURED EXAMPLESWE-bench Verified

100 / 500 tasksA 20% task budget. Two different goals.

ESTIMATE SUCCESS RATE

—%

lower execution cost than a full run

— percentage points median error

FIND FAILURES EARLIER

—×

as many failures as Random at the same task budget

Estimate your agent’s
success rate.

Uncover its
failures earlier.

RESEARCH THROUGH COLLABORATION

TU Delft × JetBrains

LessTasks4Agents is the result of a collaboration between TU Delft and JetBrains, bringing academic research and industrial needs together to make AI agent evaluation more efficient.

Meet the team ↗

THE STUDY AT A GLANCE

Broad evidence. Clear comparisons.

Inspect the statistical comparisons ↗

10benchmark settings

15selection approaches

100runs per randomized strategy

THE PAYOFF · EXPLORE THE RECORDED RESULTS

Spend less.
Keep your goal in focus.

Two goals. Three budgets. Measured execution costs on Verified and Multilingual.

Choose a benchmark

Task budget

20% of tasks evaluated

How to read these results

Costs are median shares of recorded full-suite execution cost; ordering and judging costs are excluded. Performance values are medians across agents, after averaging randomized runs within each agent. Failure uplift is a ratio of these medians, not a mean per-agent improvement. A partial evaluation does not provide full-suite coverage. Winners are retrospective; they are not guaranteed to win on a new benchmark.

Explore the full cost and results tables ↗

DOWNLOAD · ALL TEN BENCHMARKS

Go from evidence
to a task list.

Choose your benchmark, approach, and budget. Download your task subset.

Choose what to run

Compare results. Download task IDs. No account required.

Choose & download a subset ↗

See who ranks where

Compare agent rankings at each budget.

Explore agents & rankings ↗

Includes all ten benchmark settings. Cost comparisons above cover only Verified and Multilingual. Historical Mid-Range downloads are target-specific; the library asks which target agent you want to evaluate.

01 / THE RESEARCH

Choose your goal.
Then choose your strategy.

Two objectives. Different winning strategies.

How well does this agent perform?

Estimate the success rate of the complete benchmark from a smaller subset of tasks.

SUCCESS RATE ESTIMATIONLower error is better ↘

Compare estimation errors ↗

Where does this agent fail?

Prioritize tasks that reveal failures early, when only part of the benchmark can be evaluated.

EARLY FAILURE DETECTIONMore failures found is better ↗

Compare failure detection ↗

02 / THE METHOD

Three steps to a smaller run.

Prioritize tasks. Set a budget. Evaluate.

Choose a benchmark

Compare approaches on your benchmark.

Set your budget

Choose 10%, 20%, or 50% of tasks.

Download & evaluate

Download task IDs as CSV or JSON. Run them in your own harness.

How the study is evaluated

Recorded agent outcomes let us replay task orderings without rerunning the agents. Randomized strategies are averaged within each agent before results are summarized across agents. The dashboard provides the tables and statistical comparisons.

03 / BENCHMARK COVERAGE