ALL TEN BENCHMARK SETTINGS
Download task subsets
Choose your benchmark, strategy, and budget. Download the exact ordered task IDs to use with your evaluation harness.
Loading recorded data…
Task IDs only, not task descriptions or solutions. Randomized methods use fixed seed 0. The study's averaged results do not describe this one realization exactly. Use the benchmark provider's task content and evaluation harness.
Compare approaches at this budget
| Approach | Family | Subset type | Tasks | Failures found ↑ median (IQR) | SccErr ↓ percentage points · median (IQR) |
|---|
Selected task preview
| Priority | Task ID |
|---|