What the featured strategies do, how to use the downloads, and what the results mean.
These are readable summaries, not drop-in implementations. Details such as feature extraction, tied quantile boundaries, and cluster traversal are preserved in the exported orderings. No target-agent outcomes are used to build the featured task-only orderings. Strategy winners themselves are selected retrospectively from measured results.
Represent different kinds of tasks before exhausting any single category.
Partitions combine the benchmark category/repository, traceback and exception cues, and a document-length bin. Tasks within a category are ordered by decreasing length, then task ID; categories are visited in decreasing size, with deterministic tie handling.
categories ← group tasks by (category, traceback, exception, length_bin)
for each category:
sort its tasks by descending text length, then task ID
sort categories by descending size, then category key
while tasks remain:
take the next task from each nonempty category
return first round(budget × task_count) tasks
Adaptive Random
Spread the selection across the available task-embedding space.
Despite its name, this implementation does not learn from target-agent outcomes. It normalizes cached task embeddings, starts from a seeded random task, and greedily chooses the task least similar to its nearest selected neighbour.
normalize each task embedding
selected ← [random task using seed 0]
while unselected tasks remain:
next ← task minimizing its maximum cosine similarity to selected
append next to selected
return selected prefix
Boundary Value
Visit tasks with unusually short or long descriptions first.
The implemented score is the absolute distance from the median document length. This is a text-length heuristic, not a general search for software input boundaries. Saved orderings preserve the original tie resolution.
lengths ← lengths of task descriptions and hints
center ← median(lengths)
score(task) ← abs(length(task) − center)
order ← tasks sorted by descending score
return order prefix
Task length
Use description length as a simple task-side signal.
Sort descriptions plus hints from longest to shortest, breaking ties by task ID. This requires no LLM call.
score(task) ← character length(description + newline + hints)
order ← sort tasks by descending score, then task ID
return order prefix
TF-IDF Diversity (Original)
Select tasks that are lexically different from those already chosen.
Uses descriptions plus hints, TF-IDF word unigrams/bigrams, English stop-word removal, sublinear term frequency, and a maximum vocabulary of 5,000 features. After a seeded random first task, each task maximizes its minimum cosine distance from the selected tasks.
X ← normalized TF-IDF(description + hints)
selected ← [random task using seed 0]
while unselected tasks remain:
next ← argmax over remaining tasks of min cosine_distance(task, selected)
append next to selected
return selected prefix
LLM-as-Judge
Ask a judge for one overall task-difficulty score.
Use the recorded single-dimension judgment and sort hardest first, with task-ID tie-breaking. No new model requests are made when downloading these subsets.
difficulty ← recorded overall LLM difficulty score for each task
order ← sort tasks by descending difficulty, then task ID
return order prefix
Factor-Based LLM-as-Judge
Combine five difficulty dimensions into a task score.
For these coding benchmarks, the factors are localization hardness, code-change size, reasoning hardness, test complexity, and reproducibility hardness. Standardize each dimension within the benchmark and sum them with equal weights. A zero-variance dimension contributes zero. Historical Verified/Mini five-factor judgments have author-reported model provenance; the exact model version was not recorded in this package.
for each of the five recorded dimensions:
z[dimension] ← standardize scores within this benchmark
if variance is zero: z[dimension] ← 0
score(task) ← sum of the five standardized scores
order ← sort tasks by descending score, then task ID
return order prefix
Factor-Based LLM-as-Judge with Diversity
Combine difficult-first ordering with coverage across task clusters.
Use the five-factor composite, divide tasks into four quantile bands, and visit bands hardest first. Within each band, cycle through task clusters. This judge variant uses seeded TF-IDF/SVD-based k-means; its saved seed-0 ordering is downloadable.
score ← five-factor standardized composite
clusters ← seeded k-means on task representation
bands ← four quantile bands of score
order ← empty list
for band from hardest to easiest:
append round-robin traversal of clusters within band
return order prefix
Difficulty Score
Estimate difficulty from task features without an LLM judge.
The portable implementation combines five standardized feature groups: localization cues, paths/code blocks/text length, text entropy and length, requirement/output mentions, and reproducibility cues. These are hand-designed proxies, not LLM ratings or proven causes of failure.
features ← extract text-based difficulty cues
factors ← weighted, standardized feature combinations
score(task) ← sum of standardized factors
order ← sort tasks by descending score, then task ID
return order prefix
Difficulty Score with Diversity
Spread selection across clusters inside difficult-first bands.
Uses the same non-LLM composite as Difficulty Score. It groups scores into four quantile bands and cycles through cached task clusters within each band, from hardest to easiest. Its clustering source differs from the factor-based judge diversity variant.
score ← non-LLM difficulty composite
bands ← four quantile bands of score
order ← empty list
for band from hardest to easiest:
append round-robin traversal of cached clusters within band
return order prefix
Choose your evaluation budget.
Return to the comparison to download the corresponding subsets.