Can Large Language Models Forecast What Researchers Study Next?

Large language models increasingly generate research ideas, yet judging their novelty or feasibility at generation time does not establish whether they anticipate subsequent work. We introduce IdeaForecastBench to evaluate research idea forecasting. Given a community's literature up to a cutoff, a system produces up to five ranked ideas, which are evaluated against later papers. The benchmark comprises 624 rolling episodes across 52 topics, with a fixed retrieve-then-judge protocol and separately reported results from two judges. We compare five history-compression strategies across GPT-4.1, Qwen2.5-7B/14B, and Qwen3.5-9B, together with a learned Mode-Decomposition Forecaster (MDF). Under the primary GPT-4.1-mini judge, Summary improves Direct in Hit@5 and Precision@5 across all four backbones. Qwen2.5 scores above GPT-4.1, whereas Qwen3.5 scores below it. An outcome-blind assessment finds that Qwen2.5 produces broader forecasts, but does not identify how much breadth contributes to its advantage. Threshold and judge diagnostics further clarify the limits of interpreting realization as precise anticipation. IdeaForecastBench provides a common task for studying which research ideas a community subsequently pursues and how reliably this outcome can be measured.

624

rolling episodes

52

research topics

12

monthly cutoffs

21

configurations

2

judges, reported separately

Prior work on research ideation mostly judges an idea at the moment it is written — is it novel, feasible, promising? — or executes selected proposals one at a time. Neither settles whether an idea anticipates what a field goes on to do. We ask a different question: given a community's literature up to a cutoff t, can a system name the ideas that community subsequently pursues?

An episode is a topic–cutoff pair, not a single pre-specified idea. The forecaster observes the history X≤t — every paper of the topic first submitted on or before t — and returns K = 5 ranked ideas in natural language, each naming a problem, a rationale, an approach and optional key terms. The targets Y_t are the papers of that topic first submitted after t through the end of month t+3. Nothing after the cutoff is visible as input.

What the task measures is deliberately narrow. A forecast is realized when a retrieved paper satisfies the idea–paper matching rubric. That measures whether subsequent work is consistent with the forecast — not the forecaster's ability to execute the idea, not the idea's scientific value, and not verbatim agreement with an abstract.

Corpus. We collect arXiv machine-learning papers with cat:cs.ML, retaining identifiers, titles, abstracts, submission dates and category metadata, and merging repeated results and cross-listings by identifier. Papers are split by first-submission date, so later revisions never move a paper into the target pool.

Communities. Fixed names, aliases and keyword rules assign papers to 52 overlapping topics — retrieval-augmented generation, reinforcement learning, medical imaging, and so on. A paper can belong to several topics but appears once within each, and topic definitions stay fixed across methods and cutoffs.

Rolling windows. The corpus spans April 2024 to September 2025. Twelve monthly cutoffs from July 2024 through June 2025 yield 52 × 12 = 624 episodes. History pools contain 33 to 4,530 papers (mean 589.9); target pools contain 35 to 1,722 (mean 313.3). All 21 configurations share one episode manifest — the same 624 topic–cutoff keys, endpoints and pool sizes — so a release can be verified by comparing paper identifiers, not just counts.

Candidate retrieval. Each forecast and each target paper is embedded with voyage-3-large (1024 dimensions), and the top R = 10 papers by cosine similarity become candidates. This evaluation-time funnel is separate from any retrieval a method performs on the history side; similar terminology alone does not establish realization, and the judge makes that decision.

Idea–paper matching. The judge sees the forecast's title, rationale, approach and key terms beside a candidate's title and abstract, then scores Problem P, Method M and Specificity S from 0 to 3 with a short rationale. The default gate is P + M ≥ 5 ∧ S ≥ 2: close problem and method agreement, plus at least partial realization of the core idea.

Crediting. In rank order, each prediction takes credit from the first passing candidate not already credited in that episode, so every paper supports at most one idea and missing outputs receive no credit.

Two judges, reported separately. GPT-4.1-mini is primary; Qwen3.5-9B evaluates the same predictions independently. We never ensemble their scores — keeping them apart is what lets us distinguish execution failures from judgment differences.

Metrics. Hit@5 asks whether any of the five forecasts was realized; Precision@5 measures how many of the five were; MRR records how early the first credited idea appears. Novelty — embedding distance from the closest visible historical paper — is reported separately as a diagnostic and never folded into a composite score, because vague forecasts can be distant from prior work while useful extensions remain close.

With the history and target pools fixed, what remains is how a forecaster selects and organizes evidence. We organize five prompting strategies by what they retain from history, and compare complete pipelines without equalizing token or compute budgets.

A learned reference: MDF. The Mode-Decomposition Forecaster introduces a structured intermediate representation of historical innovations. It represents an idea as a typed triple z = (b, o, g) — a base direction, an operator, and a target gap — over an inventory of EXTEND, TRANSFER, COMPOSE, BENCHMARK, ANALYZE, SIMPLIFY, SCALE and ADAPT. A typed memory stores innovations with their frequency, recency and utility; a prior predicts the triple and a realization policy converts it into a grounded idea. Without matched ablations, MDF's score cannot separate the contributions of the prior, the memory, and the reinforcement learning — we present it as a trainable reference, not a tuned competitor.

Summary leads in realization frequency. It has the highest Hit@5 and Precision@5 point estimates for every backbone under both judges. Its primary-judge Hit@5 gains over Direct are 0.269, 0.378, 0.338 and 0.306. Memory and Retrieval also improve on Direct. These comparisons measure gains from complete pipelines; they do not isolate abstraction from context selection or from additional model calls.

Realization frequency, yield and rank differ. Qwen2.5-14B Summary reaches Hit@5 0.954 under the primary judge, but Precision@5 is 0.553. Almost every episode contains a credited idea, yet only about half the five-slot budget receives distinct-paper credit — which is exactly why Precision@5 stays informative once Hit@5 approaches its ceiling.

Compression is not uniformly beneficial. Topic Trend exceeds Direct in Hit@5 on all four backbones, but has lower Precision@5 on GPT-4.1 and Qwen2.5. It also underfills the output budget — five slots filled in 60.1–60.7% of windows on GPT-4.1 and Qwen2.5, and only 12.7% on Qwen3.5, where the other strategies fill at least 98.9%. Its ranking performance cannot be read as idea quality alone.

Backbone effects are large but not self-explanatory. Qwen2.5 exceeds GPT-4.1 on each strategy, but Qwen3.5 does not share that advantage. The paired GPT-4.1-versus-Qwen2.5-14B Summary difference is −0.224 (95% CI [−0.274, −0.173]). The Qwen2.5 advantage therefore does not extend uniformly across the Qwen family, and these scores do not rank general model capability.

Qwen2.5 is consistently broader. An outcome-blind study samples 832 forecasts — one per topic across the 15 GPT-4.1/Qwen2.5 configurations and MDF — and rates them without candidate papers, outcomes, backbone labels or strategy labels. On Summary, GPT-4.1 scores 3.58 (95% CI [3.15, 4.00]) against 6.58 for Qwen2.5-7B and 6.48 for 14B. Forecast-level match rates rise from 0.205 in the lowest generality bin to 0.406 in the highest, with a correlation of 0.17. Broader forecasts and higher realization scores coexist; this study cannot estimate how much of the gap is breadth.

A stricter gate is a sensitivity analysis. Raising S ≥ 2 to S ≥ 3 moves Qwen2.5-7B's paired Hit@5 advantage from +0.192 to +0.163 (95% CI [0.120, 0.207]), while the 14B advantage holds at +0.181. The advantage persists under the stricter rule, but that does not validate S as a measure of intrinsic specificity.

Agreement is not accuracy. Absolute levels move under the second judge: GPT-4.1 Summary Hit@5 rises from 0.756 to 0.822, Qwen3.5 Summary falls from 0.532 to 0.498, and MDF falls from 0.545 to 0.296. Candidate-level binary agreement is 0.950 and 0.894 with Cohen's κ of 0.538 and 0.588 — but most candidate pairs are negatives, so high raw agreement neither establishes correctness nor rules out shared semantic biases.

Representation loss is not generation failure. A field audit of all 3,120 MDF forecasts reports a median approach length of four words and empty key-term lists. The prediction adapter constructs the approach as operator:base_direction, leaves key terms at their default, and truncates the rationale to 500 characters. Conversion can produce sparse evaluation fields even when the raw output contains a method description — evaluation has to preserve information from generated text to scored idea.

These results show that the representation of historical literature affects how often forecasts align with later papers, but do not by themselves establish precise anticipation of novel scientific contributions. We interpret them as evidence of realization under a specified matching protocol, while leaving precise, novel anticipation unresolved.

• Automated matching requires further calibration. Agreement between GPT-4.1-mini and Qwen3.5-9B does not establish correctness. The auxiliary human study concerns an earlier Qwen-judged experiment and cannot calibrate the current scores.
• Judge comparisons remain sensitive to execution and measurement. Qwen-judge results include execution failures, among them all-candidate failures in 72 of 624 MDF windows. Until those are recovered, score differences cannot be attributed solely to judge behavior.
• The gate, retrieval depth, and cluster count are operating conventions. The protocol retrieves ten candidates and applies P + M ≥ 5, S ≥ 2. Raising the gate to S ≥ 3 tests matching sensitivity without controlling intrinsic specificity. These choices provide no absolute scale of scientific originality or completeness.
• The fixed snapshot does not eliminate pretraining exposure. Filtering historical input cannot exclude prior exposure to target papers. The contamination probe is an observational temporal comparison, not a causal estimate of memorization. A prospective evaluation would freeze forecasts before collecting their target literature.
• The benchmark covers selected machine-learning communities. The corpus covers arXiv cs.ML and 52 overlapping topics. Publications provide an incomplete, delayed record of research; unmatched ideas may be realized later or elsewhere.