AI Can Write Research Papers. It Can't Tell You Where the Ideas Came From.
A team of graduate students at Shanghai Jiao Tong University recently asked 14 different AI systems a simple question: given two scientific papers, can you tell which one built on the other? The best AI got it right 27.3% of the time. A coin flip would have done about as well.
Background & Context
AI research assistants have been on a tear lately. Systems like AI Scientist, ResearchAgent, and various LLM-powered tools can search the literature, run experiments, and produce paper-like reports that pass cursory inspection. Benchmarks have mostly focused on whether these systems retrieve the right papers, write fluently, and produce ideas that human judges find interesting. Those are real achievements.
But there is a harder question nobody was asking: when a system claims a new idea extends prior work, does it actually understand what it is inheriting? Scientific progress is not just about finding papers with similar keywords. Two papers can share a topic while having zero intellectual lineage. YOLO and DETR both do object detection, but one keeps the anchor mechanism and the other throws it out entirely. Some papers look textually distant but carry the same core mechanism forward. A model that cannot tell the difference is, from a scientific standpoint, just guessing.
What the Researchers Did
The team, spread across seven institutions including Carnegie Mellon and Microsoft, built a framework called IdeaGene. The metaphor is explicit: scientific ideas have “genomes.” Each paper gets broken down not into keywords or embeddings, but into typed, evidence-grounded units: a mechanism here, a limitation there, a delta (or repair) for each problem the work addresses. These Idea Genome objects are minimal enough that a single one can be inherited, mutated, or lost across successive papers.
From there, the researchers constructed GenomeDiff records: pairwise alignments between predecessor and successor papers that track what was inherited, what mutated, what was imported from outside, and what was lost. Six operational dynamics classify each transition: Mutation (same niche, modified mechanism), Adaptive Radiation (mechanism moves to a new domain), Hybridization (importing from two distinct lineages), Speciation (same niche, entirely new mechanism), Niche Competition (shared ecology, no inheritance), and Isolation (no connection at all).
The dataset is substantial. Across 10 scientific domains (from protein design to fusion plasma control), they assembled 1,961 golden lineage traces, 1,085 curated Idea Genome objects, and 920 ground-truth GenomeDiff records. Fifty graduate annotators validated the labels, reaching 84.7% inter-annotator agreement before adjudication. This is not a toy dataset scraped from citation graphs; every lineage trace was audited by someone who actually knows the field.
The benchmark has two parts. IG-Exam is a closed-form test with 42 task types and 1,029 instances across four axes: extracting Idea Genome objects from papers, tracing which ones survive across papers, classifying evolutionary dynamics, and verifying whether a claimed lineage is coherent. IG-Arena is more open-ended: systems write research proposals under three information settings (question only, paper library, or structured lineage), and proposals are scored on a Population-Evolution Score that measures Heredity (did it inherit the right ideas?), Variation (does it differ meaningfully from nearby work?), and Selection (does it open paths for future research?).
What They Found
Fourteen systems were tested: direct LLMs, research-agent frameworks like AI Scientist v2 and CoI-Agent, and CLI harnesses that pair language models with coding tools. The headline number is stark. Across all 1,029 IG-Exam instances, the strongest system (GPT-5.5 paired with Claude Code) reached only 27.3% exact-match accuracy. Most systems clustered between 10% and 20%.
| System | Overall Accuracy | T1 (Abstraction) | T2 (Tracing) | T3 (Evolution) | T4 (Verification) |
|---|---|---|---|---|---|
| GPT-5.5 + Claude Code | 27.3% | 31.5% | 25.3% | 37.9% | 12.7% |
| GPT-5.5 | 21.6% | 26.9% | 22.4% | 27.4% | 13.4% |
| Claude Opus 4.7 | 17.9% | 23.9% | 18.6% | 20.6% | 8.5% |
| DeepSeek-V4-Pro | 18.0% | 28.5% | 17.1% | 21.9% | 14.5% |
| AI Scientist v2 | 16.2% | 27.5% | 22.3% | 25.7% | 13.7% |
| MiniMax-M2.7 | 11.6% | 22.9% | 10.9% | 9.1% | 8.5% |
The error analysis is revealing. The thickest failure flows come from evolutionary reasoning (T3) and verification (T4), fanning out through dynamics misclassification and driver misidentification. In plain terms: models cannot reliably tell whether two papers share a mechanism or just a topic.
A second finding cuts deeper. When the researchers looked at IG-Arena, where systems generate new research proposals, a pattern emerged that they call the “plausibility–coherence gap.” AI-generated proposals consistently scored higher on Variation than on Heredity. They introduced changes that sounded interesting, but the changes did not actually build on the parent lineage’s core mechanisms. A proposal can read like real research while being genetically incoherent.
Even more interesting: giving systems structured lineage information (Idea Genome objects and GenomeDiff evidence) did not simply raise scores across the board. It reshuffled the rankings. Some systems, like the GPT-5.5 plus tool-use combinations, could operationalize lineage evidence well, jumping from 85.2 to 87.5 on PES. Weaker systems saw much smaller gains from the same information. Lineage context separates the systems that can actually reason about inheritance from those that just benefit from more text.
Why It Matters
The AI tools being built to assist or even replace human researchers operate on a fundamentally different substrate than the humans they aim to supplement. When a scientist reads five papers in a subfield, she does not just extract keywords. She builds a mental map of which mechanism begat which, which limitation was repaired by whom, and where the open seams are. Current AI systems do something closer to topical clustering with a layer of fluent prose on top.
That gap has consequences beyond academic benchmarking. If an AI-generated research proposal claims to “extend” prior work but does not actually inherit its mechanisms, the proposal is noise. It wastes time, misallocates resources, and can fool even experienced reviewers who are reading for plausibility rather than lineage coherence. The Shanghai team’s finding that Variation outruns Heredity means the proposals look right even when they are wrong. That is a particularly dangerous failure mode for automated scientific discovery.
The benchmark also clarifies what frontier AI systems are actually bad at. It is not information retrieval (T1: 31.5% for the best system) or even pattern matching on individual GenomeDiff records. It is the compositional reasoning across multiple evolutionary steps: tracking how a mechanism travels through four or five papers, mutating along the way, while also accounting for hybrid imports from neighboring lineages. That kind of multi-hop reasoning, the Sankey diagrams show, is where everything falls apart.
How It Could Change Our Lives
The most immediate impact is on the credibility of AI-assisted research. Over the next few years, an increasingly large fraction of new scientific manuscripts will involve AI somewhere in the pipeline: literature review, hypothesis generation, or full drafting. If those systems cannot verify whether a new idea is lineage-coherent, the scientific record accumulates proposals that are fluent but foundationless. Reviewers, already stretched, may not catch the distinction between a paper that genuinely addresses a prior limitation and one that merely sounds like it does.
A practical fix could look like this: before a research lab deploys an AI scientist tool, they run it through a lineage sanity check (something like IG-Exam) to see if it can actually trace the mechanisms it claims to extend. The benchmark is open-source. The 10-domain structure means it tests generalization, not just memorization of one field. Labs that skip this check are effectively betting their research output on a system that, on the best available evidence, gets lineage reasoning right less than a third of the time.
Longer term, IdeaGene points toward a different kind of scientific infrastructure. Instead of PubMed searches that return paper lists sorted by date or citation count, imagine a genome browser for science: zooming in on a specific mechanism, tracing its mutations across decades, seeing exactly which paper repaired which limitation, and spotting where the lineage branched. That would change how scientists do literature review. Less like Googling and more like reading a phylogenetic tree.
The Bigger Picture
Scientific lineage reasoning sits at an uncomfortable intersection. It is clearly necessary for any AI system that claims to do real research. It is also clearly unmet by current models, even the strongest ones. And unlike many AI benchmarks that become saturated within months, this one has room to grow. 27.3% leaves a 70-point gap to ceiling. The researchers explicitly designed IG-Bench to resist memorization; the Idea Genome representation, the compositional task structure, and the multi-domain span all push against the shortcut-learning that has made other benchmarks obsolete within a year.
Limitations & What’s Next
The six evolutionary dynamics in IdeaGene are operational categories for evaluation, not an exhaustive theory of how science advances. The dataset, while large by curation standards at 1,961 traces, covers 10 domains, leaving dozens of fields untested. The IG-Arena judge panel uses LLM-based scoring with human audit, and while agreement with human judges reached 80%, that still leaves room for systematic bias in what “looks like” a good proposal.
The team’s roadmap includes expanding to more domains and deeper lineage trees, tightening the judge protocol for IG-Arena, and (most provocatively) treating IG-Bench as a diagnostic tool for improving AI research systems rather than just evaluating them. The gap between plausible and coherent, between interesting variation and genuine inheritance, is now measured and named. The next step is closing it.
📄 Zhou et al., “Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation,” arXiv:2607.08758, July 2026. Shanghai Jiao Tong University, Carnegie Mellon University, Microsoft et al.