CorporateBench
Large-Scale Q&A Benchmarking with Temporal Knowledge Bases
Sil Hamilton, Albert Yu Sun, Oscar J. Romero, Carl-Leander Henneking, David Mimno, Bishan Yang, Igor Labutov
Epiq AI Labs · Cornell University
CorporateBench investigates a fundamental challenge for enterprise AI: can models reconstruct and reason over knowledge scattered across thousands of documents? Using four synthetic companies and verifiable ground truth, it reveals significant limitations in large language models, particularly in recovering relationships and tracking changes over time. The findings expose a critical gap between retrieving information and building the coherent understanding needed for reliable reasoning at scale.
The problem
Real-world knowledge work rarely involves finding an answer in a single document. Understanding an organization means piecing together people, events, decisions, and relationships from thousands of fragmented records, often spanning different points in time.
Yet most AI benchmarks test isolated questions or relatively small collections, leaving a fundamental question unanswered: Can AI build a coherent understanding of an organization from its documents?
Our approach
CorporateBench starts by simulating entire companies: their organizational structures, employees, projects, communications, and how these evolve over time. From these underlying worlds, it generates large collections of interconnected documents.
Because the underlying knowledge is known, we can evaluate not only whether models answer questions correctly, but whether they can reconstruct the facts and relationships those answers depend on.
What we found
Today’s models remain far from reliable at enterprise scale. Across five evaluated models, reconstructing relationships proved substantially harder than identifying individual entities, with temporal relationships particularly challenging.
Models also performed markedly better when given direct access to structured knowledge than when relying on document retrieval, with the gap generally widening at larger scales.
These findings highlight a critical challenge for enterprise AI: retrieving information is not the same as understanding it. Progress will require systems that can construct, connect, and reason over knowledge at scale.
Why it matters for legal work
Legal work is often downstream of the same complex enterprise information. In litigation, discovery, regulatory inquiries, and investigations, understanding what happened, who knew what, and when can require reconstructing an organization’s history from fragmented evidence.
The same challenge exists within organizations, where compliance, internal investigations, and risk management depend on understanding relationships, activities, and changes that may never be explicitly documented in one place.
These are different applications of the same fundamental problem: transforming unstructured information into reliable, connected knowledge that can support reasoning and decisions.