Melissa Pan once worked on backup, restore and recovery for IBM's Db2 database. Recovery code assumes a failure leaves a log that someone can read back, step by step, to work out what went wrong. Her later work on agents, the language-model programs that carry out a task over many steps, looks to me like recovery work on a new kind of system. That assumption has held up less well: agent researchers rarely get to read the log.
Her first widely cited agent paper did have it. With Mert Cemri and Shuyi Yang as co-leads, she built MAST, a classification of how teams of cooperating agents fail, and used it to label 1,642 execution traces, the step-by-step records of each run. The traces came from open-source research frameworks and are public, so anyone who doubts a label can open the run and argue with it.
Open frameworks are a particular sample, and Pan has described a similar blind spot. Writing about what surprised her in the production data, she said:
"At least for me, as a grad student I previously had the view that most agents were coding agents."
She doesn't say where that view came from, but coding agents are unusually visible: many run on open benchmarks and compete on public leaderboards. Researchers could easily come to treat the agents they can see as typical.
Her next project, Measuring Agents in Production, went looking for the rest and met the obstacle its authors name: "production agents are typically proprietary, limiting company participation." Unable to inspect the systems, the team asked about them: 20 interview case studies, plus 86 of 306 survey responses, those describing agents in production or pilot. Both capture what practitioners said, which may differ from what their systems logged. The study kept participants' data private but published its interview outline and survey questionnaire, so any team can apply them to its own systems.
AdaMAST, introduced in a July 2026 manuscript Pan coauthored, goes back to traces. Instead of MAST's fixed categories, it derives failure categories from the traces of the system under study. "We need more task-specific failure understanding to build better systems," Pan wrote. In principle, such a method lets a company study its failures without handing its logs to anyone.
That would ease the company's access problem but not the field's, because incentives on either side of the firewall diverge. A firm that finds a recurring failure has every reason to fix it and little reason to describe it: publishing takes staff time, invites questions from customers and may teach a competitor something. The method moves in freely; what it turns up comes out only if someone chooses to share it.
Pan's work pulls in two directions here. MAST's fixed categories let a reader compare one framework's failures with another's; AdaMAST trades that shared vocabulary for fit. Suppose two companies published only their derived categories and frequencies. The same failure might appear under two names, or one name might cover two different events. Even matching names might not mean matching counts: ten failures per hundred tasks attempted measures something different from ten among runs already flagged for review.
Settling either problem requires an agreement among firms. One arrangement, my suggestion rather than Pan's, would be a shared pool firms can see into only by contributing, with common definitions, a stated sample and rules fixed before results come in, so no participant can pick its flattering numbers afterward. Until then, the field will keep forming its beliefs from whichever systems happen to be open, the kind of picture Pan says production data corrected for her.
- Adaptive failure vocabularies: The Fantastic Adaptive Taxonomies manuscript describes failure categories derived along system, role and domain axes, which bears on whether two firms' derived categories could ever be compared.
- Reusable production questionnaires: The June 2026 Measuring Agents in Production manuscript prints its interview outline in §C.3.2 and its complete survey in §G, so a team can see exactly what was asked before trusting the headline percentages.
- The human-labelled subset: The MAST paper describes 21 human-annotated traces, while the current MAST-Data README lists 19 rows in the human-labelled file, a small discrepancy in an otherwise inspectable release.
- Funding for production research: Berkeley's account of the Laude Slingshot awards lists the production-measurement project under Pan and Matei Zaharia, though it names no shared telemetry or audit commitment.

