Yao's arc from defining how agents reason to stress-testing whether that reasoning holds up, traced through published work from 2022 to 2025, is worth paying attention to.
ReAct, published in October 2022, gave agents a grammar. Yao, then a Princeton PhD student on a Google Research internship, proposed that language models should interleave reasoning with action: think, act, observe, think again. On ALFWorld tasks, agents that showed their work succeeded 71% of the time versus 45% for those that just acted. "ReAct-style" became shorthand across the field. And embedded in its design was a quiet assumption: the reasoning trace was the evidence. If the agent's thinking looked right and the task got solved, that was success. Tree of Thoughts, seven months later, pushed reasoning deeper still, letting models explore multiple paths before committing, jumping Game of 24 success from 4% to 74%. More deliberation, more capability, same evaluative logic. The transcript said yes or no.
Both papers expanded what agents could reach. The question they left mostly untouched: when an agent says it did something, did the thing actually happen?
Tau-bench, published in June 2024 during a Sierra internship, stops reading the transcript and checks the ground. It builds simulated customer-service environments with real databases, domain policies, and users with specific requests. An agent handles a retail return or modifies an airline reservation. When the conversation ends, tau-bench checks the database rather than judging the dialogue. Did the right rows change? Did the correct refund amount land in the correct field?
This catches failures that transcript-level evaluation structurally misses. A conversation can read as flawless while the underlying state is wrong. The agent explains its reasoning, confirms details with the user, calls the return API but passes the wrong payment method. The customer hears "your refund has been processed." The database shows it went to a card they no longer use. In tau-bench's failure analysis of GPT-4o on retail tasks, wrong-argument errors like this sit alongside wrong-decision failures where the agent misapplied a domain rule it had access to. Clean reasoning, wrong state.
The numbers make the gap concrete. GPT-4o achieves roughly 61% pass@1 on the retail domain. Run the same tasks eight times and ask whether the agent succeeded every time, and that rate drops below 25%. The airline domain, layered with cabin classes, membership tiers, and cancellation policies, is harder: 35% pass@1. An agent that works most of the time and an agent that works reliably are separated by a gap that conventional evaluation misses entirely. Tau-bench found it by refusing to let the agent grade its own homework.
Yao's personal site now reads:
"I am a researcher at OpenAI. I study agents."
He's a core contributor on Deep Research and appears in the Operator system card. Building again. But the measurement he left behind doesn't go away. Once you check what changed in the world instead of reading the conversation, you learn how far apart they can be.
-
State-based evaluation spreads: WebArena reported its best GPT-4-based agent achieving 14.41% end-to-end task success versus 78.24% for humans on realistic web tasks, reinforcing how far database-state verification reveals agents still have to go.
-
Enterprise benchmarks arrive: WorkArena builds 33 tasks on the ServiceNow platform to evaluate agents against enterprise software objects like tickets, records, and approvals rather than open-web browsing.
-
Agents become behaviorally fingerprinted: A June 2026 preprint found that while some web agents bypassed all tested anti-bot mechanisms, every evaluated agent could still be distinguished from humans through multi-layer behavioral fingerprinting.
-
Payments force the record: AP reported that Visa's integration with ChatGPT includes guardrails like spending limits and approval steps, alongside a new dispute concern about failures in the agent-mediated middle between consumer intent and merchant processing.