Practitioner's Corner
Practitioner's Corner
Shunyu Yao Taught Agents to Reason, Then Learned to Stop Trusting the Transcript
In 2022, Shunyu Yao gave agents a grammar for reasoning. ReAct became the template: think, act, observe, repeat. The reasoning trace was the proof. Three years later, Yao built tau-bench, which ignores the trace entirely and checks the database. GPT-4o completes retail tasks about 61% of the time on a single attempt. Require it to succeed consistently across eight runs, and the number drops below 25%. The person who taught agents to show their work now measures how much distance sits between a convincing transcript and a correct outcome.
Shunyu Yao Taught Agents to Reason, Then Learned to Stop Trusting the Transcript
In 2022, Shunyu Yao gave agents a grammar for reasoning. ReAct became the template: think, act, observe, repeat. The reasoning trace was the proof. Three years later, Yao built tau-bench, which ignores the trace entirely and checks the database. GPT-4o completes retail tasks about 61% of the time on a single attempt. Require it to succeed consistently across eight runs, and the number drops below 25%. The person who taught agents to show their work now measures how much distance sits between a convincing transcript and a correct outcome.
The Dispute File Is the Spec
When a cardholder calls their bank, it doesn't matter that the agent completed the purchase correctly. The file decides everything: the evidence a merchant must produce to defend the transaction. Visa and Mastercard have spent decades codifying what a defensible action looks like. Now that machinery applies to agent-mediated payments. And the questions it asks about authorization, scope, and remedy are indifferent to reasoning quality or benchmark scores. They ask whether what happened can survive challenge. Most agent systems aren't producing the artifacts that would let them answer.
The Dispute File Is the Spec
When a cardholder calls their bank, it doesn't matter that the agent completed the purchase correctly. The file decides everything: the evidence a merchant must produce to defend the transaction. Visa and Mastercard have spent decades codifying what a defensible action looks like. Now that machinery applies to agent-mediated payments. And the questions it asks about authorization, scope, and remedy are indifferent to reasoning quality or benchmark scores. They ask whether what happened can survive challenge. Most agent systems aren't producing the artifacts that would let them answer.
Recognition Problem
Researchers can now identify web agents by how they type, scroll, and move a cursor. Paste-based form fills, direct jumps to click targets, repeated delete-and-retype loops. One study detected all seven agents tested; Cloudflare caught one. A June preprint went further, fingerprinting agents at the network layer too. Within weeks, developers were publishing techniques to mimic human mouse trajectories.
Sharper fingerprints produce sharper evasion, and the cycle keeps tightening without ever touching the useful question: is this automation allowed to act here? Early proposals for permission manifests hint at an alternative, but nothing is deployed. Recognition keeps improving. The ability to say "yes, this agent is authorized to be here on behalf of this user" remains at zero. Every gain in detection just funds the next round of camouflage.
Further Reading
Past Articles
One benchmark task asks an agent to create a hardware asset in ServiceNow. The agent fills in fields, clicks submit, rea...
When a person clicks a button on a webpage, the infrastructure only sees the click. It never sees the reading, the conte...
Retry logic in distributed systems works because of a quiet contract: same input, same execution path. An idempotency ke...
To a person with a browser, a form field is a text box. To a machine trying to operate that same page, it's a hundred li...