Before an enterprise agent benchmark can measure anything, someone has to answer a question most organizations never have: what, exactly, does "done" look like?
WorkArena's authors built distinct validators for each of their task types on ServiceNow, specifying goal states that actual deployments leave to institutional habit. TheAgentCompany constructed an entire simulated company with every internal site and coworker interaction made explicit, because real organizational context resists specification. Tau-bench required machine-readable policy documents and annotated database goal states for domains where policy typically lives in PDFs and manager judgment.
Each benchmark's construction is the formalization that production work hasn't undergone. Task boundaries, success criteria, evaluation logic, all had to be invented for the benchmark because they didn't exist in usable form before it. The benchmarks aren't wrong for doing this. But the effort required to make enterprise work measurable tells you something about the distance between current workflows and agent-ready ones.
WorkArena: 33 tasks across 6 object types on ServiceNow. Each type needed its own validator — list parsers, form-field comparisons, catalog-item checks — because "completion" means something different for each.
TheAgentCompany: Best agent completed 30% of tasks autonomously. Simpler tasks with clear boundaries succeeded; long-horizon tasks requiring organizational judgment did not.
tau-bench: Compares final database state against annotated goal states. Even top agents scored below 50% task success. The paper's own caveat: a passing score can still miss policy failures, like processing a return without explicit user confirmation.
Across all three: Implicit workflow knowledge had to be made explicit before measurement could begin.