Anyone who has run production systems learns an unglamorous rule early: a backup that has never been restored is a guess. Nobody knows whether the business can rebuild from it until someone tries, and the first try usually happens in the middle of an outage, the worst possible moment to find out. So careful teams rehearse restores on quiet days, and they time them.
Many companies adopting AI are building a human backup and skipping that rehearsal. The usual arrangement goes like this. The system does the hard part, and an employee reviews, adjusts and signs off. That same employee is also, usually without anyone saying so, the plan for when the system fails, and ordinary days never test that plan.
The daily review may not be keeping the person capable. In a study published last month in Nature Human Behaviour, 487 management consultants worked on unfamiliar data-science tasks. On one of them, cleaning messy data with Python code, consultants given AI and training outscored those working without it by 49 percentage points. On a short quiz at the end, with AI forbidden, the two groups scored about the same. The authors describe AI as an "exoskeleton": it makes its wearer stronger only while it is on. The participants were beginners, so the study says nothing about experts losing skills they already had. It does suggest that expert-looking work produced with AI is weak evidence that the person who produced it can judge such work.
Plenty of work can rest safely on an exoskeleton. If the tool goes down and the task can wait, the cost is a delay. Things get harder when a person has to step in, and what kind of stepping in matters a great deal.
- One kind is doing the work by hand while the tool is down.
- Another is catching a failure that announces itself: an error message, an empty report, an impossible total.
- The hardest is catching an answer that is wrong and looks right, such as a reconciliation that balances because two mistakes cancel out.
The first two can tolerate dependence, as long as the manual routine is rehearsed, because the failure itself tells people when to act. The third cannot, because nothing signals that it is time to look. The person signing off needs an independent sense of what a right answer looks like.
Aviation research suggests which of these capacities fades first. In a 2014 simulator study of 16 airline pilots, hand-flying held up despite little practice. What slipped was the thinking around it: keeping track of the aircraft's position, planning the next move, interpreting a failed instrument. Pilots often noticed that something was wrong and then could not work out what. For a reviewer paid to catch plausible errors, that diagnostic step is most of the work.
Practice should match the responsibility a person actually carries. Someone expected to keep working through an outage needs to do the task alone, regularly. Someone expected to catch plausible mistakes needs exercises seeded with them, plus time to walk a senior colleague through the diagnosis. The FAA's 2013 safety alert on manual flying offers a workable pattern: hand-fly in calm conditions, use automation under heavy workload, and train for both.
The obstacle is timing. The cost of practice shows up in this quarter's numbers, while the payoff arrives on the day something breaks, and by then the manager who cut the practice may have moved on. The Naval Academy budgeted against that gap when it restored celestial navigation in 2015, teaching midshipmen to fix a ship's position from the sun and stars. The commitment was three classroom hours plus summer practice, a modest cost accepted in advance.
A company that names a person as its fallback should be able to answer three plain questions: when that person last worked without the system, what they caught, and how long recovery took. Without those answers, it is relying on a backup it has never restored.

