Jon W. Hansen, FCIPS · Procurement Insights | Hansen Models™ · October 2026
In the November–December issue of Harvard Business Review, Das Narayandas of Harvard Business School asks why even the best employees override AI systems that work. His answer is that they don’t distrust the technology. They distrust the organization. When a system makes a decision and the decision goes wrong, the person who followed it is still the one held accountable. So people protect themselves: they add review steps, lower thresholds, flag sensitive cases and quietly turn a decision engine into an adviser.
It is an important article, and most of it matches what the Procurement Insights archive has recorded for nearly two decades. But it rests on one assumption that deserves a closer look before anyone acts on its prescription.
What the article gets right
Narayandas is right that the problem is structural, not technical. His examples are well chosen. Loan officers in automated lending pushed borderline applications over approval thresholds to earn volume incentives. A dispatcher at a global logistics firm explained his overrides in one line: if the system’s call goes wrong, it is his name on the report. And when UPS wanted drivers to follow its ORION routing system, the decisive move was changing the scorecard, so drivers were measured against the system’s route rather than their own past performance.
The loan officers are a familiar story. In a 1998 defense maintenance operation, service technicians were measured on the number of initial service calls they made each day. Stopping to submit a part order cost them a call, so they batched their orders until late afternoon. Each technician did the job exactly as measured. Collectively, they missed customs cutoffs, drove up part costs and pushed the call close rate down to 51 percent on a contract that required 90 percent. One met measure defeated the measure the contract paid for.
Narayandas also gives the article a section titled We Have Seen This Before, tracing the pattern through ERP and credit scoring. That is the argument this archive has made since 2007: the technology changes each era, and the organizational pattern does not. His insistence that the organizational work be designed before a system is scaled, not retrofitted afterward, is Phase 0™ in HBR’s words. And his example of Toyota’s Andon cord, where anyone can stop the line and stopping it brings help rather than blame, is a firm fence by another name.
The assumption in the title
The article’s title takes something for granted: these are AI systems that work. Throughout, the model is treated as right, and the override as self-protection to be designed out. The prescription follows from that: make the algorithm the operational default, make overrides visible, and move accountability to the system’s owners.
But a system that works, works inside its frame. It is accurate about the operation it was built to represent. Whether that representation is complete is a separate question, and it is the question that decides whether an override is a defensive habit or a warning.
Two kinds of override
Some overrides are exactly what Narayandas describes: a person protecting their career from a decision they did not make. Those tell you the accountability design is wrong, and his remedies are the right ones.
Other overrides are a person seeing something the model does not contain. Look again at his own dispatchers. They adjusted routes for customers they had long-standing relationships with, for shipments that mattered politically inside the firm, and for drivers they trusted. Some of that may well be self-protection. But client relationships, internal politics and trust between people are also strands of the real operation, and none of them was in the routing model. The article reads those adjustments only as containment. They are also evidence.
The archive has made this point about shadow tools: when people repeatedly route around the sanctioned system, they are producing behavioral evidence that the system may not serve the work. That evidence does not prove the cause. It tells you where to start tracing.
The difficulty is that the two kinds of override look identical on a dashboard. Narayandas suggests tracking whether exceptions are rising or falling. A falling exception rate could mean authority has moved successfully. It could equally mean that people have stopped reporting what the model cannot see. The count alone cannot tell you which.
What the evidence says to do
The 1998 operation did not solve its problem by moving authority to the system. It solved it by tracing back from the failed outcome to find what was producing it: the time of day the orders arrived, and the incentive that put them there. Only then was the system built, with self-learning algorithms working as firm fences. People could change the weighting of price or delivery, or introduce a condition the system could not see on its own, such as a freak storm or a port strike. The system recalculated within its defined authority and kept a record of who changed what, when, where and why. Next-day delivery rose from 51 percent to 97.3 percent and held for seven years.
The human did not lose authority in that design. The human’s authority changed: from overriding individual decisions to supplying context the system could not have, inside a structure that recorded it and learned from the outcome.
This month’s Isaias Strand Lab showed the other side. A wrong diesel figure passed every check in the record, including review by four independent AI models working with a fifth. A person stopped it, by noticing that it contradicted what the lab had already published. A design that had moved all authority to the system would have let the error through.
So before moving authority to an AI system, three steps:
- Validate the original. Before trusting that a system works, confirm that it represents the operation as it actually runs, including the workarounds and shadow tools that never appear on the official map. That is the purpose of Phase 0™.
- Sort the overrides before suppressing them. Trace each pattern of overrides back from the outcome. Self-protective overrides point to the accountability design. Overrides that reveal a missing strand point to the model.
- Let authority follow evidence, not the model. Whoever holds authority, the system or the person, evidence from real outcomes must be able to stop an action and change the explanation behind it. That is the firm fence test, and it applies to the system as much as to the people around it.
The question in both directions
Narayandas closes with a good test: if the system were turned off tomorrow, would the decisions that matter actually change? If not, he argues, authority has not moved.
It is worth asking the reverse question too: if the system were left on, and nobody could override it, which part of the “real-world” operation would you never see again?
His four questions for leaders are the right place to start: who loses discretion, who gains authority, who explains outcomes, and who absorbs the risk during the transition. A fifth belongs alongside them: what does the override know that the system doesn’t?
Sources and related reading:
Truth Is Believing. Accuracy Is Knowing. Outcome Is Proof.™
-30-
Related
Harvard Business Review Talks About Moving Authority to AI. My Advice: Only If You’ve Read the Override.
Posted on October 22, 2026
0
Jon W. Hansen, FCIPS · Procurement Insights | Hansen Models™ · October 2026
In the November–December issue of Harvard Business Review, Das Narayandas of Harvard Business School asks why even the best employees override AI systems that work. His answer is that they don’t distrust the technology. They distrust the organization. When a system makes a decision and the decision goes wrong, the person who followed it is still the one held accountable. So people protect themselves: they add review steps, lower thresholds, flag sensitive cases and quietly turn a decision engine into an adviser.
It is an important article, and most of it matches what the Procurement Insights archive has recorded for nearly two decades. But it rests on one assumption that deserves a closer look before anyone acts on its prescription.
What the article gets right
Narayandas is right that the problem is structural, not technical. His examples are well chosen. Loan officers in automated lending pushed borderline applications over approval thresholds to earn volume incentives. A dispatcher at a global logistics firm explained his overrides in one line: if the system’s call goes wrong, it is his name on the report. And when UPS wanted drivers to follow its ORION routing system, the decisive move was changing the scorecard, so drivers were measured against the system’s route rather than their own past performance.
The loan officers are a familiar story. In a 1998 defense maintenance operation, service technicians were measured on the number of initial service calls they made each day. Stopping to submit a part order cost them a call, so they batched their orders until late afternoon. Each technician did the job exactly as measured. Collectively, they missed customs cutoffs, drove up part costs and pushed the call close rate down to 51 percent on a contract that required 90 percent. One met measure defeated the measure the contract paid for.
Narayandas also gives the article a section titled We Have Seen This Before, tracing the pattern through ERP and credit scoring. That is the argument this archive has made since 2007: the technology changes each era, and the organizational pattern does not. His insistence that the organizational work be designed before a system is scaled, not retrofitted afterward, is Phase 0™ in HBR’s words. And his example of Toyota’s Andon cord, where anyone can stop the line and stopping it brings help rather than blame, is a firm fence by another name.
The assumption in the title
The article’s title takes something for granted: these are AI systems that work. Throughout, the model is treated as right, and the override as self-protection to be designed out. The prescription follows from that: make the algorithm the operational default, make overrides visible, and move accountability to the system’s owners.
But a system that works, works inside its frame. It is accurate about the operation it was built to represent. Whether that representation is complete is a separate question, and it is the question that decides whether an override is a defensive habit or a warning.
Two kinds of override
Some overrides are exactly what Narayandas describes: a person protecting their career from a decision they did not make. Those tell you the accountability design is wrong, and his remedies are the right ones.
Other overrides are a person seeing something the model does not contain. Look again at his own dispatchers. They adjusted routes for customers they had long-standing relationships with, for shipments that mattered politically inside the firm, and for drivers they trusted. Some of that may well be self-protection. But client relationships, internal politics and trust between people are also strands of the real operation, and none of them was in the routing model. The article reads those adjustments only as containment. They are also evidence.
The archive has made this point about shadow tools: when people repeatedly route around the sanctioned system, they are producing behavioral evidence that the system may not serve the work. That evidence does not prove the cause. It tells you where to start tracing.
The difficulty is that the two kinds of override look identical on a dashboard. Narayandas suggests tracking whether exceptions are rising or falling. A falling exception rate could mean authority has moved successfully. It could equally mean that people have stopped reporting what the model cannot see. The count alone cannot tell you which.
What the evidence says to do
The 1998 operation did not solve its problem by moving authority to the system. It solved it by tracing back from the failed outcome to find what was producing it: the time of day the orders arrived, and the incentive that put them there. Only then was the system built, with self-learning algorithms working as firm fences. People could change the weighting of price or delivery, or introduce a condition the system could not see on its own, such as a freak storm or a port strike. The system recalculated within its defined authority and kept a record of who changed what, when, where and why. Next-day delivery rose from 51 percent to 97.3 percent and held for seven years.
The human did not lose authority in that design. The human’s authority changed: from overriding individual decisions to supplying context the system could not have, inside a structure that recorded it and learned from the outcome.
This month’s Isaias Strand Lab showed the other side. A wrong diesel figure passed every check in the record, including review by four independent AI models working with a fifth. A person stopped it, by noticing that it contradicted what the lab had already published. A design that had moved all authority to the system would have let the error through.
So before moving authority to an AI system, three steps:
The question in both directions
Narayandas closes with a good test: if the system were turned off tomorrow, would the decisions that matter actually change? If not, he argues, authority has not moved.
It is worth asking the reverse question too: if the system were left on, and nobody could override it, which part of the “real-world” operation would you never see again?
His four questions for leaders are the right place to start: who loses discretion, who gains authority, who explains outcomes, and who absorbs the risk during the transition. A fifth belongs alongside them: what does the override know that the system doesn’t?
Sources and related reading:
Truth Is Believing. Accuracy Is Knowing. Outcome Is Proof.™
-30-
Share this:
Like this:
Related