The Dead Battery in the Kill Switch

· Responsible AI · 10 min read

A switch you never test is just a battery quietly going flat.

The analysis

"There is a kill switch" is the most reassuring line in any AI governance pack, and the least examined. The RBI's draft model-risk guidance makes it an explicit requirement, and the newer UAE and neighbouring regimes arrive at much the same place. The phrase quietly smuggles in four assumptions: that the switch works, that someone is authorised and willing to pull it in time, that a human can then take over, and that there is a safe state to fall back to.

The record of automated failure is a record of each assumption breaking at the worst moment. Knight Capital in 2012 could in principle be stopped, but stopping it required humans to first diagnose which of eight software components was misbehaving; roughly forty-five minutes and around USD 440 million later the firm was gone. The 2010 Flash Crash produced circuit breakers, which halt a machine but cannot repair the market state the machine created — a halt buys time, it is not itself a safe state. The 737 MAX had two trained pilots in the loop and a documented override procedure, and crews in later simulator recreations had under a minute to identify a system some had never been told existed.

For anyone running automated decisions the transferable lessons are unglamorous. Define the safe state before you define the switch: what the business runs on for the hours or days after the model is off, who staffs it, and at what throughput. Test the fallback under load rather than on paper, because an unexercised manual process degrades exactly like an unexercised battery. Give the switch an owner with unambiguous authority to use it without escalation, and rehearse the decision, since most delay is organisational rather than technical. And instrument the trigger, because a control that depends on a human noticing an anomaly in a dashboard is only as fast as the slowest observer.

The board question is therefore not whether a kill switch exists. It is when it was last pulled in anger, what happened to the business while it was off, and who signed off on the result.

Full essay on Substack: The Dead Battery in the Kill Switch.

More on Responsible AI

Discuss this with Prasanna: pw@prima-partners.com · book a 30-minute call.