Nobody Called It a Model
· Enterprise AI · 11 min read
The models that hurt you are, almost by definition, the ones your inventory does not count.
The analysis
Ask a chief risk officer how many models run inside the institution and the number will be confident, documented and wrong — because it counts the models the organisation knows about. Read the RBI's draft model-risk guidance, or the UAE and Qatar equivalents, as rulebooks for machine learning and you aim at the wrong decade. The question underneath them is far more embarrassing: do you know these systems exist at all?
The failure pattern is consistent across four decades of examples. JPMorgan's London Whale Value-at-Risk figure ran through manually copy-pasted spreadsheets containing a formula that divided by a sum instead of an average, halving apparent risk — a critical model that no validation process had seen, because on paper it was a workbook. Knight Capital's fatal deployment reactivated dormant code retired in 2003 and never deleted, a system the firm had stopped counting a decade earlier. The Dutch childcare-benefits algorithm was governed as an efficiency tool while wrongly accusing roughly 26,000 families and ultimately bringing down a government. And today's version is unmanaged: employees pasting source code and internal material into consumer AI tools, with the majority of AI users at work bringing their own.
None of those systems failed because the modelling was unsophisticated. They failed because nobody classified them as models, so no inventory, ownership, validation or decommissioning discipline attached to them. That is the practical remit of the new guidance, and it is why compliance projects that start with the LLM roadmap start in the wrong place.
The work that actually reduces risk is inventory-first: define a model by the consequence of its decisions rather than the technology behind it, sweep the spreadsheets, rules engines, dormant code paths and shadow AI tools into that definition, assign a named owner and a review cadence to each, and set an explicit retirement process so nothing sits dormant waiting to be reactivated. It is unglamorous work, and it is the part regulators will ask about first.
Full essay on Substack: Nobody Called It a Model.
More on Enterprise AI
- What the Stanford 2026 AI Index Actually Says
- Our AI Coding Numbers Were Perfect. We Were Shipping Slower.
- From Ownership to Access — And Perhaps Back Again
Discuss this with Prasanna: pw@prima-partners.com · book a 30-minute call.