Stop Asking Your AI Nicely Not to Leak Data

· AI Evaluation & Safety · 6 min read

A prompt instruction is not a security control. It is a wish, written on a sticky note and taped to a vault.

The analysis

Ask the team running your newest AI assistant what actually stops a determined customer from talking it into revealing something it should not. In most organisations the honest answer is that the system prompt tells it not to. That is an instruction to a probabilistic text generator, not a boundary, and a persuasive person walks past it the way a smooth caller walks past an undertrained service rep.

The more expensive exposure is not what the agent reads but what it does. A leaked record is damaging; an executed action is often irreversible. An agent talked into porting a phone number, releasing a payment or sending correspondence under your domain creates consequences that no incident response undoes. Most control conversations still focus almost entirely on retrieval and barely at all on the action surface, which is where the permanent loss sits.

The instinct to fix this with sharper prompts or fine-tuning misreads the problem. Authorisation is inherently per-person: whether this request may see this wallet depends on who is asking. A model's weights are identical for every user and carry no reliable notion of the requester's identity, so per-person permissions cannot be baked into them. The controls have to sit outside the model, enforced deterministically under the identity of the actual human or service making the request. Done properly, the agent can be fully jailbroken and the sensitive action still fails, because the decision was never the model's to make.

Treated this way, agent security stops being a tax on ambition and becomes the unlock. Frontier models are a commodity everyone can rent; the durable advantage belongs to the organisations that can safely point agents at their most sensitive data and their highest-value actions. That is the difference between an initiative that dies in security review and one that ships.

If you want a candid read on where your own controls stand, the role-aware AI readiness assessment on this site covers exactly this ground in about five minutes.

Full essay on Substack: Stop Asking Your AI Nicely Not to Leak Data.

More on AI Evaluation & Safety

Discuss this with Prasanna: pw@prima-partners.com · book a 30-minute call.