A Well-Mannered Demo

Your POC ran on sample data, behind a firewall, and nobody tried to make it misbehave.

That is a well-mannered demo, not a security test.

I spent years building security products before this work, and what worries me most in enterprise AI is how cleanly the security question gets deferred. Not ignored. Deferred. It arrives in the production phase, after the budget is committed and the architecture is fixed. By then the answers are expensive.

Deferring is the rational move rather than a negligent one. A POC is judged on whether the thing works at all. A security question answers nothing about that and slows the answer down, so it waits. It waits until the phase where it has stopped being a question about the design. By then the design is finished.

Expensive has a specific shape here. The architecture is fixed, so a permissions problem can be a redesign rather than a setting. The integration is live, so tightening what the system can reach breaks somebody's workflow. And somebody has usually told a customer it works. That turns an engineering decision into a commercial conversation.

Three things a POC almost never tests

What happens when the input is hostile. An agent that reads email, tickets, or documents is reading text an attacker can write. Prompt injection is not an edge case. It is the normal case for any system taking input from outside.

What the system can reach. A POC gets a scoped, read-only slice. Production gets credentials. The blast radius of a model that can be talked into acting is everything those credentials touch.

Whether you can reconstruct a decision six months later. Regulators and enterprise buyers will ask. "The model decided" does not survive an audit.

None of these are model quality problems. They are decisions about scope, permissions, and logging, and every one is cheaper before production than after.

The cheap version of each

All three have a version that costs an afternoon while you are still in the POC.

• Hostile input: give one person an hour and ask them to make it misbehave. This is not a penetration test. It is a colleague being difficult on purpose.
• What it can reach: write down which credentials production would need, before anything is built against them. One page. It is usually the page that changes the design.
• Reconstructing a decision: decide what you will log now. Retrofitting a decision trail means rebuilding history you no longer have.

Security is not the last gate before launch. It is a constraint on what you build.

A constraint stated early narrows what you design. A gate applied late invalidates what you built. They carry the same information and they do not cost the same.