Why AI Agent Governance Fails (And It's Not the Policy Doc)

Ask an engineering leader how they govern AI agents and you will usually get a document. It names approved use cases, assigns an owner, lists prohibited data, and has been through legal. Ask the same leader which credentials those agents actually run on, and the room gets quieter. The policy is the artifact everyone can point to. The access token is what decides what an agent can really do at 3am when nobody is watching. When the two disagree, the token wins, every time.

Key Takeaways

• A governance policy expresses intent. Credentials, scopes, authorization boundaries, and runtime controls determine what an agent can actually do, so governance has to reach the enforcement layer.
• You cannot govern agents you have never counted. An inventory of which agents exist, who deployed them, what identity they run as, and what they can reach is milestone one.
• Many agents begin life using human credentials or service accounts modeled on a developer's existing access because that is the fastest path from prototype to production. The inherited scope, not the model, is the vulnerability.
• Size permissions against the worst realistic outcome of an unattended action, and separate read paths from write paths before granting any autonomy.
• Put approval gates on side effects in infrastructure, not in prompts. A prompt-level guardrail is a suggestion made to a probabilistic system.

The policy says one thing, the access token says another

There are two governance systems running in most companies that have deployed AI in production. The first is written in a shared document and describes intent. The second is written in IAM roles, API keys, OAuth scopes, and database grants, and it describes capability. Only the second one executes.

This gap is not new. It is the same gap between a security policy and a firewall rule, and security-focused engineering teams have spent decades learning to close it. What is new is how quickly agents widen it. A human with excessive permissions is still constrained by working hours, attention, and a rough sense of consequence. An agent with the same permissions has none of those brakes, and acts at machine speed on an ambiguous instruction.

OWASP gave this failure mode a name worth using in meetings: Excessive Agency. Their breakdown is precise, splitting it into excessive functionality (the agent can reach tools beyond its task), excessive permissions (those tools run with broader privilege than needed), and excessive autonomy (high-impact actions proceed without a human). Notice that none of the three is a model problem. All three are things a team decided, or more often failed to decide, at deployment time.

You cannot govern the agents you have not counted

Inventory is the unglamorous first milestone, and it is routinely skipped. A Cloud Security Alliance survey of 418 IT and security professionals, published in April 2026, found that 82% of enterprises had unknown AI agents running in their environments, and 65% had experienced an agent-related incident in the previous twelve months. The same research found that 68% of organizations believed their visibility was strong, exposing a significant gap between perceived visibility and what organizations were actually discovering in their environments.

A usable inventory answers four questions per agent, and the fourth is the one that gets skipped:

1. What is it? The agent, its purpose, and the workflow it sits inside.
2. Who deployed it and who owns it now? Ownership decays fastest when the person who built the prototype changes teams.
3. What identity does it authenticate as? A named service principal, a shared key, or a developer's personal token.
4. What can that identity reach? Not what the agent is supposed to touch. What its credentials permit it to touch, transitively, through every tool and integration wired into it.

That fourth question is where the surprises live, because agents rarely talk to systems directly. They reach them through integration layers that were built for humans and were scoped generously so that nobody would have to file a ticket later. The agent inherits the generosity. The same survey flagged decommissioning as a leading risk factor, with only 21% of respondents running a formal retirement process, which leaves organizations exposed to retired agents retaining credentials longer than intended. If your organization has never done this pass, it is a bounded exercise, and it is exactly the kind of thing a structured software audit is for.

Agents inherit human access, and humans have far too much

Here is a common path an agent takes into production. An engineer builds a prototype on their own credentials because that is what is already configured on their machine. It works. It gets demoed. Someone asks how fast it can ship. The prototype moves to a server, and the fastest way to keep it working is to move the credential with it, or to mint a service account modeled on the engineer's access so that nothing breaks.

Nobody in that sequence made a bad decision in isolation. Collectively they granted an autonomous system the accumulated permissions of a senior engineer who has been added to things for years and removed from nothing. That is the actual attack surface, and it was created by a copy and paste, not by a model.

The fix is not complicated. It is just work that nobody schedules:

• One identity per agent. Never a shared key, never a human's token. If you cannot answer "which agent made this call" from your logs, you cannot investigate anything.
• Short-lived credentials. Minutes or hours, issued at task start, expiring on their own. This dramatically reduces the risk of forgotten credentials surviving long after the workflow that needed them.
• Scope derived from the task, not from the deployer. Start from an empty permission set and add only what a specific workflow provably needs.
• Separate identities per environment. An agent that reads staging should be structurally incapable of reaching production, not merely instructed to avoid it.

Budget the blast radius before you grant the autonomy

Most autonomy decisions get made backwards. The team asks what the agent should be allowed to do, gets an optimistic answer, and grants it. The better question is the inverse: if this agent behaves in the worst plausible way at 3am, unattended, on a bad input, what is the damage, and how long does recovery take?

Answer that first and the permission set derives itself. Recovery cost is the real budget. An agent whose worst case is a wrong label on a support ticket can run unattended. An agent whose worst case is a schema migration against a production database cannot, no matter how well the prompt is written, because the recovery path is a restore measured in hours.

The most useful line to draw is between read paths and write paths. Read paths generally carry disclosure risk, which can be bounded through data scoping. Write paths introduce an additional problem: side effects that may be difficult or impossible to reverse. Deletes, schema changes, financial transactions, outbound communication, and permission changes deserve their own category and their own controls. In regulated financial environments, an unattended write that cannot be attributed to a specific identity and reconstructed after the fact can quickly become an audit and compliance problem, not merely an engineering risk.

The NIST AI Risk Management Framework is worth reading here for its sequence rather than its language. Its Govern and Map functions come before Measure and Manage for a reason: you establish accountability and context before you try to control anything. Inventory and blast-radius sizing are the engineering versions of those first two steps.

Put the gate at the write boundary, not the prompt

A very common pattern is to write the guardrail into the system prompt. "Never delete production data. Always ask before modifying records." This feels like control and costs nothing to add, which is why it is everywhere.

It is not a control. It is a strongly worded request to a probabilistic system, and OWASP's own mitigation guidance is blunt about the alternative: implement authorization in downstream systems rather than relying on a model to decide whether an action is allowed. The prompt is one input competing with retrieved documents, tool output, user text, and anything an attacker managed to place in the agent's context. The IAM policy is not competing with anything.

So move the gate down a layer. Approval belongs on the side-effectful call, enforced by the system receiving it, not on the reasoning that produced it. Let the agent plan freely, reason freely, and draft freely. When it reaches for an irreversible operation, the boundary it hits should be infrastructure that has no opinion about how convincing the justification was. This is a containment problem, which makes it an architecture decision rather than a policy one.

You contain an agent with architecture. You do not contain it by trusting the model to have read the policy.

Governance that survives contact with production

Governance that holds up is mostly infrastructure and habit, not documentation. A working baseline looks like this:

• Every agent is registered with an owner, a purpose, an identity, and an expiry date on that identity. Unregistered agents get their credentials revoked, which is what makes registration real.
• Every side-effectful call is logged with the agent identity, the triggering input, the tool invoked, and the result. Reconstructing an incident after the fact is a design requirement, not a debugging nicety.
• Write paths have explicit gates at the receiving system, sized to recovery cost rather than to how confident the team feels.
• Permissions are reviewed on a schedule against what the logs show the agent actually used. Granted scope minus used scope is your standing over-permission, and it is a number you can drive down.
• Decommissioning is a real step. When an agent is retired, its identity dies with it on the same day.

None of this requires a new platform or a new vendor category. It requires knowing what you have, giving each thing its own narrow identity, and enforcing at the layer that actually enforces.

Which brings us back to the uncomfortable framing. When an agent deletes a production database, the incident review tends to reach for the model: it hallucinated, it misread the instruction, it went rogue. It did not go rogue. It did precisely what the credentials it was handed permitted it to do, at a speed no human would have matched, in a window nobody was watching. That is not a model failure. That is an access control decision, made months earlier, by people who never framed it as one.

If you don't know which agents are running in your environment or what their credentials can reach, start with the inventory. That is one of the first things we examine in a software audit.