AI Agent Governance: A Framework for AI That Takes Actions
Mark aistrategy.guide as a preferred source Our AI strategy guides show up more often, with a Preferred badgeHow to govern AI agents that call tools, move data and act without a human approving each step. Autonomy tiers, agent identity and least privilege, sandbox testing, runtime controls, and the 2026 rules that apply: Singapore's agentic framework, OWASP's agentic Top 10, NIST, the EU AI Act after the omnibus, CCPA ADMT, GDPR and HIPAA.
An AI model that writes a bad paragraph wastes a few minutes of someone’s time. An AI agent that misreads a request can refund the wrong customer, email a supplier with confidential pricing, or merge a broken change into production, and it can do it a hundred times before anyone looks. That difference, between generating output and taking actions, is why agents need governance of their own.
Most enterprise AI governance programs were designed for the first case: approve a use case, check the model, review the output. Agents break that pattern because the risky moment is not the model’s answer but the tool call that follows it. This page is the framework I would put around agents: what to control, how to set autonomy, which 2026 rules apply, and the one-page governance plan every agent should have before it touches production. For the wider program it plugs into, start with the enterprise AI governance framework. For the security tooling, see AI agent security.
Why agent governance is different
Four properties of agents change the governance problem.
They act through tools. An agent’s permissions are the sum of every API, database and inbox it can reach. A model with no tools cannot exfiltrate data; an agent with a mail tool and a CRM connector can.
They chain decisions. A small misreading in step one becomes the input to step two. Errors compound across a workflow instead of staying contained in one response, which is why output review alone misses most agent failures.
They take untrusted input as instructions. Anything an agent reads, including a web page, an email, a PDF or another agent’s message, can carry text that redirects it. That is prompt injection, and it turns every data source into a possible control channel.
They need an identity. An agent that acts on a system needs credentials. If it borrows a human’s, nobody can tell afterwards who did what, and the agent quietly inherits far more access than its task needs.
The frameworks that exist in 2026
There is still no single standard for agents, but the pieces are now published and they fit together.
| Framework | Issuer, date | What it gives you |
|---|---|---|
| Model AI Governance Framework for Agentic AI | Singapore IMDA, 22 January 2026 | The first governance framework written for agents. Four dimensions: assess and bound risks upfront, make humans accountable, implement technical controls and processes, enable end-user responsibility. |
| Top 10 for Agentic Applications (ASI01 to ASI10) | OWASP, December 2025 | The threat list: goal hijack, tool misuse, identity and privilege abuse, agentic supply chain, unexpected code execution, memory and context poisoning, insecure inter-agent communication, cascading failures, human-agent trust exploitation, rogue agents. |
| AI Agent Standards Initiative | NIST CAISI, February 2026 | US work on agent identity, authorization and security, including an NCCoE concept paper on software and AI agent identity and authorization. |
| AI Risk Management Framework | NIST, 2023, with the generative AI profile | The operating loop (Govern, Map, Measure, Manage) most enterprises already use. Agents slot in as a risk class. |
| ISO/IEC 42001 | ISO, 2023 | The certifiable AI management system. Agent controls become part of its scope rather than a parallel program. |
| EU AI Act, after the Digital Omnibus | Regulation (EU) 2026/1744 | Annex III high-risk obligations moved to 2 December 2027 and Annex I to 2 August 2028. An agent used in hiring, credit or another Annex III area is a high-risk system whatever it is called. |
My advice is not to start a separate agent governance program. Use NIST AI RMF or ISO 42001 as the spine you already have, adopt the Singapore framework’s four dimensions as the agent-specific layer, and use the OWASP list as the security test checklist.
Control 1: an inventory with two named owners
Every agent gets an entry in the AI inventory before it runs anywhere near real data: what it does, which systems it touches, which model and tools it uses, and two owners. The business owner answers for the outcome. The technical owner answers for the configuration, permissions and monitoring. If the business side will not put a name against an agent’s results, it is an experiment and should run in a sandbox as one.
The inventory is also where shadow agents surface. Low-code agent builders inside SaaS platforms make it easy for a team to wire an agent to production data without IT knowing. A quarterly sweep of platform admin consoles and API keys finds them.
Control 2: autonomy tiers by reversibility and impact
Autonomy is not a property of the agent; it is a decision per use case. I set it with two questions: if the action is wrong, can we undo it cheaply, and how far does the damage reach?
| Tier | What the agent may do alone | Human role | Typical examples |
|---|---|---|---|
| 0. Assist | Read, research, draft | Human sends or acts | Drafting replies, summarising tickets, research briefs |
| 1. Act, reversible, internal | Write to internal systems where changes are logged and undoable | Sampled review after the fact | Updating CRM fields, tagging tickets, creating draft orders |
| 2. Act with approval | Prepare the action and wait | Approves each action at the point of execution | Customer emails, refunds, supplier messages, production deploys |
| 3. Act autonomously, bounded | Execute within hard limits (amount, scope, rate) | Monitors metrics, handles exceptions | Refunds under a set amount, password resets, routine reorders |
Tier 3 is earned, not granted. An agent moves from tier 2 to tier 3 on one specific task when its approval log shows a sustained accuracy rate the business owner signs off, and it drops back automatically if the error rate rises. Hard-to-reverse actions in regulated areas (credit, hiring, health, legal) stay at tier 2 regardless, because the law requires meaningful human review, as covered below.
Control 3: identity and least privilege
Each agent gets its own non-human identity: its own service account, its own API keys, its own scopes. Never a human’s token. That gives you three things at once: an audit trail that shows the agent acted, permissions you can cut without locking a person out, and a kill switch that is one credential revocation.
Scope the permissions to the task, not the platform. An agent that answers order-status questions needs read access to orders, not write access to the customer table. For tools exposed through the Model Context Protocol, approve each server and each tool individually; an MCP server is a supply-chain dependency and should be reviewed like one. Keep secrets out of prompts and context windows entirely, since anything in context can be coaxed back out.
Control 4: test in a sandbox before release
Agents need a release gate, just like code. Before an agent reaches production data it should pass four kinds of tests in a sandbox that mirrors production systems with synthetic or masked data.
- Task tests: a fixed set of realistic tasks with known correct outcomes, scored automatically. This becomes the regression suite every time the prompt, model or tools change.
- Policy tests: cases where the right answer is to refuse, escalate or ask. Agents fail these more often than task tests.
- Adversarial tests: injected instructions in documents, emails and web pages, mapped to the OWASP agentic list. See enterprise AI red teaming.
- Tool-use tests: does the agent call the right tool with the right arguments, and does it stop when a tool errors?
Then release in stages: shadow mode (the agent proposes, humans act), a small share of live traffic, then wider rollout, with an A/B comparison against the human or rule-based baseline on the business metric. The practical detail of building these evaluations is on CTAIO’s guide to testing and evaluating AI agents.
Control 5: runtime guardrails, approval gates and a kill switch
At runtime, put controls outside the model, where the model cannot talk its way past them.
- Input and output filtering for injection attempts and sensitive data, enforced at a gateway.
- Hard limits per agent: spend per day, actions per hour, maximum transaction size, allowed recipients and domains.
- Approval gates for every tier 2 action, with the full proposed action shown to the approver, not a summary the agent wrote.
- Full tracing of every model call, tool call and decision, retained long enough to reconstruct an incident. This is where LLM observability becomes a governance capability rather than an engineering nicety.
- A kill switch with a named person who can pull it, tested, not just documented.
The pattern that scales is asymmetric: let agents draft and research freely, and gate the narrow set of actions that publish, pay or change production. The gates cost reviewers minutes a day, and they are what makes it safe to run the drafting at volume.
Control 6: map each agent to the rules that apply
Agents do not get a regulatory exemption for being new. The questions are the same as for any automated processing, applied per agent.
- GDPR Article 22: people have the right not to be subject to a decision based solely on automated processing with legal or similarly significant effects. Agents that decide on credit, employment, insurance or access need meaningful human review or another Article 22 basis, plus a data protection impact assessment.
- CCPA automated decision-making rules: California’s ADMT obligations apply from 1 January 2027, with notice and an opt-out where the technology replaces or substantially replaces human judgment in a significant decision. Risk assessments follow from 31 December 2027.
- HIPAA: an agent that touches protected health information needs a business associate agreement with every model and tool vendor in its chain, and the minimum-necessary standard applies to what goes into its context.
- EU AI Act: classify by use, not by technology. After the omnibus, Annex III high-risk obligations apply from 2 December 2027; Article 50 transparency rules for AI that interacts with people or generates synthetic content apply earlier.
The CISO, DPO and general counsel split this work; the CISO’s share of AI governance covers who owns what.
The one-page agent governance plan
Every agent in production should have one page with these fields filled in. A blank field means it is not ready.
| Field | What to write |
|---|---|
| Owners | Business owner, technical owner |
| Task | What it does, in one sentence, and what it must never do |
| Systems and tools | Every system, API and MCP server it can reach, with access level |
| Data | Data classes in scope (personal, health, financial, confidential) |
| Autonomy tier | 0 to 3, and where human approval sits |
| Identity | Service account, scopes, credential rotation |
| Pre-release tests | Task, policy, adversarial and tool-use results, with date |
| Runtime limits | Spend, rate, transaction size, allowed recipients |
| Logging | What is traced, where, retention period |
| Kill switch | How, and who can pull it |
| Regulations | GDPR, CCPA ADMT, HIPAA, EU AI Act classification, sector rules |
| Success metric | The business metric and its target, agreed before launch |
| Review date | At most six months out, and after any incident or model change |
A 90-day starting plan
- Inventory every agent in use, including those inside SaaS platforms, and assign two owners each.
- Give each agent its own identity and cut permissions to what its task needs.
- Set an autonomy tier per use case and put approval gates in front of every tier 2 action.
- Route model and tool calls through a gateway with tracing, spend limits and a kill switch.
- Build a task, policy and adversarial test suite for the highest-risk agent first, and make it the release gate.
- Write the one-page plan for each production agent and review it with legal and security.
Agent governance done this way is what lets you widen autonomy safely. The organisations that will run agents at scale in 2027 are the ones that can show, per agent, who owns it, what it can touch and how they would stop it.
Sources
- IMDA, Model AI Governance Framework for Agentic AI, 22 January 2026, summarised by Baker McKenzie.
- OWASP GenAI Security Project, Top 10 for Agentic Applications, December 2025, summarised by Giskard.
- NIST CAISI, AI Agent Standards Initiative, February 2026.
- Orrick, EU AI Act Update: Digital Omnibus Finalizes 8 Compliance Changes, July 2026.
- Hunton Andrews Kurth, CPPA finalizes CCPA regulations on automated decision-making technology.
- Regulation (EU) 2016/679 (GDPR), Article 22.
