An AI Agent With Permission to Act Is a Security Architecture Problem

An AI assistant that answers a question is one kind of risk. An AI agent that can read a customer database, send messages, change permissions, and execute workflows is another. The difference isn't philosophical. It's the difference between receiving questionable advice and allowing software to act on it.

The mistake I worry about is treating a model's apparent competence as evidence that it deserves broad operational authority. A system can be extraordinarily helpful and still be wrong, misled, or unable to explain why it chose a particular action. NIST's AI Risk Management Framework asks organizations to consider trustworthiness across governance, mapping, measurement, and management, while its generative-AI profile addresses additional risks associated with these systems (Tabassi, 2023; Autio et al., 2024). Those are useful starting points, but they don't make an authorization decision for you.

I would inventory every tool, credential, data source, and action the agent can reach. Can it only read a document, or can it update the source? Can it draft a message, or can it send one? Can it suggest a configuration change, or can it deploy it? Can it grant access to another identity? These differences are easy to blur in a product demonstration and impossible to ignore after an unwanted action.

A capability inventory should include the systems affected, the privilege level, the approval path, the likely blast radius, and the practical ability to undo a change. An agent that summarizes tickets may need read access to selected fields. It doesn't follow that the same agent should close tickets, create privileged accounts, or export customer records. The principle of limiting access to the resources needed for a task is consistent with NIST's zero-trust architecture, which rejects implicit trust based on network location or ownership (Rose et al., 2020).

The strongest boundary is one the model cannot negotiate away. If a task requires read-only access, enforce read-only access in the connected service. Don't rely on a sentence in a system prompt saying 'please don't modify anything.' Instructions are useful for behavior. Authorization belongs in the systems that grant or deny actions.

A model may encounter instructions in documents, websites, emails, logs, or search results. Those materials are often the very things the agent is supposed to analyze. They should not be allowed to redefine the agent's authority. The danger is that the agent may treat content from a lower-trust source as an instruction to perform a higher-trust action.

I would separate the user request, the model's proposed plan, the retrieved material, and the tool invocation. Those are different trust levels, even if a convenient interface presents them as one conversation. A retrieved page can supply facts to consider; it should not supply new permissions. NIST's generative-AI profile identifies information integrity and related risks as concerns requiring governance and measurement (Autio et al., 2024). The operational implication is straightforward: external content must not become a substitute administrator.

For a workflow that can make meaningful changes, I want deterministic checks between the proposed action and its execution. A tool gateway can validate the requested operation, target resource, scope, and policy before forwarding it. If the request falls outside the authorized task, it fails even when the model argues persuasively that the exception would be helpful.

Putting a human approval button into a workflow doesn't automatically create human control. If the person sees a vague summary, cannot inspect the exact proposed action, or is asked to approve hundreds of repetitive requests, the control becomes ceremonial. The same is true when the system proceeds after a timeout and treats silence as consent.

For high-impact operations, approval should display the actual change, affected resources, expected consequences, and rollback path. A reviewer should be able to reject the operation without breaking the entire application. There should be a clear record of who authorized what, and the agent should not be able to modify that record. NIST's security-control catalog addresses access control, audit and accountability, configuration management, and system integrity as separate functions (Joint Task Force, 2020). Agentic systems need those familiar controls, not a special exemption because their interface looks conversational.