Govern AI Before Scaling It: People, Agents, and Tools
In organizations that are rapidly adopting generative AI—and especially in fintech and payments, where PCI DSS and financial regulations are non-negotiable—a very similar story is unfolding. First, each team builds its own assistant, support agent, or automation tool, and before long, no one has a complete inventory of which agents exist or what they can do. Many of these agents inherit broad permissions from the application that hosts them, or use shared credentials to access APIs and tools, so access exists but no one has explicitly defined it.
Added to this is growth. When a company acquires another, it also takes on its cloud accounts, its identity provider, and its own compliance standards—and what was once a segmented environment now becomes a series of silos. New needs also arise, such as taking an agent designed for end customers and offering it to B2B clients or partners as well, which requires separating data and permissions between customers. Meanwhile, the security team needs a comprehensive view of the organization’s security posture—a view that, in most cases, is still put together manually.
The bottom line is that AI governance is, above all, an issue of identity and access. It does not require the invention of new disciplines: it is solved using the same principles we already apply in the cloud—such as a single identity plane, least privilege, preventive controls, and centralized auditing—now extended to a new type of actor.
But it’s not just a matter of security. A production agent also places an additional burden on the platform: it consumes models and computing resources, depends on integrations with other systems, and can degrade critical services if it isn’t well designed. That’s why government operations run more smoothly when Security and IT work together as validators and reviewers of what the teams build: Security oversees access and data, while IT manages the cost, performance, stability, and quality of the integrations each agent requires. But how does all of this scale without Security and IT becoming a bottleneck?
Five-Layer Framework
Layer 1 — A multi-account foundation that accommodates growth
It all starts with organizing where workloads reside. An AWS organization managed according to the Well-Architected Framework allows accounts to be grouped by company or business unit and separated by environment, so that each new acquisition is integrated following a standard process rather than remaining an isolated entity with its own set of rules. Within that structure, non-negotiable limits are defined—such as permitted regions, which AI services can be used, or the requirement that auditing cannot be disabled—and these limits are proactively enforced across the entire organization.
This layer is also the best tool for narrowing the scope of compliance. If the workloads that process card data reside on dedicated accounts, the scope of a PCI audit is limited to those accounts, and the rest of the organization can innovate more freely.
Layer 2 — A single point of entry for people
User identities should be managed in a single location. The corporate identity provider—whatever it may be—is federated (for example, with IAM Identity Center), so adding or removing a user only needs to be done once and is reflected across all accounts and tools. Permissions are assigned by role and by group, never on an individual basis.
The key is for AI tools for business to come through the same door. If an assistant like Amazon Quick uses that same identity, permissions for spaces, data, and agents are derived from existing groups, and the security team governs the AI using the same model it uses to govern everything else.
Layer 3 — Agents are also identities
This is the most important shift in mindset. An agent that queries a database, calls an API, or invokes a tool through MCP is not an invisible extension of the application: it is an actor that requires its own identity, minimal permissions, and a record of its actions. Furthermore, when an agent acts on behalf of a user, it should carry that user’s context with it, so that it can never access anything the user would not be able to see.
In practice, this is solved using a common platform to host the agents—such as Amazon Bedrock AgentCore—which assigns them identities, stores their credentials for external services without hard-coding them, and centralizes access to tools through a gateway. Instead of each agent connecting directly to each MCP, they all pass through a control point where the caller is authenticated and their permissions are verified. As a result, the question “Which agent can invoke which tool, and on whose behalf?” is answered by looking at a configuration, not by reading code.
The same principle resolves the case of an agent shared among multiple clients. If each client’s identifier is embedded in the user’s identity and the tools filter the data on the backend, separation is guaranteed by the access layer and does not depend on the model following an instruction in the prompt.
Layer 4 — Constraints on Model Behavior
Identity defines what an agent can access; guardrails define what the agent can say and do with that information. Filtering out prohibited topics, detecting and masking personal data in inputs and outputs, and adjusting those controls based on the use case—with stricter controls for an agent speaking with an end customer than for an internal one—is what enables us to deploy AI with confidence. Bedrock Guardrails allows you to define these policies once and apply them to different models and agents.
The other half of this layer involves keeping sensitive data out of the model’s reach. Tokenizing card or account numbers before they reach the agent or are used to train a model reduces the risk of exposure without losing value: consistent tokenization—where the same value always generates the same token—still allows for grouping transactions or detecting anomalous behavior without anyone seeing the actual data.
Layer 5 — Visibility for Security and IT
None of the above is useful if it cannot be audited. A centralized log of all actions—including model invocations and what each agent does—makes it possible to reconstruct what happened, on whose behalf, and with what data. Building on that foundation, consolidating security findings and compliance controls from all accounts in one place—for example, using Security Hub and Security Lake—gives the security team the big-picture view that they currently have to piece together manually.
The same logic applies to IT. Measuring latency, errors, and token consumption for each agent, and allocating costs by team or business unit through resource tagging, makes it possible to detect early on an agent that has become expensive, slow, or unstable, or an integration that is becoming overloaded. Agent observability—using tools such as CloudWatch and AgentCore Observability—along with cost management, is just as much a part of governance as access auditing.
Presenting that information on a dashboard that shows the status by company and by account changes the conversation with management: governance goes from being a vague concept to becoming a metric that can be tracked over time.
Checklist: Seven Questions Security and IT Should Be Able to Answer
- Do we have an inventory of all agents in production, including their owners and purposes?
- Does each agent have its own identity and minimum permissions, or does it inherit them from its application?
- Do agents access tools and MCPs through an auditable access point?
- Do user onboarding and offboarding take place within a single IdP that also manages the AI tools?
- Can we reconstruct what an agent did, on whose behalf, and using what data?
- Does every agent have a business metric that justifies their cost?
- Do we know how much each agent and its integrations cost, how they perform, how stable they are, and who is responsible if they experience a service degradation?
If any answer is “I don’t know,” that’s the starting point.
You can find other action-oriented questions here.
Government as an Enabler
A common mistake is to think that governance means restriction. Experience shows us the opposite: the organizations that put AI into the hands of the business the fastest are the ones that first addressed identity, guardrails, auditing, and observability. Once the framework is in place, Security and IT can say “yes” to new use cases without having to evaluate each one from scratch. The challenge is to establish a framework for all of this while the business continues to move forward.
And while we’re focusing on the technical side here, the same rule applies to every initiative: no feature should go into production without a business outcome and a metric to measure it. Effective leadership also means knowing what value each feature brings.
