TL;DR: MCP gave AI agents the keys to your infrastructure. Databases, APIs, deployment pipelines, internal services, all accessible through a standardized protocol that 97 million monthly SDK downloads have turned into a de facto standard. The problem: most teams are still running this stuff the way they prototyped it. Direct connections. No centralized auth. Fragmented logs. The result is a security surface that’s already producing real breaches and a governance gap that widens with every new tool server. The fix isn’t complicated, but it requires treating MCP as infrastructure, not a dev tool. That means gateway architecture, scoped permissions, centralized observability, and the kind of operational discipline that nobody thinks about until something goes wrong.

MCP Servers in Production: The Security and Governance Reckoning

Sometime in mid-2025, a support agent built on Supabase’s Cursor integration processed a ticket that looked ordinary. The ticket contained embedded SQL instructions. The agent, running with privileged service-role access, executed them. Sensitive integration tokens ended up in a public support thread.

Nobody had configured it to do that. The agent was doing exactly what MCP enables: reading context, reasoning about a task, calling the tools available to it. The problem was that nobody had thought hard enough about which tools should be available, to whom, and under what constraints.

That incident wasn’t unique. Asana pulled its MCP-powered feature offline for two weeks after customer data started bleeding across tenant boundaries. A critical command-injection vulnerability in mcp-remote, a popular OAuth proxy with over 437,000 downloads, turned unpatched installations into supply chain backdoors. By early 2026, security researchers had cataloged 30 CVEs in 60 days across the MCP ecosystem.

Anyone who lived through the early days of cloud infrastructure or containerization recognizes this pattern. A powerful new abstraction shows up. Developers grab it because it works. Nobody locks anything down because the focus is on shipping, not defending. Then production happens.

MCP is squarely in the “then production happens” phase.

From protocol to production: what actually changed

When Anthropic open-sourced the Model Context Protocol in November 2024, the pitch was elegant: a universal interface for connecting AI models to external tools. Think of it as a USB-C port for AI agents. Plug in a database connector, a GitHub tool, a search API. The model discovers what’s available and starts using it.

The spec defines tool schemas (what actions exist, what parameters they take, what they return) and transport layers (stdio for local processes, HTTP with SSE for remote servers, and the newer Streamable HTTP for production deployments). An agent requests a tool list, picks what it needs, calls it, incorporates the result into its reasoning.

By March 2026, the ecosystem has grown to over 6,400 servers on official registries, monthly SDK downloads past 97 million, and adoption from every major AI lab. OpenAI baked MCP into their Agents SDK. Google confirmed support in Gemini. Microsoft built it into Azure’s AI infrastructure. Anthropic donated the protocol to the Linux Foundation’s Agentic AI Foundation in December 2025.

The numbers tell one story. The architecture most teams are actually running tells another.

The architecture that works on your laptop and nowhere else

Here’s how most MCP deployments start. You have an AI agent (Claude, a custom LLM application, a Cursor-powered coding assistant). You spin up a few MCP servers: one for your database, one for GitHub, maybe one for internal documentation search. The agent connects directly to each server. Each server handles its own logic. Everything works.

This is fine for a developer experimenting on their machine. It becomes a liability the moment a second person gets involved.

The direct-connection pattern creates an N-to-N mesh. Every agent connects to every tool server independently. Each connection has its own auth setup (or, more commonly, no auth at all). Logs scatter across servers. There’s no single place to answer basic operational questions: Which agent called which tool? How many times? What did it cost? Did it access something it shouldn’t have?

A survey by Astrix Security of 2,614 MCP server implementations found that 82% use file operations vulnerable to path traversal attacks. Two-thirds have some form of code injection risk. Over a third are susceptible to command injection. These aren’t theoretical concerns sitting in an academic paper. They’re the actual state of the servers people are connecting to production databases.

The protocol itself shoulders some blame. As of the November 2025 spec revision, MCP still has no standardized approach to client authentication, no multi-user permission model, and no built-in workflow orchestration. The spec tells you how to describe tools. It doesn’t tell you how to control who uses them.

The threat model nobody drew on a whiteboard

Traditional API security assumes a human developer writes code that calls endpoints in a predictable sequence. MCP flips that assumption. The model decides which tools to call, in what order, with what parameters. That decision happens inside a neural network’s reasoning process, not in source code anyone can review.

This reshapes the threat model in ways most security teams haven’t fully absorbed.

Prompt injection goes from annoying to dangerous. When an LLM is just generating text, a prompt injection produces wrong or misleading output. When that LLM has MCP tools attached that can query databases, modify infrastructure, or trigger deployments, a prompt injection produces actions. The Supabase incident was exactly this: user-supplied input interpreted as instructions, executed against real systems with real credentials.

Tool poisoning is the supply chain attack that caught most teams off guard. MCP servers describe their own capabilities via metadata. An attacker who compromises that metadata, or publishes a malicious server to a public registry, can manipulate what the model believes a tool does. Invariant Labs demonstrated this in 2025: a poisoned tool description could trick an agent into escalating privileges or exfiltrating data while appearing to perform a routine operation.

The confused deputy problem shows up here too. MCP servers often run with elevated privileges because they need access to the systems they wrap. But the protocol doesn’t carry user context through the call chain. The server doesn’t know whether it’s acting on behalf of a senior engineer or a junior intern’s misconfigured agent. Everyone gets the server’s full privilege set.

Discovery itself becomes an attack vector. When an agent performs tool discovery, it trusts the schemas it receives. A man-in-the-middle on that discovery request, or a compromised registry, can inject tools the agent will start using without any human review. The model can’t tell the difference between a legitimate search_docs tool and one that shadows it with hidden behaviors.

None of this is unique to MCP. These are the same problems that plagued early microservice architectures, early container orchestration, early cloud deployments. The difference is velocity. MCP went from zero to 97 million monthly downloads in about 16 months. Security maturity hasn’t kept pace.

The gateway pattern: what it actually solves

The architectural response taking shape across the industry is straightforward: stop connecting agents directly to tool servers. Put a gateway in between.

An MCP gateway sits between AI clients and MCP servers, providing a single entry point that handles routing, authentication, authorization, rate limiting, logging, and policy enforcement. From the agent’s perspective, nothing changes. It still discovers tools and calls them. But every request now passes through infrastructure you control.

This isn’t a new idea. API gateways have been standard practice in web services for over a decade. Kong, NGINX, AWS API Gateway. The pattern is well-established. What’s new is applying it to AI tool infrastructure, where the “client” is an autonomous agent that makes its own decisions about what to call.

The gateway converts the N-to-N mesh into a hub-and-spoke model. Agents connect to one endpoint. The gateway routes requests to the appropriate MCP servers based on the tool being invoked. Security policies get evaluated at the gateway before traffic reaches any backend.

Here’s what that buys you in practice:

Centralized authentication and authorization. Instead of each MCP server implementing its own auth (or, realistically, skipping it), the gateway plugs into your existing identity provider. OAuth 2.0, OIDC, SAML, pick your protocol. Role-based access control restricts which tools each agent or team can invoke. Attribute-based access control adds context-aware constraints: this agent can query the staging database but not production, or it can read GitHub issues but not merge PRs.

A single pane of observability. Every tool call, every parameter, every response, every latency measurement, every error, logged in one place. When an agent does something unexpected, you don’t dig through five different server logs to reconstruct what happened. You look at the gateway.

Rate limiting and cost control. Agents are relentless callers. A poorly written loop can generate thousands of tool invocations per minute. Without rate limiting, that’s a runaway cost problem and a potential denial-of-service on your own infrastructure. The gateway enforces request quotas per agent, per team, per tool, per time window.

Request validation. Before a tool call reaches the backend, the gateway validates parameters against the tool’s schema, rejects malformed requests, strips suspicious inputs. It won’t prevent every prompt injection, but it catches the obvious ones.

Circuit breaking and load balancing. When an MCP server goes down, the gateway detects the failure and stops routing traffic to it. No cascading failures. For popular tools, it distributes requests across multiple server instances.

Kong, TrueFoundry, Apache APISIX, and purpose-built solutions like Bifrost and Requesty are all shipping MCP gateway capabilities. The implementations differ. The architectural pattern is converging.

Scoped access: the virtual key model

One pattern gaining traction across gateway implementations is the use of scoped access tokens (sometimes called virtual keys) to encode permissions at the infrastructure level.

The concept is dead simple. Instead of giving every agent an unrestricted API key that can reach every tool, you issue keys that carry policy constraints baked in. A virtual key might encode rules like: maximum $500 monthly spend, access limited to documentation search and staging APIs, rate-limited to 100 requests per minute.

When a request arrives at the gateway, the key’s policy gets evaluated before routing. If the request violates a constraint (calls a forbidden tool, exceeds a budget, trips a rate limit), the gateway rejects it immediately. The tool server never sees the request.

This moves security decisions out of application code and into infrastructure policy. Application developers don’t implement authorization logic. They use the key they’re issued, and the infrastructure handles enforcement.

For organizations managing multiple teams, environments, and agents, this simplifies operations dramatically. A development team gets a key scoped to staging tools. A production automation system gets a read-only key limited to monitoring APIs. A research team gets a key with a generous budget but restricted to search and analysis tools. All enforced at the gateway, auditable in one place, changeable without redeploying anything.

curl -X POST https://gateway.internal/v1/chat/completions \
  -H "x-gateway-key: vk-engineering-staging" \
  -d '{ ... }'

The key header is the only thing the agent needs. Everything else (which tools it can access, how much it can spend, how many requests it can make) lives in the gateway’s policy engine.

What observability actually looks like at scale

Running MCP tools in production without centralized observability is like operating a distributed system with print statements. You’ll figure out what went wrong eventually, but “eventually” might mean hours of cross-referencing timestamps across different log formats.

A gateway-based architecture captures structured metadata for every interaction:

• Request traces: Which agent called which tool, with what parameters, what it got back. Full request-response pairs with correlation IDs for tracing multi-step reasoning chains.
• Token and cost accounting: How many tokens each tool call consumed, what it cost, broken down by agent, team, and tool. This is the metric that caught a $14,000/month agent that looked healthy by every other measure.
• Latency profiling: Per-tool response times, percentile distributions, degradation alerts. When your document search MCP server starts responding in 3 seconds instead of 300 milliseconds, you want to know before users do.
• Error classification: Not just HTTP status codes, but semantic errors. The tool returned a result but the result was empty. The tool was called with invalid parameters that passed schema validation but made no logical sense.
• Access patterns: Which tools see heavy use, which are dead weight (and can be decommissioned), which show unusual spikes that might signal misuse.

Some organizations build this on OpenTelemetry, which gives them compatibility with existing distributed tracing infrastructure. Others use gateway-native dashboards. The specifics matter less than the principle: if you can’t see what your agents are doing, you can’t govern them.

The performance question

A common objection to the gateway pattern is latency. Adding a hop between agent and tool server means adding milliseconds to every tool call. For agents that chain multiple tools in a single reasoning step, those milliseconds add up.

In practice, the overhead is smaller than most people fear. TrueFoundry reports sub-3 millisecond latency for authentication and rate limiting when these operations happen in memory. Go-based gateways like Bifrost are built for high-concurrency workloads, handling thousands of requests per minute while enforcing policies and logging everything.

The more interesting performance consideration is what happens without a gateway. When every application implements its own connection pooling, retry logic, and failover behavior, you get inconsistent performance across your fleet. A gateway centralizes those concerns. Connection management, health checking, circuit breaking, load balancing: done once, in infrastructure purpose-built for it, instead of being reimplemented (often poorly) in every application.

For organizations running large agent fleets (dozens of agents, each making hundreds of tool calls per task), the gateway also enables horizontal scaling of the tool infrastructure. Multiple instances of popular MCP servers sit behind the gateway’s load balancer. The agent doesn’t know or care how many instances exist. It sends requests to one endpoint and gets results back.

When you don’t need any of this

Not every MCP deployment needs a gateway. Not every project needs centralized governance.

If you’re a solo developer running Claude Desktop with a couple of MCP servers on your laptop, direct connections are fine. If you’re prototyping an agent workflow that only touches test data, the overhead of a gateway is wasted effort.

The inflection point arrives when any of these become true:

• Multiple people or teams share the same MCP tool infrastructure
• Agents interact with production systems holding real data
• You need to answer “what did our agents do last Tuesday?”
• Someone asks how much the AI tooling costs this month
• Compliance or security policy requires audit trails for automated system access

Once you cross that line, retrofitting governance is harder than building it in from the start. The organizations that figured this out early, the ones that treated MCP as infrastructure from day one, spend their time building. Everyone else spends it firefighting.

The road from here

The MCP ecosystem is maturing fast. The 2026 roadmap from the protocol maintainers addresses several gaps that make production deployment painful: better support for stateless operations that play nicely with load balancers, server identity mechanisms for verifying authenticity, and a community-driven registry for discovering vetted servers.

But protocol improvements alone won’t close the governance gap. The protocol defines what agents can do. Governance defines what they should do. That distinction lives in infrastructure, organizational policy, and operational discipline, not in a spec document.

The trajectory mirrors what happened with containers. Docker made packaging applications trivially easy. Kubernetes made orchestrating them possible. But neither solved security, access control, or compliance. Those required an ecosystem of tools, practices, and organizational habits built on top of the core technology.

MCP is Docker in this analogy. Gateways, registries, and governance tooling are the ecosystem being assembled around it. The organizations investing in that layer now will run AI agents safely at scale. Everyone else will learn the hard way, probably from an incident report that lands on the wrong desk at the wrong time.

Your team needs MCP governance. The only question is whether you build it before or after the first breach.