- Published on
Azure API Management Adds Dedicated AI Gateway Tier, Governing Models and MCP Tools
- Authors
- Name
- DeepHorus
Microsoft has introduced a dedicated AI Gateway tier for Azure API Management, now in public preview, designed to govern AI models and tool ecosystems rather than traditional APIs. The new gateway resource organizes its control plane around models, MCP servers, and tools, reflecting a shift from single-provider integrations to multi-provider AI architectures. Teams now front many providers instead of one, and this tier is built as a separate experience rather than additional policies on the existing gateway.
The preview supports Foundry-hosted models from OpenAI, Anthropic, and Mistral, alongside models accessed through AWS Bedrock, Google Vertex AI, and OpenAI directly. All OpenAI-compatible providers share a single endpoint path, with routing based on exact model name matches, so each published model needs a unique name. Anthropic is integrated via a custom provider using Messages API passthrough. Policies are configured through visual cards instead of XML and expressions, covering token limits, request quotas, Content Safety, and model fallback.
A gateway provisions in about a minute with no scale units to plan. Telemetry exports OpenTelemetry token metrics to Application Insights, Datadog, Grafana, or other customer-controlled destinations, while the resource runs in the customer's own subscription and Entra tenant. For tools, the gateway federates backends from remote MCP servers, OpenAPI specifications, or built-in connectors covering more than a thousand SaaS applications with no server to host. Each backend's operations become tools, with authentication options including none, API key, OAuth 2.0, or managed identity.
The intended operating model separates central control from team self-service. A platform group connects and publishes approved models and tools; application teams exercise these assets in a test console without routing every change through the center, while the platform group keeps guardrails and the usage picture. Reactions from architects and platform engineers have been broadly positive on consolidation, though questions remain about the governance boundary.
Paolo Perrone of the AI Engineer newsletter highlighted the underrated value of putting cost governance at the gateway, calling it “one control plane instead of per-app patches.” Adolph White Jr., an enterprise AI systems architect, asked whether agent outputs from incomplete runs are preserved for auditable review or if the gateway fails over and retries, framing the distinction between governing AI traffic and governing the full lifecycle. The announcement does not address whether authority over what an agent's output may change belongs at the gateway or the orchestration layer above it.
Not all feedback was favorable. Sreenivasulu Kandakuru of Aer Lingus called the tier badly needed but judged Azure to be behind AWS and Databricks, citing Unity AI Gateway announced at Data + AI Summit in June. However, Microsoft has shipped AI gateway capabilities for roughly two years, so the gap is narrower than that comparison implies. One design decision warrants scrutiny: a runtime access key is gateway-scoped, reaching every model and every tool, which may require careful planning before onboarding teams.
