Quick answer

Before putting Azure OpenAI into production, disable API key authentication and use managed identities, grant apps only the Cognitive Services OpenAI User role, disable public network access and use private endpoints, attach content filters with Prompt Shields to every deployment, route traffic through API Management with token limits, enable diagnostic logging, and turn on Microsoft Defender for Cloud threat protection for AI services.

Key takeaways

  • API keys are the most common Azure OpenAI weakness. Replace them with managed identities, then set disableLocalAuth to true.
  • Public network access should be disabled for production. Users reach AI through your app or gateway, never the model endpoint directly.
  • Every model deployment needs a content filter, and Prompt Shields wherever the model reads untrusted content.
  • Run an Azure Resource Graph query across all subscriptions. The deployment you don't know about is the one that's misconfigured.

Azure OpenAI inherits a lot of security from Azure itself — but almost none of it is switched on by default. A freshly created resource accepts API keys from anywhere on the internet. This checklist covers the settings we verify in every Azure audit, in the order we recommend fixing them.

Step zero: find every Azure OpenAI resource

You can't secure what you haven't found. Teams often create Azure OpenAI and AI Services resources in sandbox or project subscriptions that never get reviewed. Run this Azure Resource Graph query across your whole tenant (Azure portal → Resource Graph Explorer):

resources
| where type =~ 'microsoft.cognitiveservices/accounts'
| where kind in~ ('OpenAI', 'AIServices')
| project name, kind, subscriptionId, resourceGroup, location,
    publicNetworkAccess = tostring(properties.publicNetworkAccess),
    disableLocalAuth = tobool(properties.disableLocalAuth)

Every row with keys enabled or public access allowed is a finding. Keep the list — you'll work through it below.

Identity: get rid of API keys

  1. Give each app a managed identity. App Service, Functions, Container Apps, AKS and VMs all support system- or user-assigned managed identities.
  2. Grant the least-privilege role. Assign Cognitive Services OpenAI User on the specific resource — not the subscription.
    az role assignment create \
      --assignee <managed-identity-principal-id> \
      --role "Cognitive Services OpenAI User" \
      --scope <azure-openai-resource-id>
  3. Switch the SDK to Entra ID. Use DefaultAzureCredential (or your language's equivalent) and a token provider instead of an API key. Deploy and confirm traffic is flowing.
  4. Disable keys. Once nothing uses them:
    az resource update --ids <azure-openai-resource-id> \
      --set properties.disableLocalAuth=true
  5. Make admin access just-in-time. Put Owner, Contributor and Cognitive Services Contributor behind Microsoft Entra Privileged Identity Management.

Watch out: role assignments can take several minutes to take effect. Test with the managed identity before disabling keys, not after.

Network: take the endpoint off the internet

  1. Create private endpoints for Azure OpenAI and link the privatelink.openai.azure.com private DNS zone to your workload VNets. Do the same for Azure AI Search, Key Vault and any storage accounts holding AI data.
  2. Disable public network access once private connectivity works:
    az resource update --ids <azure-openai-resource-id> \
      --set properties.publicNetworkAccess=Disabled
    If people use the Azure AI Foundry portal from their laptops, plan for that first — through a VPN, a jump host, or a deliberately scoped IP rule.

Guardrails: filters on every deployment

  1. Attach a content filter configuration to every model deployment. Treat any relaxed filter as an exception that needs a documented approval.
  2. Turn on Prompt Shields for user prompt attacks on everything, and for document (indirect) attacks wherever the model reads email, web pages, uploaded files or RAG results. Then test it. See what prompt injection is for the kinds of attacks to try.

Gateway: control who uses how much

  1. Route application traffic through Azure API Management. The gateway authenticates to Azure OpenAI with its own managed identity, so apps never touch the model directly. Add an llm-token-limit policy keyed to each subscription or client, so one runaway script or abusive user can't exhaust your quota or your budget.

Monitoring: see what's happening

  1. Enable diagnostic settings on every Azure OpenAI resource, sending audit and request logs to a Log Analytics workspace with defined retention and restricted read access.
  2. Enable Microsoft Defender for Cloud threat protection for AI services on every subscription running AI, and route its alerts to your SOC or Microsoft Sentinel. It detects things like jailbreak attempts and suspicious access patterns that you won't spot in raw logs.

Lock it in with Azure Policy

Fixing resources once isn't enough — new ones get created every week. Assign Azure Policy definitions at the management group level that audit or deny Azure AI Services accounts with local authentication enabled or public network access allowed. Re-run the Resource Graph query from step zero every quarter to catch drift.

For the full baseline, including data residency, customer-managed keys, Azure AI Search security trimming and agent permissions, see our Azure secure AI deployment baseline.

Frequently asked questions

How do I disable API keys on Azure OpenAI?

Set the disableLocalAuth property to true on the Azure OpenAI or Azure AI Services account, in Bicep/ARM/Terraform or with az resource update --set properties.disableLocalAuth=true. First move every application to Microsoft Entra ID authentication with a managed identity and the Cognitive Services OpenAI User role, or those applications will stop working.

Which Azure RBAC role should an application use to call Azure OpenAI?

Cognitive Services OpenAI User. It allows inference calls without letting the app change deployments, models or account settings. Avoid Cognitive Services Contributor, Contributor or Owner for application identities.

Is Azure OpenAI private by default?

No. A new Azure OpenAI resource accepts traffic from the public internet unless you restrict it. For production, set public network access to Disabled and connect through private endpoints in your virtual network.

Get certified against the leaks covered in this article

CertiSafe AI tests your live AI features every quarter and gives you a verifiable stamp for your website.

Schedule a Meeting