The rise of AI Gateways
Generative AI has opened up a wide array of applications that can enhance business operations and everyday life. Common use cases include generating personalized content and deploying instant chatbots capable of handling complex questions using your specific data.
However, the democratization of large language models (LLMs) through Model-as-a-Service offerings presents several challenges, including issues related to cost, security, and governance. These challenges can slow down or even completely block adoption. Building LLMs is complex, therefore costly and time-consuming. Fortunately, access to these intelligent services is mostly provided via web APIs, making integration simple and standardized.
Bottom line: it seems we are back to a familiar question—how can we effectively accelerate and manage the access to (AI) APIs, understand their use, and prevent misuse?
Enter the AI Gateway Pattern
Answer: delegate the governance, security and observability cross-cutting concerns of AI applications to a central AI Gateway to prevent developers reinventing the wheel which can lead to higher fragmentation and cost resulting in lower efficiency.
It sounds like an API Gateway isn’t it?
Yes, it is. The role of the gateway is not changed at all. Essentially, the narrative has shifted. So far we were using an API management solution to govern, secure and optimize our API program. Now, we can expand its functionality to govern, secure, and optimize access to AI services.
- If you are offering shared AI services, the AI Gateway essentially serves as an extension of the API Gateway, incorporating AI-specific functionalities.
- For those accessing external AI services, the AI Gateway operates as an egress gateway, proxying all the traffic generated by your intelligent applications.
From a technology standpoint, we are experiencing a new boost in the API economy. New AI gateways are emerging, while existing API management solutions are responding to market demands by continually integrating additional features to address the evolving challenges posed by AI.
[
API Gateway](https://raw.githubusercontent.com/MassimoC/apimhotrod/main/blog/api-gateway/AIGateway.png)
The AI Gateway in action
The uses cases for the AI gateway are diverse, including intelligent load balancing, model routing, prompt decorator, token rate limiting, cost tracking, prompt guarding and more. Let’s now see how we can use Azure API Management (APIM) as an AI gateway by exploring some practical scenarios for each of these cross-cutting areas: governance, security and observability.
No need to start from scratch, there is an excellent Microsoft repository with multiple AI Gateway labs available that you can mix and match as you prefer.
For this exercise, I took the Vector Search lab, merged with the built-in logging to correlate the different calls of the RAG application and created a new security policy. The access optimization to AI services is based on the excellent APIM smart load balancing solution at this repo.
Governance
Let’s start from a very common need: increase resiliency and optimize the usage of the provisioned AI capacity. Often AI workloads need to leverage multiple Azure Open AI instances with different priorities. The key to streamlining this process is :
- Prioritize PTU instances - Utilize all tokens in a Pre-Allocated Throughput Unit (PTU) instance before routing requests to other consumption instances as a fallback. Since PTUs are paid for regardless of usage, they should be prioritized.
- Distribute requests - Spread requests across token-based consumption instances to alleviate throttling.
- Implementing tiered prioritization - Route requests to multiple consumption instances based on priority, such as using geographically closer instances first.
The APIM smart load balancing policy enhanced in the AI Gateway repository provides all the necessary features for our scenario. It leverages priority groups to switch from PTU to token-based instances and transparently manage a list of multiple fallback endpoints eliminating the need for tedious client-side configuration.
[
AI load balancing](https://raw.githubusercontent.com/MassimoC/apimhotrod/main/blog/api-gateway/intro-loadbalance.png)
Observability
Observability is crucial for all applications, including intelligent apps powered by AI services. By proxying AI calls, we gain a unified view of usage, performance, cost, health and more. Enabling these insights is straightforward with Azure API Management. By simply connecting your APIM instance to a Log Analytics workspace you get access to performance, data for debugging, troubleshooting, and the metrics necessary to ensure reliability and availability.
It is very common to create reports to track the PTU and token usage and enable alerting and notification for deviations from expected behavior .
In this exercise, we leveraged Application Insights to analyze the API calls sequence in the RAG (OpenAI embedding, Azure search and OpenAI completions), the durations and the logged requests and responses.
[
RAG end to end visibility](https://raw.githubusercontent.com/MassimoC/apimhotrod/main/blog/api-gateway/ApplicationInsights.png)
Security
Security is the main concern of every enterprise, but the amount of sensitive data and intellectual property going into AI services, elevates the need for security to a whole new level. Therefore the AI gateway plays an extremely important role by securing the LLMs access, detecting malicious prompt, block prompts with inappropriate terms, filter Personally Identifiable Information (PII) and more.
As a final step of this exercise I asked Azure OpenAI to write a prompt guarding policy for APIM that blocks requests that contains prohibited words. Here what I got, it’s pretty simple but it works like a charm for a demo. Make it more dynamic and customizable will be for another round.
[
APIM prompt guard](https://raw.githubusercontent.com/MassimoC/apimhotrod/main/blog/api-gateway/PromptGuarding.png)
Conclusion
The emergence of AI gateways, akin to API gateways but tailored for AI services, helps developers and enterprises approach AI in a safer and more controlled manner. Leveraging Azure API Management (APIM) as an AI gateway offers practical solutions for governance, security, and observability needs, but writing custom policies still requires considerable effort. I hope to see more AI-dedicated out-of-the-box policies to make the developers’ lives easier and further accelerate AI adoption.
As AI gateways continue to evolve, their role in optimizing access to AI services and ensuring their secure and efficient utilization becomes increasingly vital in the modern digital landscape: API management solutions are living a second life in the form of AI gateways, reclaiming their place at the forefront of innovation!