diff --git a/en/docs/ai-gateway/next/README.md b/en/docs/ai-gateway/next/README.md index bc1a1e06d..a53ebe096 100644 --- a/en/docs/ai-gateway/next/README.md +++ b/en/docs/ai-gateway/next/README.md @@ -19,7 +19,7 @@ A gateway for managing and securing AI traffic, including Large Language Model ( ## Quick Start - [LLM Quick Start Guide](quick-start-guide.md) — Set up the gateway and route traffic to LLM providers like OpenAI -- [MCP Quick Start Guide](mcp-proxy/quick-start-guide.md) — Set up the gateway and route traffic to MCP servers +- [MCP Quick Start Guide](mcp-proxy/create-an-mcp-proxy.md) — Set up the gateway and route traffic to MCP servers ## Key Concepts @@ -35,6 +35,7 @@ Following templates are shipped out-of-the-box - AWS Bedrock - Azure AI Foundry - Gemini +- MistralAI ### LLM Provider @@ -49,7 +50,7 @@ An LLM Provider represents a connection to an AI backend service such as OpenAI, Once configured, the LLM Provider allows traffic to flow through the gateway to the AI backend. -To connect the gateway to AWS Bedrock, see [Configure an AWS Bedrock LLM Provider](llm-proxy/configure-aws-bedrock-provider.md). The guide covers both Bedrock bearer API keys and AWS Signature Version 4 (SigV4) authentication. +To connect the gateway to AWS Bedrock, see [Configure an AWS Bedrock LLM Provider](connect-llm-providers/supported-providers/aws-bedrock.md). The guide covers both Bedrock bearer API keys and AWS Signature Version 4 (SigV4) authentication. ### LLM Proxy diff --git a/en/docs/ai-gateway/next/llm-proxy/llm-templates.md b/en/docs/ai-gateway/next/connect-llm-providers/llm-templates.md similarity index 93% rename from en/docs/ai-gateway/next/llm-proxy/llm-templates.md rename to en/docs/ai-gateway/next/connect-llm-providers/llm-templates.md index 2177d7a88..e06a11706 100644 --- a/en/docs/ai-gateway/next/llm-proxy/llm-templates.md +++ b/en/docs/ai-gateway/next/connect-llm-providers/llm-templates.md @@ -1,14 +1,14 @@ --- title: "LLM Provider Templates" description: "Reference for LLM Provider Templates in API Platform AI Gateway, covering built-in templates for OpenAI, Anthropic, Gemini, and more." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/llm-templates/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/llm-templates.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/llm-templates/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/llm-templates.md tags: - ai-gateway - llm - reference author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 +last_updated: 2026-08-11 content_type: "reference" --- @@ -22,6 +22,8 @@ LLM Provider Templates define the characteristics and behaviors specific to an A - **Model Information**: Request and response model identifiers - **Rate Limiting Data**: Remaining token allowances from response +This page is the reference for that extraction mechanism: which templates ship, what each one reads out of a provider response, and how to define a template of your own. It does not carry the settings that connect the gateway to a provider. For an upstream URL, authentication and a working provider configuration, see [OpenAI](supported-providers/openai.md), [Anthropic](supported-providers/anthropic.md), or [AWS Bedrock](supported-providers/aws-bedrock.md). + ## Out-of-the-Box Supported Templates The API Platform Gateway ships with the following pre-configured LLM provider templates that platform administrators can use immediately without any additional configuration: @@ -86,7 +88,7 @@ Templates support four types of extraction locations: ### OpenAI -The OpenAI template extracts metadata from OpenAI API responses. +The OpenAI template extracts metadata from OpenAI API responses. To connect the gateway to OpenAI, see [OpenAI](supported-providers/openai.md). ```yaml apiVersion: gateway.api-platform.wso2.com/v1 @@ -172,7 +174,7 @@ spec: ### Anthropic -The Anthropic template extracts metadata from Anthropic Claude API responses. +The Anthropic template extracts metadata from Anthropic Claude API responses. To connect the gateway to Anthropic, see [Anthropic](supported-providers/anthropic.md). ```yaml apiVersion: gateway.api-platform.wso2.com/v1 @@ -271,7 +273,7 @@ spec: ### AWS Bedrock -The AWS Bedrock template is designed for the AWS Bedrock unified API. To configure and deploy a provider with bearer or AWS Signature Version 4 (SigV4) authentication, see [Configure an AWS Bedrock Large Language Model provider](configure-aws-bedrock-provider.md). +The AWS Bedrock template is designed for the AWS Bedrock unified API. To connect the gateway to Bedrock with either bearer or AWS Signature Version 4 (SigV4) authentication, see [AWS Bedrock](supported-providers/aws-bedrock.md). ```yaml apiVersion: gateway.api-platform.wso2.com/v1 diff --git a/en/docs/ai-gateway/next/connect-llm-providers/overview.md b/en/docs/ai-gateway/next/connect-llm-providers/overview.md new file mode 100644 index 000000000..573817443 --- /dev/null +++ b/en/docs/ai-gateway/next/connect-llm-providers/overview.md @@ -0,0 +1,47 @@ +--- +title: "Connect LLM providers" +description: "Connect the AI Gateway to an LLM backend: what an LLM Provider holds, who configures it, and the OpenAI, Anthropic, and AWS Bedrock connection guides." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/overview/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/overview.md +tags: + - ai-gateway + - llm-provider + - llm +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-13 +content_type: "concept" +--- + +# Connect LLM providers + +An LLM Provider represents a connection to an AI backend service such as OpenAI, Azure OpenAI, or other LLM APIs. Platform administrators configure LLM Providers to define: + +- The LLM Provider Template +- The upstream LLM service URL +- Authentication credentials (API keys, tokens) +- Access control rules for which endpoints are exposed +- Budget control policies, such as token-based rate limiting +- Organization-wide policies such as guardrails + +Once configured, the LLM Provider allows traffic to flow through the gateway to the AI backend. + +## Who configures this + +Platform administrators own LLM providers. An administrator holds the upstream credentials, decides which endpoints the provider exposes, and sets the organization-wide policies that every LLM proxy consuming the provider inherits. + +## In this section + +The three provider pages carry connection documentation: the upstream URL, the authentication each provider expects, and a working definition you can deploy as it stands. The gateway ships more provider templates than these three; [Provider templates](llm-templates.md) lists every one it loads at startup. + +This section contains the following pages: + +| Page | What it covers | +|------|----------------| +| [OpenAI](supported-providers/openai.md) | Connect the AI Gateway to OpenAI: the upstream URL, API key authentication, the endpoints the provider exposes, and a request that tests the connection. | +| [Anthropic](supported-providers/anthropic.md) | Connect the AI Gateway to the Anthropic Messages API: the upstream URL, x-api-key authentication, and the endpoint the provider exposes. | +| [AWS Bedrock](supported-providers/aws-bedrock.md) | Connect API Platform AI Gateway to AWS Bedrock using a bearer API key or AWS Signature Version 4 authentication, then invoke a model through the gateway. | +| [Provider templates](llm-templates.md) | Reference for LLM Provider Templates in API Platform AI Gateway, covering built-in templates for OpenAI, Anthropic, Gemini, and more. | + +## Related guides + +- [Set up a governed multi-model LLM proxy](../../../guides/ai-and-mcp/set-up-a-governed-multi-model-llm-proxy-with-cost-controls-and-failover.md) — adds Azure OpenAI as an LLM provider, then distributes requests across models behind a single proxy. diff --git a/en/docs/ai-gateway/next/connect-llm-providers/supported-providers/anthropic.md b/en/docs/ai-gateway/next/connect-llm-providers/supported-providers/anthropic.md new file mode 100644 index 000000000..a64c772a0 --- /dev/null +++ b/en/docs/ai-gateway/next/connect-llm-providers/supported-providers/anthropic.md @@ -0,0 +1,94 @@ +--- +title: "Anthropic" +description: "Connect the AI Gateway to the Anthropic Messages API: the upstream URL, x-api-key authentication, and the endpoint the provider exposes." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/supported-providers/anthropic/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/supported-providers/anthropic.md +tags: + - ai-gateway + - llm-provider + - anthropic + - llm +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-13 +content_type: "how-to" +--- + +# Anthropic + +Connect the AI Gateway to the Anthropic API. You end up with an LLM Provider that holds your Anthropic credentials, exposes the Messages endpoint through the gateway, and can be consumed by any LLM proxy. + +This page is for platform administrators, who hold the upstream credentials. + +## Connection details + +The `anthropic` template and these connection settings apply to every Anthropic provider you create: + +| Setting | Value | +|---------|-------| +| Template ID | `anthropic` | +| Upstream URL | `https://api.anthropic.com` | +| Auth type | `api-key` | +| Auth header | `x-api-key` | + +Anthropic reads the key from a dedicated `x-api-key` header, so the value is the key on its own with no prefix. + +## Configure the provider + +Replace *``* with your Anthropic API key and deploy the provider: + +```bash +curl -X POST http://localhost:9090/api/management/v1/llm-providers \ + -H "Content-Type: application/yaml" \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + --data-binary @- <<'EOF' +apiVersion: gateway.api-platform.wso2.com/v1 +kind: LlmProvider +metadata: + name: anthropic-provider +spec: + displayName: Anthropic Provider + version: v1.0 + template: anthropic + context: /providers/anthropic + upstream: + url: https://api.anthropic.com + auth: + type: api-key + header: x-api-key + value: + accessControl: + mode: deny_all + exceptions: + - path: /v1/messages + methods: [POST] +EOF +``` + +The `accessControl` block restricts which upstream paths the provider exposes. This provider denies everything except `POST /v1/messages`. + +The `context` value sets the URL prefix the provider answers on, so this provider serves the Messages endpoint under `/providers/anthropic`. + +## Test the provider + +Anthropic providers are exercised through an LLM proxy rather than called directly, because the proxy translates between the OpenAI request format your application sends and the Anthropic Messages format the upstream expects. + +To build that proxy and send it a request, see [Route across multiple providers](../../expose-llms/multi-provider-routing.md), which deploys this provider alongside an OpenAI one and selects between them with a request header. + +## Supported models + +Anthropic publishes its model list in the [Anthropic models reference](https://docs.anthropic.com/en/docs/about-claude/models). Any model Anthropic exposes on the endpoint above works through the gateway; set it in the `model` field of your request. + +## Provider-specific notes + +Two settings differ from a provider that follows the OpenAI wire format. Both are properties of the Anthropic API, not gateway options: + +- **The auth header is `x-api-key`, not `Authorization`.** Sending the key in `Authorization` fails at the upstream. +- **The upstream URL carries no version segment.** Anthropic puts the version in the request path, so `https://api.anthropic.com` combines with the `/v1/messages` path from `accessControl`. Adding `/v1` to the URL produces a doubled `/v1/v1/messages` path. + +When an LLM proxy routes to this provider, the proxy applies a transformer and may add its own authentication between the two. Those settings belong to the proxy, not to the provider shown here. + +## Related pages + +- [Provider templates](../llm-templates.md) — the token and model metadata the `anthropic` template extracts. +- [Route across multiple providers](../../expose-llms/multi-provider-routing.md) — the proxy that consumes this provider. +- [OpenAI](openai.md) — the same setup for a provider that uses `Authorization` and the OpenAI wire format. diff --git a/en/docs/ai-gateway/next/llm-proxy/configure-aws-bedrock-provider.md b/en/docs/ai-gateway/next/connect-llm-providers/supported-providers/aws-bedrock.md similarity index 91% rename from en/docs/ai-gateway/next/llm-proxy/configure-aws-bedrock-provider.md rename to en/docs/ai-gateway/next/connect-llm-providers/supported-providers/aws-bedrock.md index 109a39f7a..6402da4eb 100644 --- a/en/docs/ai-gateway/next/llm-proxy/configure-aws-bedrock-provider.md +++ b/en/docs/ai-gateway/next/connect-llm-providers/supported-providers/aws-bedrock.md @@ -1,19 +1,20 @@ --- -title: "Configure an AWS Bedrock Large Language Model provider" +title: "AWS Bedrock" description: "Connect API Platform AI Gateway to AWS Bedrock using a bearer API key or AWS Signature Version 4 authentication, then invoke a model through the gateway." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/configure-aws-bedrock-provider/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/configure-aws-bedrock-provider.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/supported-providers/aws-bedrock/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/supported-providers/aws-bedrock.md tags: - ai-gateway + - llm-provider - llm - aws-bedrock - authentication author: WSO2 API Platform Documentation Team -last_updated: 2026-08-04 -content_type: "guide" +last_updated: 2026-08-13 +content_type: "how-to" --- -# Configure an AWS Bedrock Large Language Model provider +# AWS Bedrock Connect API Platform AI Gateway directly to the regional AWS Bedrock Runtime endpoint. You can authenticate the gateway to Bedrock in either of these ways: @@ -22,6 +23,19 @@ Connect API Platform AI Gateway directly to the regional AWS Bedrock Runtime end Both methods expose the native Bedrock `Converse` and `ConverseStream` operations through the gateway. They use the regional Bedrock Runtime endpoint, not the Bedrock Mantle endpoint. For a base model ID, choose an AWS Region where that model is available, and use the same Region in the Bedrock endpoint, SigV4 policy, and model ID. For an inference profile ID, invoke Bedrock through the source Region endpoint where the profile is supported; the profile may route requests to destination Regions. In this case, the SigV4 Region must match the source Region in the endpoint. +## Connection details + +The `awsbedrock` template and these connection settings apply to every Bedrock provider you create: + +| Setting | Value | +|---------|-------| +| Template ID | `awsbedrock` | +| Upstream URL | `https://bedrock-runtime.${AWS_REGION}.amazonaws.com` | +| Auth type | `api-key` for a bearer key, or none for SigV4 | +| Auth header | `Authorization` | + +Under SigV4, leave `spec.upstream.auth` unset. The `aws-authentication` policy builds the `Authorization` header itself, as described in [Option 2: SigV4 authentication](#option-2-sigv4-authentication). + ## Before you begin Make sure that: @@ -465,6 +479,10 @@ URL-encode the model ID if it contains characters that are not safe in a URL pat If the gateway uses a certificate signed by a public or locally trusted certificate authority (CA), omit `--cacert`. For a local self-signed certificate, add the issuing CA certificate to your trust store or pass it with `--cacert`. +## Supported models + +AWS publishes its model list in the [AWS Bedrock supported models reference](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html). Any model Bedrock exposes in the configured Region works through the gateway; set it as the model ID in the request path, as `BEDROCK_MODEL_ID` above. + ## Troubleshooting ### Bedrock returns `AccessDeniedException` @@ -495,3 +513,9 @@ Check both sides of the relationship: the source identity must be allowed to cal - Restrict IAM permissions to the required model and inference profile resources. - Expose only the required Bedrock operations through `accessControl`. - Use HTTPS for Bedrock and production gateway endpoints. + +## Related pages + +- [Provider templates](../llm-templates.md) — the token and model metadata the `awsbedrock` template extracts. +- [Route across multiple providers](../../expose-llms/multi-provider-routing.md) — put an LLM proxy in front of this provider and route to others alongside it. +- [OpenAI](openai.md) and [Anthropic](anthropic.md) — the same setup for providers that authenticate with a single API key header. diff --git a/en/docs/ai-gateway/next/connect-llm-providers/supported-providers/openai.md b/en/docs/ai-gateway/next/connect-llm-providers/supported-providers/openai.md new file mode 100644 index 000000000..f11d91f0b --- /dev/null +++ b/en/docs/ai-gateway/next/connect-llm-providers/supported-providers/openai.md @@ -0,0 +1,103 @@ +--- +title: "OpenAI" +description: "Connect the AI Gateway to OpenAI: the upstream URL, API key authentication, the endpoints the provider exposes, and a request that tests the connection." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/supported-providers/openai/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/supported-providers/openai.md +tags: + - ai-gateway + - llm-provider + - openai + - llm +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-13 +content_type: "how-to" +--- + +# OpenAI + +Connect the AI Gateway to the OpenAI API. You end up with an LLM Provider that holds your OpenAI credentials, exposes a chosen set of OpenAI endpoints through the gateway, and can be consumed by any LLM proxy. + +This page is for platform administrators, who hold the upstream credentials. + +## Connection details + +The `openai` template and these connection settings apply to every OpenAI provider you create: + +| Setting | Value | +|---------|-------| +| Template ID | `openai` | +| Upstream URL | `https://api.openai.com/v1` | +| Auth type | `api-key` | +| Auth header | `Authorization` | + +OpenAI expects the key in the `Authorization` header with a `Bearer ` prefix, so the prefix is part of the value you configure rather than something the gateway adds. + +## Configure the provider + +Replace *``* with your OpenAI API key, keeping the `Bearer ` prefix, and deploy the provider: + +```bash +curl -X POST http://localhost:9090/api/management/v1/llm-providers \ + -H "Content-Type: application/yaml" \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + --data-binary @- <<'EOF' +apiVersion: gateway.api-platform.wso2.com/v1 +kind: LlmProvider +metadata: + name: openai-provider +spec: + displayName: OpenAI Provider + version: v1.0 + template: openai + context: /openai/latest + upstream: + url: https://api.openai.com/v1 + auth: + type: api-key + header: Authorization + value: Bearer + accessControl: + mode: deny_all + exceptions: + - path: /chat/completions + methods: [POST] + - path: /models + methods: [GET] + - path: /models/{modelId} + methods: [GET] +EOF +``` + +The `accessControl` block restricts which upstream paths the provider exposes. This provider denies everything except the three paths listed as exceptions. + +The `context` value sets the URL prefix the provider answers on, so this provider serves its exposed paths under `/openai/latest`. + +## Test the provider + +Send a chat completion request to the provider's context on the gateway: + +```bash +curl -X POST https://localhost:8443/openai/latest/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-4o-mini", + "messages": [ + { + "role": "user", + "content": "Hi" + } + ] + }' -k +``` + +The `-k` flag tells `curl` to skip Transport Layer Security (TLS) certificate verification, which a local gateway needs because it presents a self-signed listener certificate. Outside local testing, give the router a certificate from a trusted certificate authority and remove `-k`. + +## Supported models + +OpenAI publishes its model list in the [OpenAI models reference](https://platform.openai.com/docs/models). Any model OpenAI exposes on the endpoint above works through the gateway; set it in the `model` field of your request. + +## Related pages + +- [Provider templates](../llm-templates.md) — the token and model metadata the `openai` template extracts. +- [Route across multiple providers](../../expose-llms/multi-provider-routing.md) — put an LLM proxy in front of this provider and route to others alongside it. +- [Quick start guide](../../quick-start-guide.md) — deploy this provider and a proxy that consumes it, end to end. diff --git a/en/docs/ai-gateway/next/control-access/authenticate-clients.md b/en/docs/ai-gateway/next/control-access/authenticate-clients.md new file mode 100644 index 000000000..c8608abc3 --- /dev/null +++ b/en/docs/ai-gateway/next/control-access/authenticate-clients.md @@ -0,0 +1,131 @@ +--- +title: "Authenticate clients" +description: "Protect an LLM proxy or provider with the api-key-auth policy, issue consumer API keys through the management API, and manage the key lifecycle." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/control-access/authenticate-clients/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/control-access/authenticate-clients.md +tags: + - ai-gateway + - security + - authentication + - api-keys +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-13 +content_type: "how-to" +--- + +# Authenticate clients + +An application that calls an LLM proxy or provider through the gateway presents a credential with every request. This page shows you how to attach the `api-key-auth` policy to a proxy, issue a consumer API key, and call the gateway with it. Once the policy is attached to an operation, the gateway rejects requests to that operation that arrive without a valid key. + +This page is for the **platform administrator** who configures the proxy and issues keys, and for the **AI developer** who writes the application that sends them. + +This page covers the data plane, the traffic your applications send. Authentication on the control plane, the REST API you use to manage gateway configuration, is separate and is covered in [Secure the management API](secure-the-management-api.md). The two are configured independently: securing one does nothing for the other. + +Client authentication also differs from a provider's `accessControl` rules. Client authentication decides **who may call** the gateway. `accessControl` decides **which upstream endpoints** a provider exposes. See [Control access](overview.md) for how the surfaces relate. + +## Attach the API key policy + +Add `api-key-auth` to the proxy's `operationPolicies`, scoped to the paths and methods you want to protect: + +```yaml + operationPolicies: + - name: api-key-auth + version: v1 + paths: + - path: /chat/completions + methods: [POST] + params: + key: X-API-Key + in: header +``` + +The `params.key` value sets the header name the gateway reads the key from, and `params.in` sets where to look for it. Header matching is case-insensitive. Only operations listed under `paths` require a key, so an operation you leave out stays open. + +An LLM provider takes the same block under its own `spec.operationPolicies`. For a provider that protects two operations this way, see [AWS Bedrock](../connect-llm-providers/supported-providers/aws-bedrock.md). + +!!! note "`operationPolicies` compared with `policies`" + Some configurations attach policies under `spec.policies`. That field is deprecated and the gateway treats it identically to `operationPolicies`. Use `operationPolicies` in new configuration. + +## Create a consumer key + +Create a key for the application that calls the proxy. The following command stores the key in a shell variable, so the value is never printed to your terminal or shell history: + +```bash +PROXY_CONSUMER_KEY=$(curl -s -X POST \ + http://localhost:9090/api/management/v1/llm-proxies/openai-multi/api-keys \ + -u admin:admin \ + -H "Content-Type: application/json" \ + -d '{"name":"openai-multi-client"}' \ + | jq -r '.apiKey.apiKey') +``` + +Replace *`openai-multi`* with your proxy's name and *`openai-multi-client`* with a name for the key. The `name` field is optional; if you omit it, the gateway generates an identifier. + +Verify that a key was returned: + +```bash +test -n "$PROXY_CONSUMER_KEY" && test "$PROXY_CONSUMER_KEY" != "null" +``` + +Two things about the response matter: + +- **The key value is returned only when it is created or regenerated.** Store it securely. If you lose it, you cannot read it back — regenerate the key instead. +- **The response reports `remainingApiKeyQuota`.** Keys are quota-limited, so this tells you how many more you can create. + +To create a key for a provider rather than a proxy, use the same request with `llm-proxies` replaced by `llm-providers`. + +## Call the gateway with the key + +Send the key in the header named by the policy's `params.key`: + +```bash +curl -k -X POST https://localhost:8443/openai-multi/chat/completions \ + -H "Content-Type: application/json" \ + -H "X-API-Key: ${PROXY_CONSUMER_KEY}" \ + -d '{ + "model": "gpt-4o-mini", + "messages": [ + { + "role": "user", + "content": "Explain multi-provider routing in one sentence." + } + ] + }' +``` + +`X-API-Key` is the value the policy sets in this example, not a fixed gateway constant. If you set `params.key` to something else, clients send that header instead. The policy's default is `API-Key`. + +A request to a protected operation without a valid key returns `401`. + +## Manage keys + +The management API exposes five operations for the key lifecycle. All of them require Basic Auth and the `admin` or `consumer` role. Replace *`{id}`* with the proxy's name and *`{apiKeyName}`* with the key's name. + +| Operation | Request | What it does | +|-----------|---------|--------------| +| Create | `POST /llm-proxies/{id}/api-keys` | Generates a new API key for the proxy and returns its value. | +| List | `GET /llm-proxies/{id}/api-keys` | Lists the proxy's keys. Key values aren't included. | +| Regenerate | `POST /llm-proxies/{id}/api-keys/{apiKeyName}/regenerate` | Issues a new value for an existing key and returns it. The previous value stops working. | +| Update | `PUT /llm-proxies/{id}/api-keys/{apiKeyName}` | Sets a custom value on a key instead of generating one, for injecting an externally issued key. | +| Revoke | `DELETE /llm-proxies/{id}/api-keys/{apiKeyName}` | Revokes a key. Once revoked, it can no longer authenticate requests. | + +Providers expose the same five operations under `/llm-providers`. For the full request and response reference, see [LLM proxy management](../reference/management-api/llm-proxy-management.md) and [LLM provider management](../reference/management-api/llm-provider-management.md). + +Regenerate a key when you rotate credentials, and revoke one as soon as you believe it's exposed. Use a separate key per application and per environment, so revoking one doesn't interrupt the others. + +## Other authentication methods + +The gateway accepts any Policy Hub authentication policy in the same `operationPolicies` block. Each policy's parameters and behavior are documented in the [Policy Hub](https://wso2.com/api-platform/policy-hub). + +| Policy | What it does | +|--------|--------------| +| [JWT Auth](https://wso2.com/api-platform/policy-hub/policies/jwt-auth) | Validates JWT access tokens against one or more JWKS providers | +| [Basic Auth](https://wso2.com/api-platform/policy-hub/policies/basic-auth) | Enforces HTTP Basic Authentication | +| [Opaque Token Auth](https://wso2.com/api-platform/policy-hub/policies/opaque-token-auth) | Validates opaque OAuth 2.0 access tokens via RFC 7662 token introspection | +| [Subscription Validation](https://wso2.com/api-platform/policy-hub/policies/subscription-validation) | Confirms the caller holds an active subscription for the target API | + +## Related topics + +- [Secure the management API](secure-the-management-api.md) — authentication and role-based authorization on the control plane. +- [Control cost and traffic](../control-cost-and-traffic/overview.md) — rate limits and cost controls you can apply to the same operations. +- [Multi-provider routing](../expose-llms/multi-provider-routing.md) — the worked proxy example this page draws its configuration from. diff --git a/en/docs/ai-gateway/next/control-access/overview.md b/en/docs/ai-gateway/next/control-access/overview.md new file mode 100644 index 000000000..5a91aeedd --- /dev/null +++ b/en/docs/ai-gateway/next/control-access/overview.md @@ -0,0 +1,36 @@ +--- +title: "Control access" +description: "Control access to the AI Gateway: client authentication on proxies and providers, authentication and role-based authorization on the management API, and provider-level rules for upstream endpoints." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/control-access/overview/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/control-access/overview.md +tags: + - ai-gateway + - security + - access-control +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-13 +content_type: "concept" +--- + +# Control access + +The AI Gateway controls access on three surfaces. + +An application that calls an LLM proxy or provider presents a credential with every request. Once you attach an authentication policy to an operation, the gateway rejects calls to that operation that arrive without a valid credential. For the API key walkthrough and the other authentication policies, see [Authenticate clients](authenticate-clients.md). + +The management API is the REST API you use to manage gateway configuration. It authenticates callers with locally configured users, with JWTs validated against an identity provider, or with both. Authorization is role-based, and the gateway enforces it per route. + +An LLM provider carries its own `accessControl` rules. These decide which upstream endpoints the provider exposes through the gateway. For a provider configuration that sets them, see [Quick Start Guide](../quick-start-guide.md). + +## Who configures this + +Platform administrators configure LLM providers, including the access control rules that decide which endpoints a provider exposes. They also attach the authentication policies that protect proxies and providers, and issue the API keys that applications present. AI developers send those keys from the applications they build. The management API's own authentication and role mapping live in the gateway configuration, under `controller.auth`. + +## In this section + +This section contains the following pages: + +| Page | What it covers | +|------|----------------| +| [Authenticate clients](authenticate-clients.md) | Protect an LLM proxy or provider with the `api-key-auth` policy, issue consumer API keys through the management API, and manage the key lifecycle. | +| [Secure the management API](secure-the-management-api.md) | Configure Basic Auth or JWT/IDP authentication and role-based authorization for the AI Gateway Controller REST API. | diff --git a/en/docs/ai-gateway/next/gateway-controller-management-api/authentication.md b/en/docs/ai-gateway/next/control-access/secure-the-management-api.md similarity index 94% rename from en/docs/ai-gateway/next/gateway-controller-management-api/authentication.md rename to en/docs/ai-gateway/next/control-access/secure-the-management-api.md index 46c49c5ae..6c597bc9d 100644 --- a/en/docs/ai-gateway/next/gateway-controller-management-api/authentication.md +++ b/en/docs/ai-gateway/next/control-access/secure-the-management-api.md @@ -1,14 +1,14 @@ --- title: "Secure the management API" description: "Configure Basic Auth or JWT/IDP authentication and role-based authorization for the AI Gateway Controller REST API." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/gateway-controller-management-api/authentication/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/gateway-controller-management-api/authentication.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/control-access/secure-the-management-api/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/control-access/secure-the-management-api.md tags: - ai-gateway - security - authentication author: WSO2 API Platform Documentation Team -last_updated: 2026-08-10 +last_updated: 2026-08-11 content_type: "how-to" --- diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/execution-order.md b/en/docs/ai-gateway/next/control-content/execution-order.md similarity index 77% rename from en/docs/ai-gateway/next/llm-proxy/guardrails/execution-order.md rename to en/docs/ai-gateway/next/control-content/execution-order.md index 4698a15b1..d3bd0d2e6 100644 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/execution-order.md +++ b/en/docs/ai-gateway/next/control-content/execution-order.md @@ -1,8 +1,8 @@ --- title: "Guardrail execution order" description: "How guardrails execute across phases in the AI Gateway dual-hop model: the LLM Proxy chain runs before the LLM Provider chain on request, and in reverse on response." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/execution-order/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/execution-order.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/control-content/execution-order/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/control-content/execution-order.md tags: - ai-gateway - guardrails @@ -18,7 +18,7 @@ content_type: "concept" The AI Gateway routes every request through two policy chains in sequence: the **LLM Proxy chain** and the **LLM Provider chain**. Each chain runs the same four execution phases, but the order in which the two chains execute differs between the request and response paths. -This document covers the dual-hop execution model. For foundational concepts — within-chain policy ordering, streaming mode, and short-circuit behavior — see [Policy execution order](../../../../../api-gateway/next/policies/policy-execution-order.md). The same rules apply to each chain individually. +This document covers the dual-hop execution model. For foundational concepts — within-chain policy ordering, streaming mode, and short-circuit behavior — see [Policy execution order](../../../api-gateway/next/policies/policy-execution-order.md). The same rules apply to each chain individually. ## The two chains @@ -80,7 +80,7 @@ Client The following diagram shows how requests pass through the LLM Proxy chain and LLM Provider chain in sequence, and how responses return through both chains in reverse order: -![Dual-hop guardrail execution diagram showing request flowing through LLM Proxy chain then LLM Provider chain in forward order across request phases, and response flowing in reverse order through LLM Provider chain then LLM Proxy chain back to the client](../../../../assets/img/api-gateway/gateway-policy-execution-order-llm-provider-proxy.png) +![Diagram of dual-hop guardrail execution: a request traverses the LLM Proxy chain then the LLM Provider chain, and the response returns in reverse order](../../../assets/img/api-gateway/gateway-policy-execution-order-llm-provider-proxy.png) This mirrors the request wrapping at the chain level: the LLM Proxy wraps the LLM Provider on the way in, so on the way back the inner chain (Provider) unwinds first, then the outer chain (Proxy). @@ -99,10 +99,10 @@ When an LLM returns a streaming response (such as SSE from a chat completion end The following diagram shows how streaming body chunks flow through the policy chain at each hop: -![Streaming policy chain execution order diagram showing request body chunks flowing through the policy chain via OnRequestBodyChunk in forward order, and response chunks flowing in reverse order via OnResponseBodyChunk](../../../../assets/img/api-gateway/gateway-policy-execution-order-streaming.png) +![Diagram of streaming policy chain execution order, with request body chunks flowing forward through the chain and response chunks flowing in reverse](../../../assets/img/api-gateway/gateway-policy-execution-order-streaming.png) ## Related topics - [Guardrails overview](overview.md) -- [Policy execution order](../../../../../api-gateway/next/policies/policy-execution-order.md) — within-chain ordering, streaming, and short-circuit rules that apply to each chain individually +- [Policy execution order](../../../api-gateway/next/policies/policy-execution-order.md) — within-chain ordering, streaming, and short-circuit rules that apply to each chain individually diff --git a/en/docs/ai-gateway/next/control-content/overview.md b/en/docs/ai-gateway/next/control-content/overview.md new file mode 100644 index 000000000..7a233587e --- /dev/null +++ b/en/docs/ai-gateway/next/control-content/overview.md @@ -0,0 +1,68 @@ +--- +title: "Guardrails overview" +description: "AI Gateway guardrails: LLM-aware policies for content filtering, safety, and compliance, with per-policy reference in the WSO2 API Platform Policy Hub." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/control-content/overview/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/control-content/overview.md +tags: + - ai-gateway + - guardrails + - llm-proxy +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-12 +content_type: "concept" +--- + +# Guardrails overview + +Guardrails are policies that run in the LLM Proxy's request and response pipeline to validate, filter, or transform content before it reaches an LLM or is returned to the client. + +AI Guardrails allow you to enforce safety, content, and compliance policies on AI traffic flowing through the AI Gateway. They can be applied at the LLM Provider level (organization-wide), at the LLM Proxy level (per-application), or on MCP Proxies. + +Guardrails use the same underlying policy engine as [API Gateway policies](../../../api-gateway/next/policies/overview.md). Each guardrail declares which execution phases it participates in, and the engine calls the appropriate hook for each phase. + +## What guardrails do + +- **Content filtering**: Block or flag requests and responses that violate configured topic, word, or content policies. +- **PII detection and masking**: Detect and mask or redact personally identifiable information in prompts and responses. +- **Schema validation**: Enforce structure on requests or responses using JSON Schema. +- **Pattern matching**: Detect prohibited content using regular expressions. +- **Length and count limits**: Enforce word count, sentence count, and byte length constraints on prompts and responses. +- **External validation**: Delegate content validation to managed services such as AWS Bedrock Guardrails or Azure Content Safety. + +## Available guardrails + +Guardrail policies are documented in the [Policy Hub](https://wso2.com/api-platform/policy-hub), the versioned reference for every API Platform policy. For policy categories and how policies chain, see the [Policy Hub overview](../../../policy-hub/overview.md). + +| Guardrail | What it checks | +|-----------|----------------| +| [Regex Guardrail](https://wso2.com/api-platform/policy-hub/policies/regex-guardrail) | Validates content against a regular expression | +| [JSON Schema Guardrail](https://wso2.com/api-platform/policy-hub/policies/json-schema-guardrail) | Enforces a JSON Schema on request or response payloads | +| [Word Count Guardrail](https://wso2.com/api-platform/policy-hub/policies/word-count-guardrail) | Enforces word-count limits on payloads | +| [Sentence Count Guardrail](https://wso2.com/api-platform/policy-hub/policies/sentence-count-guardrail) | Enforces sentence-count limits on payloads | +| [Content Length Guardrail](https://wso2.com/api-platform/policy-hub/policies/content-length-guardrail) | Enforces byte-length limits on payloads | +| [URL Guardrail](https://wso2.com/api-platform/policy-hub/policies/url-guardrail) | Validates URLs found in request or response bodies | +| [PII Masking](https://wso2.com/api-platform/policy-hub/policies/pii-masking-regex) | Masks or redacts PII from request/response bodies using configurable regex patterns | +| [Semantic Prompt Guard](https://wso2.com/api-platform/policy-hub/policies/semantic-prompt-guard) | Blocks or allows prompts based on semantic similarity to configured allow/deny phrases | +| [Azure Content Safety](https://wso2.com/api-platform/policy-hub/policies/azure-content-safety-content-moderation) | Screens content against Azure Content Safety API | +| [AWS Bedrock Guardrail](https://wso2.com/api-platform/policy-hub/policies/aws-bedrock-guardrail) | Validates content against AWS Bedrock Guardrails | +| [Granite Guardian Prompt Injection](https://wso2.com/api-platform/policy-hub/policies/granite-guardian-prompt-injection) | Detects prompt injection and jailbreak attempts in LLM API requests using IBM Granite Guardian 3.3 8B | +| [NeMo Guard Content Safety](https://wso2.com/api-platform/policy-hub/policies/nvidia-nemoguard-content-safety) | Validates request and/or response content using NVIDIA NeMo Guard (llama-3.1-nemoguard-8b-content-safety) | + +## Prompt management + +These policies shape a prompt rather than validating it, and they are documented in the [Policy Hub](https://wso2.com/api-platform/policy-hub) alongside the guardrails. + +| Policy | What it does | +|--------|--------------| +| [Prompt Decorator](https://wso2.com/api-platform/policy-hub/policies/prompt-decorator) | Injects system instructions or context into prompts at the gateway layer | +| [Prompt Template](https://wso2.com/api-platform/policy-hub/policies/prompt-template) | Applies configurable templates to transform prompts before they reach the model | + +## Custom guardrails + +You can extend the AI Gateway with custom guardrail policies by building a custom gateway image using the `ap` CLI. See [Customizing the Gateway by Adding and Removing Policies](../../../tools/cli/customizing-gateway-policies.md). + +## How guardrails execute + +When multiple guardrails are attached to an LLM Proxy, they run as an ordered chain across request and response phases. The AI Gateway's architecture — with a separate LLM Proxy chain and LLM Provider chain — means every request passes through two chains in sequence. + +For a full explanation of phase execution, multi-guardrail ordering, and the dual-hop execution model specific to AI Gateway, see [Guardrail execution order](execution-order.md). diff --git a/en/docs/ai-gateway/next/control-cost-and-traffic/overview.md b/en/docs/ai-gateway/next/control-cost-and-traffic/overview.md new file mode 100644 index 000000000..bdef6b5de --- /dev/null +++ b/en/docs/ai-gateway/next/control-cost-and-traffic/overview.md @@ -0,0 +1,47 @@ +--- +title: "Control cost and traffic" +description: "Keep LLM spend and traffic volume predictable: distribute requests across model endpoints, cache equivalent prompts, and set timeouts on a resource." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/control-cost-and-traffic/overview/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/control-cost-and-traffic/overview.md +tags: + - ai-gateway + - cost + - traffic +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-12 +content_type: "concept" +--- + +# Control cost and traffic + +The policies in this section keep LLM spend and traffic volume predictable. They spread requests across a pool of model endpoints, and they avoid an upstream call altogether when a semantically equivalent prompt has already been answered. + +Timeouts work differently. You configure them through the `resilience` block on an `LlmProvider`, `LlmProxy`, or `Mcp` resource, rather than by attaching a policy. + +## In this section + +This section contains the following page: + +| Page | What it covers | +|------|----------------| +| [Timeouts](timeouts.md) | Configure gateway-level and API-level timeouts (connect, route, idle, and HTTP connection manager) in the API Platform AI Gateway to protect against slow or unreachable backends and slow clients. | + +## Policies + +These policies are documented in the [Policy Hub](https://wso2.com/api-platform/policy-hub), the versioned reference for every API Platform policy. For policy categories and how policies chain, see the [Policy Hub overview](../../../policy-hub/overview.md). + +| Policy | What it does | +|--------|--------------| +| [LLM Cost](https://wso2.com/api-platform/policy-hub/policies/llm-cost) | Calculates the monetary cost of each LLM call and stores it for downstream policies | +| [LLM Cost-Based Rate Limit](https://wso2.com/api-platform/policy-hub/policies/llm-cost-based-ratelimit) | Enforces monetary budget quotas on LLM usage | +| [Token-Based Rate Limit](https://wso2.com/api-platform/policy-hub/policies/token-based-ratelimit) | Caps usage by token count rather than request count | +| [Semantic Cache](https://wso2.com/api-platform/policy-hub/policies/semantic-cache) | Caches LLM responses using vector similarity, returning cached results for semantically equivalent prompts | +| [Prompt Compressor](https://wso2.com/api-platform/policy-hub/policies/prompt-compressor) | Compresses prompt text to reduce token usage before upstream calls | +| [Model Round Robin](https://wso2.com/api-platform/policy-hub/policies/model-round-robin) | Distributes requests evenly across a pool of AI model endpoints | +| [Model Weighted Round Robin](https://wso2.com/api-platform/policy-hub/policies/model-weighted-round-robin) | Distributes requests across model endpoints according to configured weights | +| [Respond](https://wso2.com/api-platform/policy-hub/policies/respond) | Returns an immediate response without forwarding to the upstream backend | + +## Related guides + +- [Enforce token-based rate limiting on an LLM proxy](../../../guides/ai-and-mcp/enforce-token-based-rate-limiting-on-an-llm-proxy.md) — caps token consumption on a proxy within a rolling window, so one application cannot exhaust the budget. +- [Set up a governed multi-model LLM proxy](../../../guides/ai-and-mcp/set-up-a-governed-multi-model-llm-proxy-with-cost-controls-and-failover.md) — distributes traffic across models behind one proxy, with per-team token budgets, PII masking, and semantic caching. diff --git a/en/docs/ai-gateway/next/resiliency/timeouts.md b/en/docs/ai-gateway/next/control-cost-and-traffic/timeouts.md similarity index 98% rename from en/docs/ai-gateway/next/resiliency/timeouts.md rename to en/docs/ai-gateway/next/control-cost-and-traffic/timeouts.md index 25750345f..71339872d 100644 --- a/en/docs/ai-gateway/next/resiliency/timeouts.md +++ b/en/docs/ai-gateway/next/control-cost-and-traffic/timeouts.md @@ -1,15 +1,15 @@ --- title: "Configure Gateway Timeouts" description: "Configure gateway-level and API-level timeouts (connect, route, idle, and HTTP connection manager) in the API Platform AI Gateway to protect against slow or unreachable backends and slow clients." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/resiliency/timeouts/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/resiliency/timeouts.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/control-cost-and-traffic/timeouts/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/control-cost-and-traffic/timeouts.md tags: - ai-gateway - configuration - networking - resiliency author: WSO2 API Platform Documentation Team -last_updated: 2026-06-30 +last_updated: 2026-08-11 content_type: "how-to" --- diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/expose-llms/multi-provider-routing.md similarity index 98% rename from en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md rename to en/docs/ai-gateway/next/expose-llms/multi-provider-routing.md index 7381f67f4..038ecdb53 100644 --- a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/next/expose-llms/multi-provider-routing.md @@ -1,14 +1,14 @@ --- title: "Multi-Provider Routing for LLM Proxies" description: "Route OpenAI-compatible LLM proxy requests to multiple providers using header-based selection and provider-specific transformers." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/multi-provider-routing/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/multi-provider-routing.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/expose-llms/multi-provider-routing/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/expose-llms/multi-provider-routing.md tags: - ai-gateway - llm - routing author: WSO2 API Platform Documentation Team -last_updated: 2026-08-05 +last_updated: 2026-08-11 content_type: "guide" --- @@ -85,7 +85,7 @@ operationPolicies: The policy rewrites the model at the location defined by the provider template. It can rewrite a model in the request payload, a header, a query parameter, or a path parameter. -See [Model Round Robin](load-balancing/model-round-robin.md) for its complete configuration. +See [Model Round Robin](https://wso2.com/api-platform/policy-hub/policies/model-round-robin) for its complete configuration. ### Model weighted round robin @@ -110,7 +110,7 @@ operationPolicies: This example produces the repeating sequence `gpt-4o`, `gpt-4o`, `claude-sonnet-4-5-20250929` while both targets are available. It provides proportional deterministic distribution, not random or performance-based load balancing. -See [Model Weighted Round Robin](load-balancing/model-weighted-round-robin.md) for its complete configuration. +See [Model Weighted Round Robin](https://wso2.com/api-platform/policy-hub/policies/model-weighted-round-robin) for its complete configuration. ## Configure Providers @@ -984,7 +984,7 @@ For a larger configuration containing OpenAI, Anthropic, Azure OpenAI, Mistral, For automatic traffic distribution across models and providers, see: -- [Model Round Robin](load-balancing/model-round-robin.md) -- [Model Weighted Round Robin](load-balancing/model-weighted-round-robin.md) +- [Model Round Robin](https://wso2.com/api-platform/policy-hub/policies/model-round-robin) +- [Model Weighted Round Robin](https://wso2.com/api-platform/policy-hub/policies/model-weighted-round-robin) AWS Bedrock usage can also be evaluated by the [LLM Cost policy](../../../ai-workspace/next/policies/overview.md#llm-cost). diff --git a/en/docs/ai-gateway/next/expose-llms/overview.md b/en/docs/ai-gateway/next/expose-llms/overview.md new file mode 100644 index 000000000..71b69a0df --- /dev/null +++ b/en/docs/ai-gateway/next/expose-llms/overview.md @@ -0,0 +1,53 @@ +--- +title: "Expose LLMs to applications" +description: "Expose an LLM provider to applications through an LLM proxy: its own URL context, per-application policies, and the provider-level rules it inherits." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/expose-llms/overview/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/expose-llms/overview.md +tags: + - ai-gateway + - llm-proxy + - llm +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-12 +content_type: "concept" +--- + +# Expose LLMs to applications + +An LLM Proxy allows developers to create custom API endpoints that consume an LLM Provider, while inheriting administrator-enforced access control, budgeting and organization-wide policies defined at the provider level. Each proxy gets its own URL context (e.g., `/assistant`) and can have its own policies applied. This enables: + +- Multiple AI applications to share a single LLM Provider +- A single OpenAI-compatible endpoint to route requests to multiple LLM providers. See [Multi-Provider Routing for LLM Proxies](./multi-provider-routing.md). +- Per-application policies such as prompt management and guardrails +- Separation between platform administration and application development + +To deploy your first LLM proxy against a configured provider, see [Quick Start Guide](../quick-start-guide.md). + +## Who configures this + +AI developers own LLM proxies. A developer creates the proxy, names the LLM provider it consumes, and attaches the policies one application needs. The access control, budgeting, and organization-wide policies set on the provider still apply. + +## In this section + +This section contains the following pages: + +| Page | What it covers | +|------|----------------| +| [Route across multiple providers](multi-provider-routing.md) | Route OpenAI-compatible LLM proxy requests to multiple providers using header-based selection and provider-specific transformers. | +| [Stream responses](streaming-responses.md) | Stream responses through API Platform AI Gateway chunk by chunk across LLM providers, LLM proxies, and MCP proxies, and understand how policies, analytics, and token usage behave. | + +## Policies + +One proxy-layer routing policy is documented in the [Policy Hub](https://wso2.com/api-platform/policy-hub), the versioned reference for every API Platform policy. For policy categories and how policies chain, see the [Policy Hub overview](../../../policy-hub/overview.md). + +| Policy | What it does | +|--------|--------------| +| [LLM Header Router](https://wso2.com/api-platform/policy-hub/policies/llm-header-router) | Selects an LLM provider for OpenAI Chat Completions requests using a configurable request header | + +## Related guides + +- [Set up a governed multi-model LLM proxy](../../../guides/ai-and-mcp/set-up-a-governed-multi-model-llm-proxy-with-cost-controls-and-failover.md) — distributes traffic across models behind one proxy, with per-team token budgets, PII masking, and semantic caching. +- [Enforce a consistent AI persona with the prompt decorator policy](../../../guides/ai-and-mcp/using-prompt-decorator-policy.md) — prepends a persona system message to every request on an LLM proxy, without changing client code. +- [Configure Claude Code with AI Gateway](../../../guides/ai-and-mcp/ai-coding-assistants/claude-code-configuration-with-ai-gateway.md) — routes Claude Code through an Anthropic provider and an LLM proxy. +- [Configure Gemini CLI with AI Gateway](../../../guides/ai-and-mcp/ai-coding-assistants/gemini-cli-configuration-with-ai-gateway.md) — routes Google Gemini CLI through a Gemini provider and an LLM proxy. +- [Configure OpenAI Codex CLI with AI Gateway](../../../guides/ai-and-mcp/ai-coding-assistants/codex-configuration-with-ai-gateway.md) — routes OpenAI Codex CLI through an OpenAI provider and an LLM proxy. diff --git a/en/docs/ai-gateway/next/streaming-responses.md b/en/docs/ai-gateway/next/expose-llms/streaming-responses.md similarity index 91% rename from en/docs/ai-gateway/next/streaming-responses.md rename to en/docs/ai-gateway/next/expose-llms/streaming-responses.md index 8c4f42aa4..d0a7d37f6 100644 --- a/en/docs/ai-gateway/next/streaming-responses.md +++ b/en/docs/ai-gateway/next/expose-llms/streaming-responses.md @@ -1,15 +1,15 @@ --- title: "Real-Time AI Streaming" description: "Stream responses through API Platform AI Gateway chunk by chunk across LLM providers, LLM proxies, and MCP proxies, and understand how policies, analytics, and token usage behave." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/streaming-responses/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/streaming-responses.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/expose-llms/streaming-responses/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/expose-llms/streaming-responses.md tags: - ai-gateway - llm - mcp - streaming author: WSO2 API Platform Documentation Team -last_updated: 2026-08-10 +last_updated: 2026-08-11 content_type: "concept" --- @@ -98,6 +98,6 @@ If a streamed response carries no `usage` block, the gateway has no token counts ## Related documentation -- [Quick Start Guide](quick-start-guide.md) — deploy a provider and a proxy, then send your first request -- [MCP Proxy Quick Start Guide](mcp-proxy/quick-start-guide.md) — deploy an MCP proxy -- [Sentence Count Guardrail](llm-proxy/guardrails/sentence-count.md) — a guardrail that gates a stream until it can evaluate the content +- [Quick Start Guide](../quick-start-guide.md) — deploy a provider and a proxy, then send your first request +- [MCP Proxy Quick Start Guide](../mcp-proxy/create-an-mcp-proxy.md) — deploy an MCP proxy +- [Sentence Count Guardrail](https://wso2.com/api-platform/policy-hub/policies/sentence-count-guardrail) — a guardrail that gates a stream until it can evaluate the content diff --git a/en/docs/ai-gateway/next/how-it-works.md b/en/docs/ai-gateway/next/how-it-works.md new file mode 100644 index 000000000..b9aeaec8b --- /dev/null +++ b/en/docs/ai-gateway/next/how-it-works.md @@ -0,0 +1,76 @@ +--- +title: "How the AI Gateway works" +description: "How the AI Gateway handles AI traffic: LLM proxies, LLM providers, MCP proxies, where policies attach, provider templates, and streaming responses." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/how-it-works/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/how-it-works.md +tags: + - ai-gateway + - llm + - mcp + - architecture +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-12 +content_type: "concept" +--- + +# How it works + +This page describes the artifacts a request passes through in the AI Gateway, where you attach policies, and how the gateway handles streaming responses. + +## Architecture + +The following diagram shows the artifacts a request passes through: + +``` + AI apps MCP clients + │ │ + ▼ ▼ +┌────────────────────────────────────────────┐ +│ AI Gateway │ +│ │ +│ ┌───────────────┐ ┌───────────────┐ │ +│ │ LLM Proxy │ │ MCP Proxy │ │ +│ └───────┬───────┘ └───────┬───────┘ │ +│ │ │ │ +│ ▼ │ │ +│ ┌───────────────┐ │ │ +│ │ LLM Provider │ │ │ +│ └───────┬───────┘ │ │ +└──────────┼──────────────────────┼──────────┘ + ▼ ▼ + LLM services MCP servers +``` + +Client traffic reaches the gateway on the router ports: 8443 for HTTPS and 8080 for HTTP. An AI application calls an LLM Proxy at its own URL context, such as `/assistant`. An MCP client calls an MCP Proxy at its context, such as `/everything`. + +An LLM Proxy names the provider it consumes in its `provider.id` field, so a request that arrives at the proxy leaves through that LLM Provider. The provider holds the template, the upstream service URL, the credentials for that service, and the `accessControl` rules that decide which upstream endpoints it exposes. An MCP Proxy routes to its MCP server directly, without a provider. + +You attach policies at three points: on an LLM Proxy, on an LLM Provider, and on an MCP Proxy. A request through an LLM Proxy runs the proxy's policy chain first, then the provider's chain. For the phase-by-phase order, and how the two chains reverse on the response path, see [Guardrail execution order](control-content/execution-order.md). + +## How a request flows + +1. Administrators verify the Gateway-Controller admin health endpoint and configure LLM Providers and MCP Proxies via the Gateway-Controller API +2. Developers create LLM Proxies to build AI applications on top of available providers +3. The gateway routes traffic, applies policies, and manages authentication + +## LLM provider templates + +An LLM Provider Template defines the characteristics and behaviors specific to an AI service provider, such as OpenAI, Azure OpenAI, or other LLM platforms. It describes how the gateway should interpret and extract usage and operational metadata, including prompt, completion, total, and remaining token information, as well as request and response model metadata. + +The gateway ships with these provider templates, loaded at startup: + +| Template ID | Provider | +|-------------|----------| +| `openai` | OpenAI | +| `azure-openai` | Azure OpenAI | +| `anthropic` | Anthropic | +| `gemini` | Gemini | +| `mistralai` | MistralAI | +| `awsbedrock` | AWS Bedrock | +| `azureai-foundry` | Azure AI Foundry | + +For the extraction configuration each template applies, see [LLM provider templates](./connect-llm-providers/llm-templates.md). + +## Streaming + +When an upstream service streams its response, the gateway relays it to the client chunk by chunk instead of buffering the whole response. This holds for LLM providers and LLM proxies, and needs no configuration. On MCP proxies, request bodies stream, but response bodies stay buffered. See [Real-time AI streaming](./expose-llms/streaming-responses.md). diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/aws-bedrock-guardrail.md b/en/docs/ai-gateway/next/llm-proxy/guardrails/aws-bedrock-guardrail.md deleted file mode 100644 index 12f1e458d..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/aws-bedrock-guardrail.md +++ /dev/null @@ -1,291 +0,0 @@ ---- -title: "AWS Bedrock Guardrail" -description: "Validate LLM request and response content against AWS Bedrock Guardrails for content filtering, topic detection, and PII masking." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/aws-bedrock-guardrail/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/aws-bedrock-guardrail.md -tags: - - ai-gateway - - guardrails - - aws -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# AWS Bedrock Guardrail - -## Overview - -The AWS Bedrock Guardrail policy validates request or response body content against AWS Bedrock Guardrails, which provide enterprise-grade content filtering, topic detection, word filtering, and PII (Personally Identifiable Information) detection and masking. This guardrail enables you to enforce content safety policies consistently across your LLM applications using AWS Bedrock's managed guardrail service. - -The policy supports multiple authentication modes including AWS IAM role assumption, static credentials, and default credential chain, making it flexible for various AWS deployment scenarios. It can mask or redact PII entities in requests and restore them in responses, ensuring data privacy while maintaining functionality. - -## Features - -- **Content filtering**: Detects and blocks prohibited content based on guardrail policies -- **Topic detection**: Validates content against configured topic restrictions -- **Word filtering**: Blocks content containing prohibited words or phrases -- **PII detection and masking**: Identifies and masks PII entities (emails, phone numbers, SSNs, etc.) -- **PII restoration**: Restores masked PII in responses when configured (masking mode) -- **PII redaction**: Permanently removes PII by replacing with "*****" (redaction mode) -- **Multiple authentication modes**: Supports role assumption, static credentials, or default AWS credential chain -- **JSONPath support**: Extract and validate specific fields within JSON payloads -- **Separate request/response configuration**: Independent configuration for request and response phases -- **Detailed assessment information**: Optional detailed violation information in error responses - -## Configuration - -### Parameters - -#### Request Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `redactPII` | boolean | No | `false` | If `true`, redacts PII by replacing with "*****" (permanent). If `false`, masks PII with placeholders that can be restored in responses. | -| `passthroughOnError` | boolean | No | `false` | If `true`, allows requests to proceed if AWS Bedrock Guardrail API call fails. If `false`, blocks requests on API errors. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information from AWS Bedrock Guardrail in error responses. | - -#### Response Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `passthroughOnError` | boolean | No | `false` | If `true`, allows requests to proceed if AWS Bedrock Guardrail API call fails. If `false`, blocks requests on API errors. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information from AWS Bedrock Guardrail in error responses. | - -### System Parameters (Required) - -These parameters are typically configured at the gateway level and automatically injected, or you can override those values from the params section in the api artifact definition file as well: - -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `region` | string | Yes | AWS region where the Bedrock Guardrail is located (e.g., "us-east-1", "us-west-2"). | -| `guardrailID` | string | Yes | AWS Bedrock Guardrail identifier (the unique ID of your guardrail). | -| `guardrailVersion` | string | Yes | AWS Bedrock Guardrail version (e.g., "DRAFT", "1", "2"). Use "DRAFT" for testing, numbered versions for production. | -| `awsAccessKeyID` | string | No | AWS access key ID (for static credentials or role assumption). If omitted, runtime uses default AWS credential chain (environment variables, IAM roles, etc.). | -| `awsSecretAccessKey` | string | No | AWS secret access key (for static credentials or role assumption). If omitted, runtime uses default AWS credential chain. | -| `awsSessionToken` | string | No | AWS session token (optional, for temporary credentials). | -| `awsRoleARN` | string | No | AWS IAM role ARN to assume (for role-based authentication). If specified, runtime assumes this role instead of using static credentials. | -| `awsRoleRegion` | string | No | AWS region for role assumption (required if `awsRoleARN` is specified). | -| `awsRoleExternalID` | string | No | External ID for role assumption (optional, for cross-account access security). | - - -### Configuring System Parameters in config.toml - -System parameters can be configured globally in the gateway's `config.toml` file. These values serve as defaults for all AWS Bedrock Guardrail policy instances and can be overridden per-policy in the API configuration if needed. - -#### Location in config.toml - -Add the following configuration section to your `config.toml` file: - -```toml -awsbedrock_guardrail_region = "us-east-1" -awsbedrock_guardrail_id = "your-guardrail-id" -awsbedrock_guardrail_version = "DRAFT" -awsbedrock_access_key_id = "" -awsbedrock_secret_access_key = "" -awsbedrock_session_token = "" -awsbedrock_role_arn = "" -awsbedrock_role_region = "" -awsbedrock_role_external_id = "" -``` - -## JSONPath Support - -The guardrail supports JSONPath expressions to extract and validate specific fields within JSON payloads. Common examples: - -- `$.message` - Extracts the `message` field from the root object -- `$.data.content` - Extracts nested content from `data.content` -- `$.items[0].text` - Extracts text from the first item in an array -- `$.messages[0].content` - Extracts content from the first message in a messages array -- `$.messages[-1].content` - Extracts content from the last message in a messages array - -If `jsonPath` is empty or not specified, the entire payload is treated as a string and validated. - -## PII Handling - -### Masking Mode (redactPII: false) - -When `redactPII` is `false`: -- **Request phase**: PII entities are masked with placeholders like `EMAIL_0001`, `PHONE_0002`, etc. -- Use this mode when you need PII to flow through the system but want it masked during processing - -### Redaction Mode (redactPII: true) - -When `redactPII` is `true`: -- PII entities are permanently replaced with `*****` -- Original values cannot be restored -- Use this mode when you want to completely remove PII from content - -## Examples - -### Example 1: Basic Guardrail with Static Credentials - -Deploy an LLM provider with AWS Bedrock Guardrail validation: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: bedrock-guardrail-provider -spec: - displayName: AWS Bedrock Guardrail Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - policies: - - name: aws-bedrock-guardrail - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - jsonPath: "$.messages[0].content" - redactPII: false - showAssessment: true - response: - jsonPath: "$.choices[0].message.content" - showAssessment: true -EOF -``` - -**Test the guardrail:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the LLM provider configuration and use localhost to invoke. - -```bash -# Request with prohibited content (should fail with HTTP 422) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "This is prohibited content" - } - ] - }' - -# Request with PII (should mask PII and proceed) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Contact me at user@example.com or call 555-123-4567" - } - ] - }' -``` - -### Example 2: PII Redaction Mode - -Configure to redact PII: - -```yaml -policies: - - name: aws-bedrock-guardrail - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - jsonPath: "$.messages[0].content" - redactPII: true # Redact mode - showAssessment: false - response: - jsonPath: "$.choices[0].message.content" -``` - -## Use Cases - -1. **Content Safety**: Enforce enterprise content policies to prevent inappropriate or harmful content from being processed or returned. - -2. **Compliance**: Meet regulatory requirements (HIPAA, GDPR, etc.) by detecting and masking PII in LLM interactions. - -3. **Topic Control**: Restrict LLM usage to approved topics only, preventing misuse or access to sensitive domains. - -4. **Data Privacy**: Mask sensitive information during processing while maintaining the ability to restore it in responses when needed. - -5. **Prohibited Word Filtering**: Block content containing prohibited words, phrases, or patterns defined in your guardrail. - -6. **Multi-tenant Security**: Isolate content policies per tenant or application using different guardrail configurations. - -7. **Audit and Monitoring**: Use detailed assessment information to audit content violations and improve policies. - -## Error Response - -When validation fails, the guardrail returns an HTTP 422 status code with the following structure: - -```json -{ - "type": "AWS_BEDROCK_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "AWS Bedrock Guardrail", - "actionReason": "Violation of AWS Bedrock Guardrail detected.", - "direction": "REQUEST" - } -} -``` - -If `showAssessment` is enabled, additional details are included: - -```json -{ - "type": "AWS_BEDROCK_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "AWS Bedrock Guardrail", - "actionReason": "Violation of AWS Bedrock Guardrail detected.", - "direction": "REQUEST", - "assessments": { - "topicPolicy": { - "topics": ["Topic1", "Topic2"] - }, - "contentPolicy": { - "filters": ["Filter1"] - }, - "sensitiveInformationPolicy": { - "piiEntities": [...], - "regexes": [...] - } - } - } -} -``` - -## Notes - -- The guardrail must be created in AWS Bedrock before use. Use AWS Console, CLI, or SDK to create guardrails with your policies. -- Guardrail version "DRAFT" is useful for testing. Use numbered versions (e.g., "1", "2") for production. -- PII masking with restoration (`redactPII: false`) stores mapping between original and masked values in request metadata, which is used during response processing. -- When using role assumption, ensure the IAM role has `bedrock:ApplyGuardrail` permission. -- The policy uses AWS SDK v2 for authentication and API calls. -- JSONPath extraction failures result in error responses unless `passthroughOnError: true`. -- Content modifications (PII masking) are applied to the payload and forwarded to upstream if no blocking violation occurs. -- The policy validates both request and response phases independently when both are configured. -- Ensure your guardrail is in the specified AWS region; cross-region calls are not supported. diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/azure-content-safety.md b/en/docs/ai-gateway/next/llm-proxy/guardrails/azure-content-safety.md deleted file mode 100644 index 2982b0c36..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/azure-content-safety.md +++ /dev/null @@ -1,363 +0,0 @@ ---- -title: "Azure Content Safety" -description: "Validate LLM request and response content against Azure Content Safety API, detecting hate speech, sexual content, and violence." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/azure-content-safety/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/azure-content-safety.md -tags: - - ai-gateway - - guardrails - - azure -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# Azure Content Safety - -## Overview - -The Azure Content Safety guardrail validates request or response body content against Microsoft Azure Content Safety API for content moderation. It detects and blocks harmful content across four categories: hate speech, sexual content, self-harm, and violence. Each category can be configured with a severity threshold (0-7) or disabled entirely, providing flexible content moderation policies tailored to your application's requirements. - -The policy uses Azure Content Safety's text analysis API to evaluate content and blocks requests or responses that exceed configured severity thresholds. This enables enterprise-grade content filtering for LLM applications integrated with Azure services. - -## Features - -- **Multi-category detection**: Detects hate speech, sexual content, self-harm, and violence -- **Configurable severity thresholds**: Set per-category thresholds (0-7) or disable categories -- **Eight severity levels**: Uses Azure's 8-level severity scale (0=Safe, 7=Most severe) -- **JSONPath support**: Extract and validate specific fields within JSON payloads -- **Separate request/response configuration**: Independent configuration for request and response phases -- **Detailed assessment information**: Optional detailed violation information in error responses -- **Error handling**: Configurable passthrough behavior on API errors -- **Retry logic**: Automatic retry with exponential backoff for transient API failures - -## Configuration - -### Parameters - -#### Request Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `passthroughOnError` | boolean | No | `false` | If `true`, allows requests to proceed if Azure Content Safety API call fails. If `false`, blocks requests on API errors. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information in error responses. | -| `hateCategory` | integer | No | `-1` | Severity threshold for hate category (0-7). `-1` disables this category. Content with severity >= threshold will be blocked. | -| `sexualCategory` | integer | No | `-1` | Severity threshold for sexual category (0-7). `-1` disables this category. Content with severity >= threshold will be blocked. | -| `selfHarmCategory` | integer | No | `-1` | Severity threshold for self-harm category (0-7). `-1` disables this category. Content with severity >= threshold will be blocked. | -| `violenceCategory` | integer | No | `-1` | Severity threshold for violence category (0-7). `-1` disables this category. Content with severity >= threshold will be blocked. | - -#### Response Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `passthroughOnError` | boolean | No | `false` | If `true`, allows requests to proceed if Azure Content Safety API call fails. If `false`, blocks requests on API errors. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information in error responses. | -| `hateCategory` | integer | No | `-1` | Severity threshold for hate category (0-7). `-1` disables this category. Content with severity >= threshold will be blocked. | -| `sexualCategory` | integer | No | `-1` | Severity threshold for sexual category (0-7). `-1` disables this category. Content with severity >= threshold will be blocked. | -| `selfHarmCategory` | integer | No | `-1` | Severity threshold for self-harm category (0-7). `-1` disables this category. Content with severity >= threshold will be blocked. | -| `violenceCategory` | integer | No | `-1` | Severity threshold for violence category (0-7). `-1` disables this category. Content with severity >= threshold will be blocked. | - -### System Parameters (Required) - -These parameters are typically configured at the gateway level and automatically injected, or you can override those values from the params section in the api artifact definition file as well: - -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `azureContentSafetyEndpoint` | string | Yes | Azure Content Safety API endpoint URL (without trailing slash). Example: `https://your-resource.cognitiveservices.azure.com` | -| `azureContentSafetyKey` | string | Yes | Azure Content Safety API subscription key for authentication. Found in Azure Portal under your Content Safety resource's "Keys and Endpoint" section. | - -### Configuring System Parameters in config.toml - -System parameters can be configured globally in the gateway's `config.toml` file. These values serve as defaults for all Azure Content Safety guardrail policy instances and can be overridden per-policy in the API configuration if needed. - -#### Location in config.toml - -Add the following configuration section to your `config.toml` file: - -```toml -azurecontentsafety_endpoint = "https://your-resource.cognitiveservices.azure.com" -azurecontentsafety_key = "" -``` - -## Severity Levels - -Azure Content Safety uses an 8-level severity scale (0-7): - -- **0**: Safe - No harmful content detected -- **1-2**: Low severity - Mildly concerning content -- **3-4**: Medium severity - Moderately concerning content -- **5-6**: High severity - Highly concerning content -- **7**: Maximum severity - Most severe harmful content - -**Threshold Configuration**: -- Set a threshold value (0-7) to block content at or above that severity level -- Set to `-1` to disable monitoring for that category -- Example: `hateCategory: 3` blocks content with hate severity >= 3 - -## JSONPath Support - -The guardrail supports JSONPath expressions to extract and validate specific fields within JSON payloads. Common examples: - -- `$.message` - Extracts the `message` field from the root object -- `$.data.content` - Extracts nested content from `data.content` -- `$.items[0].text` - Extracts text from the first item in an array -- `$.messages[0].content` - Extracts content from the first message in a messages array -- `$.messages[-1].content` - Extracts content from the last message in a messages array -- `$.choices[0].message.content` - Extracts assistant response content - -If `jsonPath` is empty or not specified, the entire payload is treated as a string and validated. - -## Examples - -### Example 1: Basic Content Moderation - -Deploy an LLM provider with Azure Content Safety validation: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: azure-safety-provider -spec: - displayName: Azure Content Safety Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - policies: - - name: azure-content-safety-content-moderation - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - jsonPath: "$.messages[0].content" - hateCategory: 2 - sexualCategory: 2 - selfHarmCategory: 2 - violenceCategory: 2 - showAssessment: true - response: - jsonPath: "$.choices[0].message.content" - hateCategory: 2 - sexualCategory: 2 - selfHarmCategory: 2 - violenceCategory: 2 - showAssessment: true -EOF -``` - -**Test the guardrail:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the LLM provider configuration and use localhost to invoke. - -```bash -# Request with harmful content (should fail with HTTP 422) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Summarize this article and suggest ways someone could take revenge on Mr. Anonymous Senator violently." - } - ] - }' - -# Valid request (should pass) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Explain quantum computing in simple terms" - } - ] - }' -``` - -### Example 2: Strict Moderation with All Categories - -Configure strict moderation thresholds: - -```yaml -policies: - - name: azure-content-safety-content-moderation - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - jsonPath: "$.messages[-1].content" - hateCategory: 1 - sexualCategory: 1 - selfHarmCategory: 1 - violenceCategory: 1 - showAssessment: true - passthroughOnError: false - response: - jsonPath: "$.choices[0].message.content" - hateCategory: 1 - sexualCategory: 1 - selfHarmCategory: 1 - violenceCategory: 1 - showAssessment: true -``` - -### Example 3: Selective Category Monitoring - -Monitor only specific categories: - -```yaml -policies: - - name: azure-content-safety-content-moderation - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - jsonPath: "$.messages[0].content" - hateCategory: 3 - sexualCategory: -1 # Disabled - selfHarmCategory: 2 - violenceCategory: -1 # Disabled -``` - -### Example 4: Lenient Moderation - -Allow more content with higher thresholds: - -```yaml -policies: - - name: azure-content-safety-content-moderation - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - jsonPath: "$.messages[0].content" - hateCategory: 5 - sexualCategory: 5 - selfHarmCategory: 4 - violenceCategory: 5 - passthroughOnError: true -``` - -## Use Cases - -1. **Content Safety**: Protect users from harmful, offensive, or inappropriate content in LLM interactions. - -2. **Regulatory Compliance**: Meet content moderation requirements for regulated industries or geographies. - -3. **Brand Safety**: Ensure LLM responses align with brand values and don't generate problematic content. - -4. **User Protection**: Prevent exposure to self-harm content, especially important for mental health applications. - -5. **Community Guidelines**: Enforce community standards for user-generated content processed through LLMs. - -6. **Multi-tenant Applications**: Apply different moderation policies per tenant or application context. - -7. **Gradual Rollout**: Start with lenient thresholds and tighten based on actual content patterns. - -8. **Audit and Analytics**: Use detailed assessment information to analyze content patterns and refine policies. - -## Severity Threshold Guidelines - -**Recommended thresholds by use case**: - -- **Strict (Family-friendly applications)**: 1-2 across all categories -- **Moderate (General business applications)**: 3-4 across all categories -- **Lenient (Technical/professional contexts)**: 5-6 for most categories, disable non-applicable ones -- **Educational/Research**: 4-5 with selective category monitoring - -**Category-specific considerations**: - -- **Hate**: Typically set to 2-3 for most applications -- **Sexual**: Set based on application context (1 for family apps, 3-4 for general use) -- **Self-harm**: Often set lower (1-2) due to safety concerns -- **Violence**: Depends on context (1-2 for general use, higher for educational/historical content) - -## Error Response - -When validation fails, the guardrail returns an HTTP 422 status code with the following structure: - -```json -{ - "type": "AZURE_CONTENT_SAFETY_CONTENT_MODERATION", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "azure-content-safety-content-moderation", - "actionReason": "Violation of Azure content safety content moderation detected.", - "direction": "REQUEST" - } -} -``` - -If `showAssessment` is enabled, additional details are included: - -```json -{ - "type": "AZURE_CONTENT_SAFETY_CONTENT_MODERATION", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "azure-content-safety-content-moderation", - "actionReason": "Violation of Azure content safety content moderation detected.", - "direction": "REQUEST", - "assessments": { - "inspectedContent": "The content that was analyzed", - "categories": [ - { - "category": "Hate", - "severity": 4, - "result": "FAIL" - }, - { - "category": "Violence", - "severity": 2, - "result": "FAIL" - } - ] - } - } -} -``` - -## Notes - -- Azure Content Safety API requires an active Azure subscription and Content Safety resource. -- The API endpoint URL must not include a trailing slash (e.g., `https://resource.cognitiveservices.azure.com`). -- API keys are found in Azure Portal under your Content Safety resource's "Keys and Endpoint" section. -- Category thresholds are independent - you can disable any category by setting it to `-1`. -- Only categories with thresholds >= 0 are sent to the Azure API for analysis (performance optimization). -- JSONPath extraction failures result in error responses unless `passthroughOnError: true`. -- The policy validates both request and response phases independently when both are configured. -- Content is sent to Azure Content Safety API for analysis, so ensure compliance with data residency requirements. -- Rate limits may apply based on your Azure Content Safety subscription tier. -- The API uses Azure's 8-severity-level analysis, providing fine-grained control over content moderation. -- For production deployments, monitor API response times and adjust retry/timeout settings if needed. - diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/content-length.md b/en/docs/ai-gateway/next/llm-proxy/guardrails/content-length.md deleted file mode 100644 index 75d75f0cf..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/content-length.md +++ /dev/null @@ -1,207 +0,0 @@ ---- -title: "Content Length Guardrail" -description: "Enforce minimum and maximum byte length limits on LLM request and response payloads using the Content Length guardrail." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/content-length/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/content-length.md -tags: - - ai-gateway - - guardrails - - policies -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# Content Length Guardrail - -## Overview - -The Content Length Guardrail validates the byte length of request or response body content against configurable minimum and maximum thresholds. This guardrail is essential for controlling payload sizes, preventing resource exhaustion, and ensuring efficient data transfer. - -## Features - -- Validates byte length against minimum and maximum thresholds -- Supports JSONPath extraction to validate specific fields within JSON payloads -- Configurable inverted logic to pass when content length is outside the range -- Separate configuration for request and response phases -- Optional detailed assessment information in error responses - -## Configuration - -### Parameters - -#### Request Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `min` | integer | Yes | - | Minimum allowed byte length (inclusive). Must be >= 0. | -| `max` | integer | Yes | - | Maximum allowed byte length (inclusive). Must be >= 1. | -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `invert` | boolean | No | `false` | If `true`, validation passes when content length is NOT within the min-max range. If `false`, validation passes when content length is within the range. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information in error responses. | - -#### Response Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `min` | integer | Yes | - | Minimum allowed byte length (inclusive). Must be >= 0. | -| `max` | integer | Yes | - | Maximum allowed byte length (inclusive). Must be >= 1. | -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `invert` | boolean | No | `false` | If `true`, validation passes when content length is NOT within the min-max range. If `false`, validation passes when content length is within the range. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information in error responses. | - -## JSONPath Support - -The guardrail supports JSONPath expressions to extract and validate specific fields within JSON payloads. Common examples: - -- `$.message` - Extracts the `message` field from the root object -- `$.data.content` - Extracts nested content from `data.content` -- `$.items[0].text` - Extracts text from the first item in an array -- `$.messages[0].content` - Extracts content from the first message in a messages array - -If `jsonPath` is empty or not specified, the entire payload is treated as a string and validated. - -## Examples - -### Example 1: Basic Content Length Validation - -Deploy an LLM provider that limits request payloads to between 100 bytes and 1MB: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: content-length-provider -spec: - displayName: Content Length Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - - path: /models - methods: [GET] - - path: /models/{modelId} - methods: [GET] - policies: - - name: content-length-guardrail - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - min: 100 - max: 1048576 -EOF -``` - -**Test the guardrail:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the llm provider configuration and use localhost to invoke. - -```bash -# Valid request (should pass) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Please explain artificial intelligence in simple terms for beginners" - } - ] - }' - -# Invalid request - too small (should fail with HTTP 422) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Hi" - } - ] - }' -``` - -### Additional Configuration Options - -You can customize the guardrail behavior by modifying the `policies` section: - -- **Request and Response Validation**: Configure both `request` and `response` parameters to validate byte lengths in both directions. Use `showAssessment: true` to include detailed assessment information in error responses. - -- **Inverted Logic**: Set `invert: true` to allow only content *outside* the specified byte range. This is useful for blocking content that falls within a prohibited size range. - -- **Full Payload Validation**: Omit the `jsonPath` parameter to validate the entire request body without JSONPath extraction. - -- **Field-Specific Validation**: Use `jsonPath` to extract and validate specific fields within JSON payloads (e.g., `"$.messages[0].content"` for message content or `"$.choices[0].message.content"` for response content). - -## Use Cases - -1. **Resource Protection**: Prevent excessively large payloads that could exhaust system resources or cause performance degradation. - -2. **Network Optimization**: Control payload sizes to optimize network transfer times and reduce bandwidth costs. - -3. **Storage Management**: Limit content sizes to manage storage requirements effectively. - -4. **API Rate Limiting**: Enforce size constraints as part of rate limiting strategies. - -5. **Quality Assurance**: Ensure responses meet minimum size requirements for completeness. - -## Error Response - -When validation fails, the guardrail returns an HTTP 422 status code with the following structure: - -```json -{ - "type": "CONTENT_LENGTH_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "content-length-guardrail", - "actionReason": "Violation of applied content length constraints detected.", - "direction": "REQUEST" - } -} -``` - -If `showAssessment` is enabled, additional details are included: - -```json -{ - "type": "CONTENT_LENGTH_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "content-length-guardrail", - "actionReason": "Violation of applied content length constraints detected.", - "assessments": "Violation of content length detected. Expected between 10 and 100 bytes.", - "direction": "REQUEST" - } -} -``` - -## Notes - -- Byte length is calculated on the UTF-8 encoded representation of the content. -- When using JSONPath, if the path does not exist or the extracted value is not a string, validation will fail. -- Inverted logic is useful for blocking content that falls outside acceptable size ranges. -- Consider network and storage constraints when setting maximum values. -- Minimum values help ensure content quality and completeness. diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/json-schema.md b/en/docs/ai-gateway/next/llm-proxy/guardrails/json-schema.md deleted file mode 100644 index 6a3a7b41e..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/json-schema.md +++ /dev/null @@ -1,234 +0,0 @@ ---- -title: "JSON Schema Guardrail" -description: "Validate LLM request and response body content against a JSON Schema definition to enforce structured data formats." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/json-schema/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/json-schema.md -tags: - - ai-gateway - - guardrails - - policies -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# JSON Schema Guardrail - -## Overview - -The JSON Schema Guardrail validates request or response body content against a JSON Schema definition. This guardrail enables structured data validation, ensuring that JSON payloads conform to expected formats, data types, and constraints. - -## Features - -- Validates content against JSON Schema Draft 7 -- Supports JSONPath extraction to validate specific fields within JSON payloads -- Configurable inverted logic to pass when schema validation fails -- Separate configuration for request and response phases -- Detailed validation error information in error responses - -## Configuration - -### Parameters - -#### Request Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `schema` | string | Yes | - | JSON Schema as a string (must be valid JSON). Supports all JSON Schema draft 7 features. | -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload for validation. If empty, validates the entire payload against the schema. | -| `invert` | boolean | No | `false` | If `true`, validation passes when schema validation FAILS. If `false`, validation passes when schema validation succeeds. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed validation error information in error responses. | - -#### Response Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `schema` | string | Yes | - | JSON Schema as a string (must be valid JSON). Supports all JSON Schema draft 7 features. | -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload for validation. If empty, validates the entire payload against the schema. | -| `invert` | boolean | No | `false` | If `true`, validation passes when schema validation FAILS. If `false`, validation passes when schema validation succeeds. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed validation error information in error responses. | - -## JSONPath Support - -The guardrail supports JSONPath expressions to extract and validate specific fields within JSON payloads. Common examples: - -- `$.data` - Extracts the `data` object for validation -- `$.userInfo` - Extracts user information object -- `$.items[0]` - Extracts the first item in an array -- `$.messages[0]` - Extracts the first message object - -If `jsonPath` is empty or not specified, the entire payload is validated against the schema. - -## JSON Schema Features - -The guardrail supports JSON Schema Draft 7, including: - -- **Types**: `string`, `number`, `integer`, `boolean`, `object`, `array`, `null` -- **Properties**: Define object properties and their schemas -- **Required Fields**: Specify which properties are mandatory -- **Constraints**: `minLength`, `maxLength`, `minimum`, `maximum`, `pattern`, `enum` -- **Nested Structures**: Complex nested objects and arrays -- **Conditional Logic**: `if`, `then`, `else`, `allOf`, `anyOf`, `oneOf`, `not` - -## Examples - -### Example 1: Basic Object Validation - -Deploy an LLM provider that validates that request contains a user object with required fields: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: json-schema-provider -spec: - displayName: JSON Schema Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - - path: /models - methods: [GET] - - path: /models/{modelId} - methods: [GET] - policies: - - name: json-schema-guardrail - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - schema: | - { - "type": "object", - "properties": { - "name": {"type": "string", "minLength": 1}, - "email": {"type": "string", "format": "email"}, - "age": {"type": "integer", "minimum": 18} - }, - "required": ["name", "email"] - } -EOF -``` - -**Test the guardrail:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the llm provider configuration and use localhost to invoke. - -```bash -# Valid request (should pass) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Hello" - } - ], - "name": "John Doe", - "email": "john@example.com", - "age": 25 - }' - -# Invalid request - missing required fields (should fail with HTTP 422) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Hello" - } - ] - }' -``` - -### Additional Configuration Options - -You can customize the guardrail behavior by modifying the `policies` section: - -- **Request and Response Validation**: Configure both `request` and `response` parameters to validate JSON schemas in both directions. Use `showAssessment: true` to include detailed validation error information in error responses. - -- **Inverted Logic**: Set `invert: true` to allow only content that does *not* match the schema. This is useful for blocking requests that match specific schema patterns. - -- **Full Payload Validation**: Omit the `jsonPath` parameter to validate the entire request body against the schema. - -- **Field-Specific Validation**: Use `jsonPath` to extract and validate specific fields within JSON payloads (e.g., `"$.messages[0]"` for message objects or `"$.results"` for response arrays). - -## Use Cases - -1. **API Contract Enforcement**: Ensure requests and responses conform to API specifications. - -2. **Data Quality**: Validate data structure and types before processing. - -3. **Security**: Enforce required fields and prevent injection of unexpected data structures. - -4. **Integration**: Ensure compatibility with downstream systems that expect specific formats. - -5. **Compliance**: Enforce data formats required by regulatory standards. - -## Error Response - -When validation fails, the guardrail returns an HTTP 422 status code with the following structure: - -```json -{ - "type": "JSON_SCHEMA_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "json-schema-guardrail", - "actionReason": "Violation of JSON schema detected.", - "direction": "REQUEST" - } -} -``` - -If `showAssessment` is enabled, detailed validation errors are included: - -```json -{ - "type": "JSON_SCHEMA_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "json-schema-guardrail", - "actionReason": "Violation of JSON schema detected.", - "assessments": [ - { - "description": "String length must be greater than or equal to 5", - "field": "messages.0.content", - "value": "Hi" - } - ], - "direction": "REQUEST" - } -} -``` - -## Notes - -- The schema must be valid JSON. Use proper escaping when embedding in YAML. -- JSON Schema Draft 7 is supported with all standard features. -- When using JSONPath, if the path does not exist or the extracted value is not valid JSON, validation will fail. -- Inverted logic is useful for blocking content that matches specific schema patterns. -- Complex schemas may impact performance; test thoroughly with expected content volumes. -- The guardrail validates the structure and types but does not validate business logic or semantic meaning. diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/overview.md b/en/docs/ai-gateway/next/llm-proxy/guardrails/overview.md deleted file mode 100644 index c28259c64..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/overview.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -title: "Guardrails overview" -description: "Overview of AI Gateway guardrails: LLM-aware policies for content filtering, safety, and compliance that run in the LLM Proxy request and response pipeline." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/overview/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/overview.md -tags: - - ai-gateway - - guardrails - - llm-proxy -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-10 -content_type: "concept" ---- - -# Guardrails overview - -Guardrails are policies that run in the LLM Proxy's request and response pipeline to validate, filter, or transform content before it reaches an LLM or is returned to the client. - -Guardrails use the same underlying policy engine as [API Gateway policies](../../../../../api-gateway/next/policies/overview.md). Each guardrail declares which execution phases it participates in, and the engine calls the appropriate hook for each phase. - -## What guardrails do - -- **Content filtering**: Block or flag requests and responses that violate configured topic, word, or content policies. -- **PII detection and masking**: Detect and mask or redact personally identifiable information in prompts and responses. -- **Schema validation**: Enforce structure on requests or responses using JSON Schema. -- **Pattern matching**: Detect prohibited content using regular expressions. -- **Length and count limits**: Enforce word count, sentence count, and byte length constraints on prompts and responses. -- **External validation**: Delegate content validation to managed services such as AWS Bedrock Guardrails or Azure Content Safety. - -## Available guardrails - -| Guardrail | What it does | -|-----------|-------------| -| [AWS Bedrock Guardrail](aws-bedrock-guardrail.md) | Validates content against AWS Bedrock Guardrails for content filtering, topic detection, and PII masking | -| [Azure Content Safety](azure-content-safety.md) | Validates content against Azure Content Safety API for content moderation | -| [Content Length Guardrail](content-length.md) | Enforces a maximum byte length on request or response content | -| [JSON Schema Guardrail](json-schema.md) | Validates request or response content against a JSON Schema | -| [PII Masking Regex Guardrail](pii-masking-regex.md) | Detects and masks PII using configurable regex patterns | -| [Regex Guardrail](regex.md) | Validates content against a regular expression pattern | -| [Semantic Prompt Guardrail](semantic-prompt-guard.md) | Blocks or allows prompts based on semantic similarity to configured allow or deny phrases | -| [Sentence Count Guardrail](sentence-count.md) | Enforces a sentence count limit on request or response content | -| [URL Guardrail](url.md) | Validates URLs found in request or response content | -| [Word Count Guardrail](word-count.md) | Enforces a word count limit on request or response content | - -## How guardrails execute - -When multiple guardrails are attached to an LLM Proxy, they run as an ordered chain across request and response phases. The AI Gateway's architecture — with a separate LLM Proxy chain and LLM Provider chain — means every request passes through two chains in sequence. - -For a full explanation of phase execution, multi-guardrail ordering, and the dual-hop execution model specific to AI Gateway, see [Guardrail execution order](execution-order.md). diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/pii-masking-regex.md b/en/docs/ai-gateway/next/llm-proxy/guardrails/pii-masking-regex.md deleted file mode 100644 index b6144a164..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/pii-masking-regex.md +++ /dev/null @@ -1,213 +0,0 @@ ---- -title: "PII Masking Regex Guardrail" -description: "Mask or redact personally identifiable information from LLM request and response bodies using configurable regex patterns." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/pii-masking-regex/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/pii-masking-regex.md -tags: - - ai-gateway - - guardrails - - security -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# PII Masking Regex Guardrail - -## Overview - -The PII Masking Regex Guardrail masks or redacts Personally Identifiable Information (PII) from request and response bodies using configurable regular expression patterns. This guardrail helps protect sensitive user data by replacing PII with placeholders or redaction markers before content is processed or returned. - -## Features - -- Configurable PII entity detection using regular expressions -- Two modes: masking (reversible) and redaction (permanent) -- Automatic PII restoration in responses when using masking mode -- Supports JSONPath extraction to process specific fields within JSON payloads - -## Configuration - -### Parameters - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `piiEntities` | array | Yes | - | Array of PII entity configurations. Each entity contains `piiEntity` (name/type) and `piiRegex` (regular expression pattern). | -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, processes the entire payload as a string. | -| `redactPII` | boolean | No | `false` | If `true`, redacts PII by replacing with "*****" (permanent, cannot be restored). If `false`, masks PII with placeholders that can be restored in responses. | - -### PII Entity Configuration - -Each PII entity in the `piiEntities` array must contain: - -| Field | Type | Required | Description | -|-------|------|----------|-------------| -| `piiEntity` | string | Yes | Name/type of the PII entity (e.g., "EMAIL", "PHONE", "SSN", "CREDIT_CARD"). Must contain only uppercase letters and underscores (matches `^[A-Z_]+$`). | -| `piiRegex` | string | Yes | Regular expression pattern to match the PII entity. Must be a valid Go regexp pattern. | - -## JSONPath Support - -The guardrail supports JSONPath expressions to extract and process specific fields within JSON payloads. Common examples: - -- `$.message` - Extracts the `message` field from the root object -- `$.data.content` - Extracts nested content from `data.content` -- `$.items[0].text` - Extracts text from the first item in an array -- `$.messages[0].content` - Extracts content from the first message in a messages array - -If `jsonPath` is empty or not specified, the entire payload is processed as a string. - -## PII Masking Modes - -### Masking Mode (`redactPII: false`) - -- PII is replaced with placeholders in the format `[ENTITY_TYPE_XXXX]` where XXXX is a 4-digit hexadecimal number (e.g., `[EMAIL_0000]`, `[EMAIL_0001]`, `[PHONE_000a]`) -- Placeholders are automatically restored in responses to their original values -- Original PII values are stored temporarily in request metadata for restoration -- Recommended when you need to preserve data for downstream processing or response generation - -### Redaction Mode (`redactPII: true`) - -- PII is permanently replaced with "*****" -- Cannot be restored in responses -- More secure but loses original data -- Recommended for maximum privacy protection when original values are not needed - -## Examples - -### Example 1: Basic PII Masking - -Deploy an LLM provider that masks email addresses and phone numbers in requests and restores them in responses: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: pii-masking-provider -spec: - displayName: PII Masking Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - - path: /models - methods: [GET] - - path: /models/{modelId} - methods: [GET] - policies: - - name: pii-masking-regex - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - piiEntities: - - piiEntity: "EMAIL" - piiRegex: "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}" - - piiEntity: "PHONE" - piiRegex: "\\+?[1-9]\\d{1,14}" - jsonPath: "$.messages[0].content" - redactPII: true -EOF -``` - -**Test the guardrail:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the llm provider configuration and use localhost to invoke. - -```bash -# Request with PII (should be masked) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Contact me at john.doe@example.com or call +1234567890" - } - ] - }' -``` - -### Additional Configuration Options - -You can customize the guardrail behavior by modifying the `policies` section: - -- **PII Redaction**: Set `redactPII: true` to permanently replace PII with "*****" (cannot be restored). Set `redactPII: false` to use masking mode with placeholders that can be restored in responses. - -- **Multiple PII Types**: Configure multiple `piiEntities` in the array to detect and mask/redact various PII types (e.g., EMAIL, PHONE, CREDIT_CARD, SSN, IP_ADDRESS, DATE_OF_BIRTH). - -- **Full Payload Processing**: Omit the `jsonPath` parameter to process the entire request body without JSONPath extraction. - -- **Field-Specific Processing**: Use `jsonPath` to extract and process PII from specific fields within JSON payloads (e.g., `"$.messages[0].content"` for message content). - -- **Response Restoration**: When using masking mode (`redactPII: false`), PII is automatically restored in responses. No separate response configuration is needed. If `redactPII: true`, no restoration occurs in the response phase. - -## Use Cases - -1. **Privacy Protection**: Mask or redact PII before sending data to AI services or external systems. - -2. **Compliance**: Meet regulatory requirements (GDPR, CCPA, HIPAA) for PII handling. - -3. **Data Minimization**: Reduce exposure of sensitive data in logs, analytics, or third-party integrations. - -4. **Secure Processing**: Allow AI processing while protecting user privacy through masking. - -5. **Audit Trail**: Maintain masked versions of data for auditing while protecting original values. - -## How It Works - -### Request Phase (Masking) - -1. Extract content using JSONPath (if specified) or use entire payload -2. Apply each PII regex pattern to find matches -3. Replace matches with placeholders (`[ENTITY_TYPE_XXXX]`) or redaction markers (`*****`) -4. Store mapping of placeholders to original values (for masking mode) -5. Forward masked content to upstream service - -### Response Phase (Restoration) - -1. Check if PII was masked in the request phase (metadata contains PII mappings) -2. If `redactPII: false` and mappings exist, replace placeholders with original PII values -3. If `redactPII: true`, no restoration is performed (PII was permanently redacted) -4. Return restored content or original response - - -#### Sample Payload after intervention from Regex PII Masking with redact=true - -``` -{ - "messages": [ - { - "role": "user", - "content": "Prepare an email with my contact information, email: *****, and website: https://example.com." - } - ] -} -``` - -## Notes - -- Regular expressions use Go's regexp package (RE2 syntax). -- PII detection is case-sensitive by default. Use `(?i)` flag for case-insensitive matching. -- The `piiEntity` name must contain only uppercase letters and underscores (e.g., "EMAIL", "PHONE_NUMBER", "SSN"). -- When using masking mode, the placeholder-to-original mapping is stored in request metadata and automatically used for response restoration. -- Multiple PII entities can match the same content; each match is processed according to its entity type. -- Placeholder format is `[ENTITY_TYPE_XXXX]` where XXXX is a 4-digit hexadecimal number (e.g., `[EMAIL_0000]`, `[EMAIL_0001]`, `[PHONE_000a]`). -- When using JSONPath, if the path does not exist or the extracted value is not a string, an error response (HTTP 500) is returned. -- Redaction mode is irreversible; use masking mode if you need to restore PII in responses. -- Complex regex patterns may impact performance; test thoroughly with expected content volumes. diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/regex.md b/en/docs/ai-gateway/next/llm-proxy/guardrails/regex.md deleted file mode 100644 index f7f682dc6..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/regex.md +++ /dev/null @@ -1,217 +0,0 @@ ---- -title: "Regex Guardrail" -description: "Validate LLM request and response content against regular expression patterns to enforce formats or detect prohibited content." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/regex/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/regex.md -tags: - - ai-gateway - - guardrails - - policies -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# Regex Guardrail - -## Overview - -The Regex Guardrail validates request or response body content against regular expression patterns. This guardrail enables pattern-based content validation, allowing you to enforce specific formats, detect prohibited patterns, or ensure content matches expected structures. - -## Features - -- Pattern matching using regular expressions -- Supports JSONPath extraction to validate specific fields within JSON payloads -- Configurable inverted logic to pass when pattern does not match -- Separate configuration for request and response phases -- Optional detailed assessment information in error responses - -## Configuration - -### Parameters - -#### Request Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `regex` | string | Yes | - | Regular expression pattern to match against the content. Must be at least 1 character. | -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `invert` | boolean | No | `false` | If `true`, validation passes when regex does NOT match. If `false`, validation passes when regex matches. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information in error responses. | - -#### Response Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `regex` | string | Yes | - | Regular expression pattern to match against the content. Must be at least 1 character. | -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `invert` | boolean | No | `false` | If `true`, validation passes when regex does NOT match. If `false`, validation passes when regex matches. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information in error responses. | - -## JSONPath Support - -The guardrail supports JSONPath expressions to extract and validate specific fields within JSON payloads. Common examples: - -- `$.message` - Extracts the `message` field from the root object -- `$.data.content` - Extracts nested content from `data.content` -- `$.items[0].text` - Extracts text from the first item in an array -- `$.messages[0].content` - Extracts content from the first message in a messages array - -If `jsonPath` is empty or not specified, the entire payload is treated as a string and validated. - -## Regular Expression Syntax - -The guardrail uses Go's standard regexp package, which supports RE2 syntax. Key features: - -- Case-sensitive matching by default -- Use `(?i)` flag for case-insensitive matching -- Anchors: `^` (start), `$` (end) -- Character classes: `[a-z]`, `[0-9]`, `\d`, `\w`, `\s` -- Quantifiers: `*`, `+`, `?`, `{n}`, `{n,m}` -- Groups and alternation: `(abc|def)`, `(?:non-capturing)` - -## Examples - -### Example 1: Email Validation - -Deploy an LLM provider that protects against sensitive data leaks by blocking any payloads that mention the word "password" (case-insensitive) in either the user’s message or the LLM’s response. This is achieved by using the regex policy to validate both request and response payloads: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: regex-provider -spec: - displayName: Regex Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - - path: /models - methods: [GET] - - path: /models/{modelId} - methods: [GET] - policies: - - name: regex-guardrail - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - regex: "(?i).*password.*" - invert: true - jsonPath: "$.messages[0].content" -EOF -``` - -**Test the guardrail:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the llm provider configuration and use localhost to invoke. - -```bash -# Valid request (should pass) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "This is a safe message without sensitive data" - } - ] - }' - -# Invalid request - no email (should fail with HTTP 422) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "My password is 1234567" - } - ] - }' -``` - -### Additional Configuration Options - -You can customize the guardrail behavior by modifying the `policies` section: - -- **Request and Response Validation**: Configure both `request` and `response` parameters to validate patterns in both directions. Use `showAssessment: true` to include detailed assessment information in error responses. - -- **Inverted Logic**: Set `invert: true` to allow only content that does *not* match the regex pattern. This is useful for blocking prohibited patterns (e.g., password-related content, admin keywords). - -- **Full Payload Validation**: Omit the `jsonPath` parameter to validate the entire request body without JSONPath extraction. - -- **Field-Specific Validation**: Use `jsonPath` to extract and validate specific fields within JSON payloads (e.g., `"$.messages[0].content"` for message content or `"$.choices[0].message.content"` for response content). - -## Use Cases - -1. **Format Validation**: Ensure user inputs match expected formats (emails, phone numbers, IDs). - -2. **Content Filtering**: Block or allow content based on pattern matching (prohibited words, sensitive patterns). - -3. **Security Enforcement**: Detect and block potentially malicious patterns or injection attempts. - -4. **Data Quality**: Ensure responses follow specific formatting requirements or contain required elements. - -5. **Compliance**: Enforce patterns required by regulatory standards or business rules. - -## Error Response - -When validation fails, the guardrail returns an HTTP 422 status code with the following structure: - -```json -{ - "type": "REGEX_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "regex-guardrail", - "actionReason": "Violation of regular expression detected.", - "direction": "REQUEST" - } -} -``` - -If `showAssessment` is enabled, additional details are included: - -```json -{ - "type": "REGEX_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "regex-guardrail", - "actionReason": "Violation of regular expression detected.", - "assessments": "Violation of regular expression detected. (?i)ignore\\s+all\\s+previous\\s+instructions", - "direction": "REQUEST" - } -} -``` - -## Notes - -- Regular expressions are evaluated using Go's regexp package (RE2 syntax). -- Pattern matching is case-sensitive by default. Use `(?i)` flag for case-insensitive matching. -- When using JSONPath, if the path does not exist or the extracted value is not a string, validation will fail. -- Inverted logic is useful for blocking content that matches prohibited patterns. -- Complex regex patterns may impact performance; test thoroughly with expected content volumes. diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/semantic-prompt-guard.md b/en/docs/ai-gateway/next/llm-proxy/guardrails/semantic-prompt-guard.md deleted file mode 100644 index 6b63d7daf..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/semantic-prompt-guard.md +++ /dev/null @@ -1,410 +0,0 @@ ---- -title: "Semantic Prompt Guardrail" -description: "Block semantically similar prompts using embedding-based similarity matching against configured allow and deny phrase lists." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/semantic-prompt-guard/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/semantic-prompt-guard.md -tags: - - ai-gateway - - guardrails - - llm -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# Semantic Prompt Guardrail - -## Overview - -The Semantic Prompt Guardrail validates prompts using semantic similarity matching against configured allow and deny phrase lists. Unlike keyword-based filtering, this guardrail understands the meaning of prompts by converting them to vector embeddings and comparing them using cosine similarity. This enables more intelligent content filtering that can catch semantically similar content even when exact keywords differ. - -The policy uses embedding models (OpenAI, Mistral, or Azure OpenAI) to convert prompts and configured phrases into high-dimensional vectors, then performs similarity comparisons. Prompts are blocked if they are too similar to denied phrases or not similar enough to allowed phrases, based on configurable similarity thresholds. - -## Features - -- **Semantic similarity matching**: Uses embeddings to understand meaning, not just keywords -- **Allow/Deny phrase lists**: Configure lists of allowed and denied phrases for flexible filtering -- **Configurable similarity thresholds**: Control matching sensitivity separately for allow and deny lists (0.0 to 1.0) -- **Multiple embedding provider support**: Works with OpenAI, Mistral, and Azure OpenAI embedding services -- **JSONPath extraction**: Extract specific fields from request body for validation -- **Detailed assessment information**: Optional detailed violation information in error responses - -## How It Works - -1. **Text Extraction**: Extracts prompt text from the request body using JSONPath (if configured) or uses the entire request body -2. **Embedding Generation**: Generates a vector embedding from the extracted prompt using the configured embedding provider -3. **Validation Strategy**: The validation logic depends on which lists are configured: - - **Deny list only**: Compares prompt embedding against all denied phrases. If any denied phrase has similarity >= `denySimilarityThreshold`, the request is blocked. Otherwise, it proceeds. - - **Allow list only**: Compares prompt embedding against all allowed phrases. If no allowed phrase has similarity >= `allowSimilarityThreshold`, the request is blocked. Otherwise, it proceeds. - - **Both lists**: First checks the deny list (blocks if similarity >= `denySimilarityThreshold`), then checks the allow list (blocks if similarity < `allowSimilarityThreshold`). Request proceeds only if it passes both checks. -4. **Validation Result**: Request proceeds if validation passes, or is blocked with HTTP 422 if validation fails - -## Configuration - -### Parameters - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `jsonPath` | string | No | `""` | JSONPath expression to extract the prompt from JSON payload. If empty, validates the entire payload as a string. Examples: `"$.messages[0].content"`, `"$.prompt"` | -| `allowSimilarityThreshold` | number | No | `0.65` | Minimum similarity threshold (0.0 to 1.0) for a prompt to be considered similar to an allowed phrase. Higher values mean stricter matching. If set, the prompt must match at least one allowed phrase within this threshold. | -| `denySimilarityThreshold` | number | No | `0.65` | Maximum similarity threshold (0.0 to 1.0) for a prompt to be considered similar to a denied phrase. If any denied phrase has similarity >= this threshold, the request is blocked. Higher values mean stricter blocking. | -| `allowedPhrases` | array | No* | `[]` | List of phrases that are considered safe. The prompt must match one of these within `allowSimilarityThreshold` if the threshold is configured. Embeddings are automatically generated during policy initialization. | -| `deniedPhrases` | array | No* | `[]` | List of phrases that should block the prompt when similar within the `denySimilarityThreshold`. Embeddings are automatically generated during policy initialization. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information in error responses. If `false`, returns minimal error information. | - -\* At least one of `allowedPhrases` or `deniedPhrases` must be provided. - -### System Parameters (Required) - -These parameters are typically configured at the gateway level and automatically injected, or you can override those values from the params section in the api artifact definition file as well: - -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `embeddingProvider` | string | Yes | Embedding provider type. Must be one of: `OPENAI`, `MISTRAL`, `AZURE_OPENAI` | -| `embeddingEndpoint` | string | Yes | Endpoint URL for the embedding service. Examples: OpenAI: `https://api.openai.com/v1/embeddings`, Mistral: `https://api.mistral.ai/v1/embeddings`, Azure OpenAI: Your Azure OpenAI endpoint URL | -| `embeddingModel` | string | Conditional | - | Embedding model name. **Required for OPENAI and MISTRAL**, not required for AZURE_OPENAI (deployment name is in endpoint URL). Examples: OpenAI: `text-embedding-ada-002` or `text-embedding-3-small`, Mistral: `mistral-embed` | -| `apiKey` | string | Yes | API key for the embedding service authentication | - -### Configuring System Parameters in config.toml - -System parameters can be configured globally in the gateway's `config.toml` file. These values serve as defaults for all Semantic Prompt Guard policy instances and can be overridden per-policy in the API configuration if needed. - -#### Location in config.toml - -Add the following configuration section to your `config.toml` file: - -```toml -embedding_provider = "MISTRAL" # Supported: MISTRAL, OPENAI, AZURE_OPENAI -embedding_provider_endpoint = "https://api.mistral.ai/v1/embeddings" -embedding_provider_model = "mistral-embed" -embedding_provider_dimension = 1024 -embedding_provider_api_key = "" -``` - -## Similarity Threshold Guidelines - -The similarity thresholds control how similar prompts must be to trigger allow/deny decisions: - -### Allow Similarity Threshold - -- **0.95-1.0**: Very strict matching. Only near-identical prompts to allowed phrases will pass. Use for exact-match scenarios. -- **0.85-0.94**: Recommended for most use cases. Catches semantically equivalent prompts with some wording variation. -- **0.75-0.84**: More flexible matching. Useful for broader conceptual similarity. -- **0.60-0.74**: Very flexible. May allow loosely related prompts. -- **Below 0.60**: Not recommended. Risk of allowing unrelated prompts. - -**Recommendation**: Start with 0.65 and adjust based on your use case. Monitor false positives/negatives to fine-tune. - -### Deny Similarity Threshold - -- **0.95-1.0**: Very strict blocking. Only near-identical prompts to denied phrases will be blocked. -- **0.85-0.94**: Recommended for most use cases. Catches semantically equivalent prompts with some wording variation. -- **0.75-0.84**: More flexible blocking. Useful for catching variations of prohibited content. -- **0.60-0.74**: Very flexible. May block loosely related prompts. -- **Below 0.60**: Not recommended. Risk of blocking legitimate prompts. - -**Recommendation**: Start with 0.65 and adjust based on your use case. Monitor false positives to fine-tune. - -## JSONPath Support - -The guardrail supports JSONPath expressions to extract specific text from request bodies before validation. This is useful for: -- Extracting message content from chat completion requests -- Focusing on specific prompt fields while ignoring metadata -- Handling structured JSON payloads - -### Common JSONPath Examples - -- `$.messages[0].content` - First message's content in chat completions -- `$.messages[-1].content` - Last message's content -- `$.prompt` - Extract prompt field from completions API -- `$.input` - Extract input field from embeddings API -- `$` - Entire request body (default if jsonPath is not specified) - -## Examples - -### Example 1: Deny List Only - Blocking Prohibited Content - -Deploy an LLM provider that blocks prompts similar to prohibited phrases: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: semantic-guard-provider -spec: - displayName: Semantic Guard Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - policies: - - name: semantic-prompt-guard - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - jsonPath: "$.messages[0].content" - denySimilarityThreshold: 0.80 - deniedPhrases: - - "How to hack into a system" - - "Create malicious code" - - "Bypass security measures" - showAssessment: true -EOF -``` - -**Test the guardrail:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the LLM provider configuration and use localhost to invoke. - -```bash -# Valid request (should pass) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Explain how computer security works" - } - ] - }' - -# Invalid request - similar to denied phrase (should fail with HTTP 422) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "How can I break into a computer system?" - } - ] - }' -``` - -### Example 2: Allow List Only - Whitelist Approach - -Deploy an LLM provider that only allows prompts similar to approved phrases: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: whitelist-provider -spec: - displayName: Whitelist Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - policies: - - name: semantic-prompt-guard - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - jsonPath: "$.messages[0].content" - allowSimilarityThreshold: 0.75 - allowedPhrases: - - "How can I help you with customer service?" - - "What product information do you need?" - - "Tell me about your order status" - - "I need help with my account" -EOF -``` - -### Example 3: Combined Allow and Deny Lists - -Use both allow and deny lists for comprehensive filtering: - -```yaml -policies: - - name: semantic-prompt-guard - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - jsonPath: "$.messages[0].content" - allowSimilarityThreshold: 0.70 - denySimilarityThreshold: 0.75 - allowedPhrases: - - "Customer service inquiry" - - "Product information request" - - "Technical support question" - deniedPhrases: - - "How to hack" - - "Create malware" - - "Bypass authentication" - showAssessment: true -``` - -### Example 4: Azure OpenAI with Custom Timeout - -Configure semantic prompt guardrail with Azure OpenAI and extended timeout: - -```yaml -policies: - - name: semantic-prompt-guard - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - jsonPath: "$.messages[-1].content" - denySimilarityThreshold: 0.80 - deniedPhrases: - - "Prohibited content example" - - "Another prohibited phrase" -``` - -## Use Cases - -1. **Content Safety**: Block prompts that are semantically similar to prohibited content, even when exact keywords differ. - -2. **Whitelist Filtering**: Only allow prompts that match approved use cases or topics, ensuring LLM usage stays within defined boundaries. - -3. **Compliance**: Enforce content policies by blocking prompts similar to non-compliant examples. - -4. **Abuse Prevention**: Detect and block variations of known abuse patterns, even when attackers try to evade keyword filters. - -5. **Domain Restriction**: Restrict LLM usage to specific domains by allowing only prompts similar to approved domain-specific phrases. - -6. **Multi-tenant Security**: Apply different allow/deny lists per tenant or application to enforce tenant-specific content policies. - -7. **Prompt Injection Prevention**: Block prompts that are semantically similar to known prompt injection attacks. - -8. **Quality Control**: Ensure prompts match expected patterns for better response quality and consistency. - -## Error Response - -When validation fails, the guardrail returns an HTTP 422 status code with the following structure: - -```json -{ - "type": "SEMANTIC_PROMPT_GUARD", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "semantic-prompt-guard", - "actionReason": "Violation of applied semantic prompt guard constraints detected.", - "direction": "REQUEST" - } -} -``` - -If `showAssessment` is enabled, additional details are included in the `assessments` field: - -```json -{ - "type": "SEMANTIC_PROMPT_GUARD", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "semantic-prompt-guard", - "actionReason": "Violation of applied semantic prompt guard constraints detected.", - "direction": "REQUEST", - "assessments": "prompt is too similar to denied phrase 'How to hack into a system' (similarity=0.8500)" - } -} -``` - -For allow list violations, the assessment message format is: - -```json -{ - "type": "SEMANTIC_PROMPT_GUARD", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "semantic-prompt-guard", - "actionReason": "Violation of applied semantic prompt guard constraints detected.", - "direction": "REQUEST", - "assessments": "prompt is not similar enough to allowed phrases (similarity=0.6000 < threshold=0.6500)" - } -} -``` - -For errors during processing (e.g., JSONPath extraction failures, embedding generation errors), the `actionReason` contains the specific error message: - -```json -{ - "type": "SEMANTIC_PROMPT_GUARD", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "semantic-prompt-guard", - "actionReason": "Error extracting value from JSONPath", - "direction": "REQUEST" - } -} -``` - -## Performance Considerations - -1. **Embedding Generation Latency**: Generating embeddings adds ~100-500ms to request processing. This is a one-time cost per request. - -2. **Batch Processing**: All allow/deny phrase embeddings are generated in a single batch during policy initialization, minimizing initialization overhead. - -3. **Similarity Calculation**: Cosine similarity calculations are fast (typically < 10ms) even with many phrases. - -4. **Embedding Provider Selection**: - - OpenAI: Fast, reliable, good for most use cases - - Mistral: Alternative option with good performance - - Azure OpenAI: Good for Azure-integrated environments - - -## Notes - -- The policy validates prompts in the request phase only (before sending to LLM). Response validation is not supported. - -- Embeddings for allow/deny phrases are generated automatically during policy initialization. Ensure the embedding provider is accessible at initialization time. - -- The policy uses cosine similarity to compare embeddings. This is optimal for semantic similarity matching. - -- At least one of `allowedPhrases` or `deniedPhrases` must be provided. An empty list for both will cause policy initialization to fail. - -- Similarity thresholds are independent for allow and deny lists. You can use different thresholds for each list based on your requirements. - -- JSONPath extraction is optional. If not specified, the entire request body (as string) is used for embedding generation. - -- The `embeddingModel` parameter is required for `OPENAI` and `MISTRAL` providers, but not for `AZURE_OPENAI` (the deployment name is included in the endpoint URL). - -- For Azure OpenAI, the authentication header is automatically set to `api-key`. For other providers, it's set to `Authorization`. - -- The policy processes all phrases in batch during initialization for efficiency. Large phrase lists (100+ phrases) may take a few seconds to initialize. - -- Similarity scores range from 0.0 (no similarity) to 1.0 (identical meaning). Higher thresholds mean stricter matching. - -- For production deployments, monitor false positive/negative rates and adjust similarity thresholds accordingly. diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/sentence-count.md b/en/docs/ai-gateway/next/llm-proxy/guardrails/sentence-count.md deleted file mode 100644 index c534aab5a..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/sentence-count.md +++ /dev/null @@ -1,217 +0,0 @@ ---- -title: "Sentence Count Guardrail" -description: "Enforce minimum and maximum sentence count limits on LLM request and response content using the Sentence Count guardrail." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/sentence-count/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/sentence-count.md -tags: - - ai-gateway - - guardrails - - policies -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# Sentence Count Guardrail - -## Overview - -The Sentence Count Guardrail validates the sentence count of request or response body content against configurable minimum and maximum thresholds. This guardrail is useful for ensuring content completeness, controlling response verbosity, and maintaining consistent communication standards. - -## Features - -- Validates sentence count against minimum and maximum thresholds -- Supports JSONPath extraction to validate specific fields within JSON payloads -- Configurable inverted logic to pass when sentence count is outside the range -- Separate configuration for request and response phases -- Optional detailed assessment information in error responses - -## Configuration - -### Parameters - -#### Request Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `min` | integer | Yes | - | Minimum allowed sentence count (inclusive). Must be >= 0. | -| `max` | integer | Yes | - | Maximum allowed sentence count (inclusive). Must be >= 1. | -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `invert` | boolean | No | `false` | If `true`, validation passes when sentence count is NOT within the min-max range. If `false`, validation passes when sentence count is within the range. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information in error responses. | - -#### Response Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `min` | integer | Yes | - | Minimum allowed sentence count (inclusive). Must be >= 0. | -| `max` | integer | Yes | - | Maximum allowed sentence count (inclusive). Must be >= 1. | -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `invert` | boolean | No | `false` | If `true`, validation passes when sentence count is NOT within the min-max range. If `false`, validation passes when sentence count is within the range. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information in error responses. | - -## JSONPath Support - -The guardrail supports JSONPath expressions to extract and validate specific fields within JSON payloads. Common examples: - -- `$.message` - Extracts the `message` field from the root object -- `$.data.content` - Extracts nested content from `data.content` -- `$.items[0].text` - Extracts text from the first item in an array -- `$.messages[0].content` - Extracts content from the first message in a messages array - -If `jsonPath` is empty or not specified, the entire payload is treated as a string and validated. - -## Sentence Detection - -Sentences are detected based on standard sentence-ending punctuation marks: -- Period (.) -- Exclamation mark (!) -- Question mark (?) - -The guardrail counts sequences of characters ending with these punctuation marks as sentences. - -## Examples - -### Example 1: Basic Sentence Count Validation - -Deploy an LLM provider that ensures requests contain between 1 and 10 sentences: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: sentence-count-provider -spec: - displayName: Sentence Count Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - - path: /models - methods: [GET] - - path: /models/{modelId} - methods: [GET] - policies: - - name: sentence-count-guardrail - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - min: 2 - max: 10 - jsonPath: "$.messages[0].content" -EOF -``` - -**Test the guardrail:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the llm provider configuration and use localhost to invoke. - -```bash -# Valid request (should pass) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "What is machine learning?. How does it work?. Can you explain it simply?" - } - ] - }' - -# Invalid request - too few sentences (should fail with HTTP 422) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Hi" - } - ] - }' -``` - -### Additional Configuration Options - -You can customize the guardrail behavior by modifying the `policies` section: - -- **Request and Response Validation**: Configure both `request` and `response` parameters to validate sentence counts in both directions. Use `showAssessment: true` to include detailed assessment information in error responses. - -- **Inverted Logic**: Set `invert: true` to allow only content *outside* the specified sentence range. This is useful for blocking content that falls within a prohibited sentence count range. - -- **Full Payload Validation**: Omit the `jsonPath` parameter to validate the entire request body without JSONPath extraction. - -- **Field-Specific Validation**: Use `jsonPath` to extract and validate specific fields within JSON payloads (e.g., `"$.messages[0].content"` for message content or `"$.choices[0].message.content"` for response content). - -## Use Cases - -1. **Content Quality Assurance**: Ensure responses meet minimum sentence requirements for completeness and clarity. - -2. **Response Length Control**: Limit verbosity to maintain concise communication standards. - -3. **Input Validation**: Ensure user prompts contain sufficient context (minimum sentences) without being excessive. - -4. **Consistency Enforcement**: Maintain consistent response formats across different AI interactions. - -5. **Cost Management**: Control response length to manage token usage and associated costs. - -## Error Response - -When validation fails, the guardrail returns an HTTP 422 status code with the following structure: - -```json -{ - "type": "SENTENCE_COUNT_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "sentence-count-guardrail", - "actionReason": "Violation of applied sentence count constraints detected.", - "direction": "REQUEST" - } -} -``` - -If `showAssessment` is enabled, additional details are included: - -```json -{ - "type": "SENTENCE_COUNT_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "sentence-count-guardrail", - "actionReason": "Violation of applied sentence count constraints detected.", - "assessments": "Violation of sentence count detected. Expected between 1 and 3 sentences.", - "direction": "REQUEST" - } -} -``` - -## Notes - -- Sentence counting is performed on the extracted or full content after trimming whitespace. -- Sentences are identified by standard punctuation marks (., !, ?). -- When using JSONPath, if the path does not exist or the extracted value is not a string, validation will fail. -- Inverted logic is useful for blocking content that falls outside acceptable sentence count ranges. -- Consider the nature of your content when setting thresholds, as some content types may naturally have different sentence counts. diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/url.md b/en/docs/ai-gateway/next/llm-proxy/guardrails/url.md deleted file mode 100644 index a9b000e7a..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/url.md +++ /dev/null @@ -1,237 +0,0 @@ ---- -title: "URL Guardrail" -description: "Validate URLs found in LLM request and response content by checking reachability through DNS resolution or HTTP HEAD requests." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/url/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/url.md -tags: - - ai-gateway - - guardrails - - policies -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# URL Guardrail - -## Overview - -The URL Guardrail validates URLs found in request or response body content by checking their reachability and validity. This guardrail helps prevent broken links, malicious URLs, and ensures that referenced resources are accessible. - -## Features - -- Validates URLs via DNS resolution or HTTP HEAD requests -- Supports JSONPath extraction to validate specific fields within JSON payloads -- Configurable timeout for URL validation -- Separate configuration for request and response phases -- Optional detailed assessment information including invalid URLs in error responses - -## Configuration - -### Parameters - -#### Request Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `onlyDNS` | boolean | No | `false` | If `true`, validates URLs only via DNS resolution (faster, less reliable). If `false`, validates URLs via HTTP HEAD request (slower, more reliable). | -| `timeout` | integer | No | `3000` | Timeout in milliseconds for DNS lookup or HTTP HEAD request. Default is 3000ms (3 seconds). | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information including invalid URLs in error responses. | - -#### Response Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `onlyDNS` | boolean | No | `false` | If `true`, validates URLs only via DNS resolution (faster, less reliable). If `false`, validates URLs via HTTP HEAD request (slower, more reliable). | -| `timeout` | integer | No | `3000` | Timeout in milliseconds for DNS lookup or HTTP HEAD request. Default is 3000ms (3 seconds). | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information including invalid URLs in error responses. | - -## JSONPath Support - -The guardrail supports JSONPath expressions to extract and validate specific fields within JSON payloads. Common examples: - -- `$.message` - Extracts the `message` field from the root object -- `$.data.content` - Extracts nested content from `data.content` -- `$.items[0].text` - Extracts text from the first item in an array -- `$.messages[0].content` - Extracts content from the first message in a messages array - -If `jsonPath` is empty or not specified, the entire payload is treated as a string and validated. - -## URL Validation Modes - -### DNS-Only Validation (`onlyDNS: true`) - -- Faster validation method -- Only checks if the domain name resolves via DNS -- Does not verify HTTP/HTTPS accessibility -- Less reliable for detecting broken links -- Suitable for quick validation when HTTP checks are not necessary - -### HTTP HEAD Request Validation (`onlyDNS: false`) - -- More thorough validation method -- Performs DNS lookup and HTTP HEAD request -- Verifies that the URL is actually reachable -- More reliable for detecting broken or inaccessible URLs -- Slower due to network request overhead -- Recommended for production use - -## Examples - -### Example 1: Basic URL Validation - -Deploy an LLM provider that validates URLs in request content using HTTP HEAD requests: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: url-guardrail-provider -spec: - displayName: URL Guardrail Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - - path: /models - methods: [GET] - - path: /models/{modelId} - methods: [GET] - policies: - - name: url-guardrail - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - jsonPath: "$.messages[0].content" - onlyDNS: false - timeout: 5000 -EOF -``` - -**Test the guardrail:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the llm provider configuration and use localhost to invoke. - -```bash -# Valid request (should pass) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Visit https://www.example.com for more information" - } - ] - }' - -# Invalid request - invalid URL (should fail with HTTP 422) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Visit https://invalid-url-that-does-not-exist-12345.com" - } - ] - }' -``` - -### Additional Configuration Options - -You can customize the guardrail behavior by modifying the `policies` section: - -- **Request and Response Validation**: Configure both `request` and `response` parameters to validate URLs in both directions. Use `showAssessment: true` to include detailed assessment information including invalid URLs in error responses. - -- **DNS-Only Validation**: Set `onlyDNS: true` for faster validation that only checks DNS resolution. This is less reliable but faster than HTTP HEAD validation. - -- **HTTP HEAD Validation**: Set `onlyDNS: false` (default) for more thorough validation that performs both DNS lookup and HTTP HEAD request to verify URL reachability. - -- **Timeout Configuration**: Adjust the `timeout` parameter (in milliseconds) based on network conditions and acceptable latency. Default is 3000ms (3 seconds). - -- **Full Payload Validation**: Omit the `jsonPath` parameter to validate URLs in the entire request body without JSONPath extraction. - -- **Field-Specific Validation**: Use `jsonPath` to extract and validate URLs from specific fields within JSON payloads (e.g., `"$.messages[0].content"` for message content or `"$.choices[0].message.content"` for response content). - -## Use Cases - -1. **Link Validation**: Ensure all URLs in AI-generated content are valid and accessible. - -2. **Security**: Detect and block potentially malicious or suspicious URLs. - -3. **Quality Assurance**: Prevent broken links from being included in responses. - -4. **Content Moderation**: Validate URLs before allowing them in user-generated content. - -5. **Resource Verification**: Ensure referenced resources are available before processing. - -## Error Response - -When validation fails, the guardrail returns an HTTP 422 status code with the following structure: - -```json -{ - "type": "URL_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "url-guardrail", - "actionReason": "Violation of url validity detected.", - "direction": "REQUEST" - } -} -``` - -If `showAssessment` is enabled, additional details including invalid URLs are included: - -```json -{ - "type": "URL_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "url-guardrail", - "actionReason": "Violation of url validity detected.", - "assessments": { - "invalidUrls": [ - "http://example.com/suspicious-link", - "https://foo.bar.baz" - ], - "message": "One or more URLs in the payload failed validation." - }, - "direction": "REQUEST" - } -} -``` - -## Notes - -- URL validation extracts all URLs from the content using pattern matching. -- DNS-only validation is faster but less reliable than HTTP HEAD validation. -- Timeout values should be set based on network conditions and acceptable latency. -- HTTP HEAD requests may fail for URLs that require specific headers or authentication. -- Some URLs may be temporarily unavailable; consider retry logic for production use. -- When using JSONPath, if the path does not exist or the extracted value is not a string, validation will fail. -- The guardrail validates all URLs found in the content; if any URL is invalid, validation fails. diff --git a/en/docs/ai-gateway/next/llm-proxy/guardrails/word-count.md b/en/docs/ai-gateway/next/llm-proxy/guardrails/word-count.md deleted file mode 100644 index ee9070463..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/guardrails/word-count.md +++ /dev/null @@ -1,206 +0,0 @@ ---- -title: "Word Count Guardrail" -description: "Enforce minimum and maximum word count limits on LLM request and response content using the Word Count guardrail." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/word-count/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/word-count.md -tags: - - ai-gateway - - guardrails - - policies -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# Word Count Guardrail - -## Overview - -The Word Count Guardrail validates the word count of request or response body content against configurable minimum and maximum thresholds. This guardrail is useful for enforcing content length policies, ensuring responses meet quality standards, or preventing excessively long inputs that could impact system performance. - -## Features - -- Validates word count against minimum and maximum thresholds -- Supports JSONPath extraction to validate specific fields within JSON payloads -- Configurable inverted logic to pass when word count is outside the range -- Separate configuration for request and response phases -- Optional detailed assessment information in error responses - -## Configuration - -### Parameters - -#### Request Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `min` | integer | Yes | - | Minimum allowed word count (inclusive). Must be >= 0. | -| `max` | integer | Yes | - | Maximum allowed word count (inclusive). Must be >= 1. | -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `invert` | boolean | No | `false` | If `true`, validation passes when word count is NOT within the min-max range. If `false`, validation passes when word count is within the range. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information in error responses. | - -#### Response Phase - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `min` | integer | Yes | - | Minimum allowed word count (inclusive). Must be >= 0. | -| `max` | integer | Yes | - | Maximum allowed word count (inclusive). Must be >= 1. | -| `jsonPath` | string | No | `""` | JSONPath expression to extract a specific value from JSON payload. If empty, validates the entire payload as a string. | -| `invert` | boolean | No | `false` | If `true`, validation passes when word count is NOT within the min-max range. If `false`, validation passes when word count is within the range. | -| `showAssessment` | boolean | No | `false` | If `true`, includes detailed assessment information in error responses. | - -## JSONPath Support - -The guardrail supports JSONPath expressions to extract and validate specific fields within JSON payloads. Common examples: - -- `$.message` - Extracts the `message` field from the root object -- `$.data.content` - Extracts nested content from `data.content` -- `$.items[0].text` - Extracts text from the first item in an array -- `$.messages[0].content` - Extracts content from the first message in a messages array - -If `jsonPath` is empty or not specified, the entire payload is treated as a string and validated. - -## Examples - -### Example 1: Basic Word Count Validation - -Deploy an LLM provider that validates request messages contain between 10 and 500 words: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: word-count-provider -spec: - displayName: Word Count Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - - path: /models - methods: [GET] - - path: /models/{modelId} - methods: [GET] - policies: - - name: word-count-guardrail - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - request: - min: 5 - max: 500 - jsonPath: "$.messages[0].content" -EOF -``` - -**Test the guardrail:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the llm provider configuration and use localhost to invoke. - -```bash -# Valid request (should pass) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Please explain artificial intelligence in simple terms for beginners" - } - ] - }' - -# Invalid request - too few words (should fail with HTTP 422) -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Hi" - } - ] - }' -``` - -### Additional Configuration Options - -You can customize the guardrail behavior by modifying the `policies` section: - -- **Request and Response Validation**: Configure both `request` and `response` parameters to validate word counts in both directions. Use `showAssessment: true` to include detailed assessment information in error responses. - -- **Inverted Logic**: Set `invert: true` to allow only content *outside* the specified word range. This is useful for blocking content that falls within a prohibited range. - -- **Full Payload Validation**: Omit the `jsonPath` parameter to validate the entire request body without JSONPath extraction. - -- **Field-Specific Validation**: Use `jsonPath` to extract and validate specific fields within JSON payloads (e.g., `"$.messages[0].content"` for message content or `"$.choices[0].message.content"` for response content). - - -## Use Cases - -1. **Input Length Control**: Prevent users from submitting extremely long prompts that could impact system performance or costs. - -2. **Response Quality Assurance**: Ensure AI-generated responses meet minimum length requirements for completeness. - -3. **Cost Management**: Limit response lengths to control token usage and associated costs. - -4. **Content Filtering**: Use inverted logic to block content that falls outside acceptable word count ranges. - -## Error Response - -When validation fails, the guardrail returns an HTTP 422 status code with the following structure: - -```json -{ - "type": "WORD_COUNT_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "word-count-guardrail", - "actionReason": "Violation of applied word count constraints detected.", - "direction": "REQUEST" - } -} -``` - -If `showAssessment` is enabled, additional details are included: - -```json -{ - "type": "WORD_COUNT_GUARDRAIL", - "message": { - "action": "GUARDRAIL_INTERVENED", - "interveningGuardrail": "word-count-guardrail", - "actionReason": "Violation of applied word count constraints detected.", - "assessments": "Violation of word count detected. Expected between 2 and 10 words.", - "direction": "REQUEST" - } -} -``` - -## Notes - -- Word counting is performed on the extracted or full content after trimming whitespace. -- The validation is case-sensitive and counts all words separated by whitespace. -- When using JSONPath, if the path does not exist or the extracted value is not a string, validation will fail. -- Inverted logic is useful for blocking content that falls outside acceptable ranges rather than within them. diff --git a/en/docs/ai-gateway/next/llm-proxy/load-balancing/model-round-robin.md b/en/docs/ai-gateway/next/llm-proxy/load-balancing/model-round-robin.md deleted file mode 100644 index a3d55ed6d..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/load-balancing/model-round-robin.md +++ /dev/null @@ -1,243 +0,0 @@ ---- -title: "Model Round Robin" -description: "Distribute LLM requests evenly across multiple AI models in a cyclic round-robin pattern with automatic failure suspension." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/load-balancing/model-round-robin/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/load-balancing/model-round-robin.md -tags: - - ai-gateway - - llm - - load-balancing -author: WSO2 API Platform Documentation Team -last_updated: 2026-07-30 -content_type: "reference" ---- - -# Model Round Robin - -## Overview - -The Model Round Robin policy implements round-robin load balancing for AI models. It distributes requests evenly across multiple configured AI models in a cyclic manner, ensuring equal request allocation over time and preventing overloading of any single model. This policy is useful for distributing load across multiple models, improving availability, and managing resource utilization. - -## Features - -- Even distribution of requests across multiple models in a cyclic pattern -- Automatic model suspension on failures (5xx or 429 responses) -- Configurable suspension duration for failed models -- Multi-provider routing by assigning a provider to each model -- Support for extracting model identifier from payload, headers, query parameters, or path parameters -- Dynamic model selection based on availability - -## Configuration - -### Parameters - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `models` | array | Yes | - | List of models for round-robin distribution. Each entry must have a `model` name and can optionally select a `provider`. | -| `suspendDuration` | integer | No | `30` | Suspend duration in seconds for failed models. If set to 0, failed model knowledge is not persisted. Must be >= 0. | - -### Model Configuration - -Each model in the `models` array is an object with the following properties: - -| Property | Type | Required | Description | -|----------|------|----------|-------------| -| `model` | string | Yes | The AI model name to use for load balancing. | -| `provider` | string | No | Effective provider name to route the selected model to. Omit this field to use the proxy's primary provider. When an additional provider has an `as` alias, use the alias instead of its provider ID. | - -### LLM provider template - -The policy requires `requestModel` configuration from the LLM provider template to extract the model identifier from the request. This configuration is mandatory and must be provided by the LLM provider template. - -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `requestModel.location` | string | Yes | Location of the model identifier: `payload`, `header`, `queryParam`, or `pathParam` | -| `requestModel.identifier` | string | Yes | JSONPath (for payload), header name (for header), query param name (for queryParam), or regex pattern (for pathParam) to extract model | - -## How It Works - -1. **Model Selection**: On each request, the policy selects the next available model in the configured list using a round-robin algorithm. -2. **Model Extraction**: The policy extracts the original model from the request (if configured) and stores it for reference. -3. **Model Modification**: The policy modifies the request to use the selected model based on the `requestModel` configuration. -4. **Failure Handling**: If a model returns a 5xx or 429 response, and `suspendDuration` is configured, the model is suspended for the specified duration. -5. **Availability Check**: Suspended models are skipped during selection until their suspension period expires. - -## Examples - -### Multi-provider round robin - -Configure the policy on an LLM proxy to distribute requests across models from its primary and additional providers: - -```yaml -operationPolicies: - - name: model-round-robin - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - models: - - model: gpt-4o - - model: claude-sonnet-4-5-20250929 - provider: anthropic-provider - - model: anthropic.claude-3-5-sonnet-20240620-v1:0 - provider: bedrock-provider - suspendDuration: 60 -``` - -The entry without `provider` uses the proxy's primary provider. The other entries route to providers declared under `spec.additionalProviders`. See [Multi-Provider Routing for LLM Proxies](../multi-provider-routing.md) for provider, authentication, transformer, and alias configuration. - -### Example 1: Basic Round Robin with Payload-based Model - -Deploy an LLM provider with round-robin load balancing across multiple models: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: round-robin-provider -spec: - displayName: Round Robin Provider - version: v1.0 - template: openai - context: /providers/round-robin - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - operationPolicies: - - name: model-round-robin - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - models: - - model: gpt-4 - - model: gpt-3.5-turbo - - model: gpt-4-turbo - suspendDuration: 60 -EOF -``` - -**Test the round-robin distribution:** - -```bash -# First request - will use gpt-4 -curl -k -X POST https://localhost:8443/providers/round-robin/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Hello" - } - ] - }' - -# Second request - will use gpt-3.5-turbo -curl -k -X POST https://localhost:8443/providers/round-robin/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Hello" - } - ] - }' - -# Third request - will use gpt-4-turbo -curl -k -X POST https://localhost:8443/providers/round-robin/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Hello" - } - ] - }' -``` - -## Model Suspension - -When a model returns a 5xx or 429 response, the policy can automatically suspend that model for a configurable duration: - -- **Suspension Duration**: Configured via the `suspendDuration` parameter (in seconds) -- **Automatic Recovery**: Suspended models are automatically re-enabled after the suspension period expires -- **Availability Check**: Suspended models are skipped during round-robin selection until they recover - -### Suspension Behavior - -- Suspension is tracked per model across all requests -- If all models are suspended, the policy returns HTTP 503 with error: "All models are currently unavailable" -- Suspension period starts from the time of failure - -## Use Cases - -1. **Load Distribution**: Distribute requests evenly across multiple models to prevent overloading any single model. - -2. **High Availability**: Automatically route requests to available models when some models are experiencing issues. - -3. **Cost Optimization**: Distribute requests across different model tiers (e.g., expensive and cheaper models) to balance cost and performance. - -4. **A/B Testing**: Test different models with equal traffic distribution to compare performance and quality. - -5. **Multi-Provider Support**: Distribute requests across models from different providers while maintaining equal distribution. - -## Request Model Locations - -The policy supports extracting the model identifier from different locations in the request: - -### Payload (JSONPath) - -Extract model from JSON payload using JSONPath: - -- **Location**: `payload` -- **Identifier**: JSONPath expression (e.g., `$.model`, `$.messages[0].model`) - -### Header - -Extract model from HTTP header: - -- **Location**: `header` -- **Identifier**: Header name (e.g., `X-Model-Name`, `X-LLM-Model`) - -### Query Parameter - -Extract model from URL query parameter: - -- **Location**: `queryParam` -- **Identifier**: Query parameter name (e.g., `model`, `llm_model`) - -### Path Parameter - -Extract model from URL path using regex: - -- **Location**: `pathParam` -- **Identifier**: Regex pattern to match model in path (e.g., `models/([a-zA-Z0-9.\-]+)`) - -**Note**: For path parameters, the regex pattern should include a capturing group to extract the model name. The policy uses the first capturing group as the model identifier. - -## Notes - -- The round-robin index is maintained per policy instance and increments for each request. -- Model selection is deterministic and follows a strict cyclic pattern. -- The original model from the request is stored in metadata but is replaced with the selected model for routing. -- If `suspendDuration` is 0, failed models are not suspended and will continue to be selected in the round-robin cycle. -- The `requestModel` configuration is required and must be provided by the LLM provider template. diff --git a/en/docs/ai-gateway/next/llm-proxy/load-balancing/model-weighted-round-robin.md b/en/docs/ai-gateway/next/llm-proxy/load-balancing/model-weighted-round-robin.md deleted file mode 100644 index 39b6f391c..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/load-balancing/model-weighted-round-robin.md +++ /dev/null @@ -1,262 +0,0 @@ ---- -title: "Model Weighted Round Robin" -description: "Distribute LLM requests across AI models by assigned weights, giving higher-capacity or lower-cost models proportionally more traffic." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/load-balancing/model-weighted-round-robin/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/load-balancing/model-weighted-round-robin.md -tags: - - ai-gateway - - llm - - load-balancing -author: WSO2 API Platform Documentation Team -last_updated: 2026-07-30 -content_type: "reference" ---- - -# Model Weighted Round Robin - -## Overview - -The Model Weighted Round Robin policy implements weighted round-robin load balancing for AI models. It distributes requests based on predefined weight values assigned to each model, enabling probabilistic control over request distribution and giving higher priority to models with greater processing power or availability. This policy is useful for distributing load proportionally across models based on their capacity, cost, or performance characteristics. - -## Features - -- Weighted distribution of requests across multiple models based on assigned weights -- Proportional request allocation (models with higher weights receive more requests) -- Automatic model suspension on failures (5xx or 429 responses) -- Configurable suspension duration for failed models -- Multi-provider routing by assigning a provider to each weighted model -- Support for extracting model identifier from payload, headers, query parameters, or path parameters -- Dynamic model selection based on availability and weights - -## Configuration - -### Parameters - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `models` | array | Yes | - | List of models with weights for weighted round-robin distribution. Each entry must have a `model` name and `weight`, and can optionally select a `provider`. | -| `suspendDuration` | integer | No | `30` | Suspend duration in seconds for failed models. If set to 0, failed model knowledge is not persisted. Must be >= 0. | - -### Model Configuration - -Each model in the `models` array is an object with the following properties: - -| Property | Type | Required | Description | -|----------|------|----------|-------------| -| `model` | string | Yes | The AI model name to use for load balancing. | -| `weight` | integer | Yes | The weight assigned to this model for distribution. Higher weights mean more requests will be routed to this model. Weight is relative to total weight of all models. Must be at least 1. | -| `provider` | string | No | Effective provider name to route the selected model to. Omit this field to use the proxy's primary provider. When an additional provider has an `as` alias, use the alias instead of its provider ID. | - -### LLM provider template - -The policy requires `requestModel` configuration from the LLM provider template to extract the model identifier from the request. This configuration is mandatory and must be provided by the LLM provider template. - -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `requestModel.location` | string | Yes | Location of the model identifier: `payload`, `header`, `queryParam`, or `pathParam` | -| `requestModel.identifier` | string | Yes | JSONPath (for payload), header name (for header), query param name (for queryParam), or regex pattern (for pathParam) to extract model | - -## How It Works - -1. **Weight Calculation**: During policy initialization, the policy calculates the total weight of all configured models and builds a weighted sequence where each model appears a number of times proportional to its weight. This sequence is built once and reused for all requests. -2. **Model Selection**: On each request, the policy selects the next available model from the pre-computed weighted sequence using a round-robin algorithm. -3. **Model Extraction**: The policy extracts the original model from the request using the `requestModel` configuration and stores it for reference. -4. **Model Modification**: The policy modifies the request to use the selected model based on the `requestModel` configuration. -5. **Failure Handling**: If a model returns a 5xx or 429 response, and `suspendDuration` is configured, the model is suspended for the specified duration. -6. **Availability Check**: Suspended models are skipped during selection until their suspension period expires. - -### Weight Distribution Example - -If you configure three models with weights: -- Model A: weight 3 -- Model B: weight 2 -- Model C: weight 1 - -The weighted sequence would be: `[A, A, A, B, B, C]`, meaning: -- Model A receives 50% of requests (3 out of 6) -- Model B receives 33.3% of requests (2 out of 6) -- Model C receives 16.7% of requests (1 out of 6) - -## Examples - -### Multi-provider weighted round robin - -Configure the policy on an LLM proxy to distribute traffic by weight across models from its primary and additional providers: - -```yaml -operationPolicies: - - name: model-weighted-round-robin - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - models: - - model: gpt-4o - weight: 5 - - model: claude-sonnet-4-5-20250929 - provider: anthropic-provider - weight: 3 - - model: anthropic.claude-3-5-sonnet-20240620-v1:0 - provider: bedrock-provider - weight: 2 - suspendDuration: 60 -``` - -The entry without `provider` uses the proxy's primary provider. The other entries route to providers declared under `spec.additionalProviders`. See [Multi-Provider Routing for LLM Proxies](../multi-provider-routing.md) for provider, authentication, transformer, and alias configuration. - -### Example 1: Basic Weighted Round Robin with Payload-based Model - -Deploy an LLM provider with weighted round-robin load balancing: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: weighted-round-robin-provider -spec: - displayName: Weighted Round Robin Provider - version: v1.0 - template: openai - context: /providers/weighted-round-robin - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - operationPolicies: - - name: model-weighted-round-robin - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - models: - - model: gpt-4 - weight: 3 - - model: gpt-3.5-turbo - weight: 2 - - model: gpt-4-turbo - weight: 1 - suspendDuration: 60 -EOF -``` - -**Test the weighted round-robin distribution:** - -```bash -# Requests will be distributed: 50% gpt-4, 33.3% gpt-3.5-turbo, 16.7% gpt-4-turbo -curl -k -X POST https://localhost:8443/providers/weighted-round-robin/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Hello" - } - ] - }' -``` - -## Model Suspension - -When a model returns a 5xx or 429 response, the policy can automatically suspend that model for a configurable duration: - -- **Suspension Duration**: Configured via the `suspendDuration` parameter (in seconds) -- **Automatic Recovery**: Suspended models are automatically re-enabled after the suspension period expires -- **Availability Check**: Suspended models are skipped during weighted round-robin selection until they recover -- **Weight Preservation**: When a model is suspended, the remaining models continue to be selected based on their relative weights - -### Suspension Behavior - -- If all models are suspended, the policy returns HTTP 503 with error: "All models are currently unavailable" -- Suspension period starts from the time of failure -- When a model is suspended, the weighted sequence is dynamically adjusted to exclude that model - -## Use Cases - -1. **Capacity-Based Distribution**: Distribute requests based on model capacity, giving higher weights to models that can handle more load. - -2. **Cost Optimization**: Route more requests to cheaper models while maintaining some traffic to premium models for quality assurance. - -3. **Performance Tiers**: Prioritize high-performance models for critical requests while using standard models for regular traffic. - -4. **Gradual Migration**: Gradually shift traffic from old models to new models by adjusting weights over time. - -5. **Multi-Provider Balancing**: Distribute requests across models from different providers with different weights based on SLA or cost agreements. - -6. **A/B Testing with Bias**: Test different models with weighted traffic distribution to compare performance while maintaining a bias toward preferred models. - -## Request Model Locations - -The policy supports extracting the model identifier from different locations in the request: - -### Payload (JSONPath) - -Extract model from JSON payload using JSONPath: - -- **Location**: `payload` -- **Identifier**: JSONPath expression (e.g., `$.model`, `$.messages[0].model`) - -### Header - -Extract model from HTTP header: - -- **Location**: `header` -- **Identifier**: Header name (e.g., `X-Model-Name`, `X-LLM-Model`) - -### Query Parameter - -Extract model from URL query parameter: - -- **Location**: `queryParam` -- **Identifier**: Query parameter name (e.g., `model`, `llm_model`) - -### Path Parameter - -Extract model from URL path using regex: - -- **Location**: `pathParam` -- **Identifier**: Regex pattern to match model in path (e.g., `models/([a-zA-Z0-9.\-]+)`) - -**Note**: For path parameters, the regex pattern should include a capturing group to extract the model name. The policy uses the first capturing group as the model identifier. - -## Weight Calculation - -The policy builds a weighted sequence by repeating each model a number of times equal to its weight: - -- **Total Weight**: Sum of all model weights -- **Sequence Length**: Equal to the total weight -- **Distribution**: Each model appears in the sequence `weight` times -- **Proportional Selection**: Over time, each model receives requests proportional to `model_weight / total_weight` - -### Example Weight Distribution - -For models with weights [5, 3, 2]: -- Total weight: 10 -- Sequence: [Model1, Model1, Model1, Model1, Model1, Model2, Model2, Model2, Model3, Model3] -- Model1: 50% of requests -- Model2: 30% of requests -- Model3: 20% of requests - -## Notes - -- The weighted sequence is pre-computed once during policy initialization and reused for all requests. It is not rebuilt on each request. -- The round-robin index is maintained per policy instance and increments for each request. -- Model selection follows the weighted sequence in a deterministic cyclic pattern. -- The original model from the request is stored in metadata but is replaced with the selected model for routing. -- If `suspendDuration` is 0, failed models are not suspended and will continue to be selected in the weighted round-robin cycle. -- Higher weights result in more frequent selection but do not guarantee exact proportional distribution in small request volumes. -- The weighted sequence ensures long-term proportional distribution, but short-term distribution may vary due to suspension and availability. -- The `requestModel` configuration is required and must be provided by the LLM provider template. There is no default behavior. diff --git a/en/docs/ai-gateway/next/llm-proxy/prompt-management/prompt-decorator.md b/en/docs/ai-gateway/next/llm-proxy/prompt-management/prompt-decorator.md deleted file mode 100644 index 344f8ca69..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/prompt-management/prompt-decorator.md +++ /dev/null @@ -1,325 +0,0 @@ ---- -title: "Prompt Decorator" -description: "Prepend or append custom content to LLM prompts, injecting system messages or standard instructions before requests reach the model." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/prompt-management/prompt-decorator/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/prompt-management/prompt-decorator.md -tags: - - ai-gateway - - llm - - prompt-management -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# Prompt Decorator - -## Overview - -The Prompt Decorator policy dynamically modifies prompts by prepending or appending custom content to specific fields in JSON payloads. This policy supports two decoration modes: **text prompt decoration** (for string content fields) and **chat prompt decoration** (for message arrays). It's useful for adding consistent instructions, system messages, or standardized prefixes/suffixes to prompts before they're sent to AI services. - -## Features - -- Two decoration modes: text decoration (string fields) and chat decoration (message arrays) -- Configurable prepend or append behavior -- JSONPath support for targeting specific fields in JSON payloads -- Flexible decoration format: simple strings or structured message objects -- Processes request body only (response phase not supported) - -## Configuration - -### Parameters - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `promptDecoratorConfig` | string | Yes | - | JSON string containing decoration configuration. For text decoration: `{"decoration": "string"}`. For chat decoration: `{"decoration": [{"role": "system", "content": "..."}]}` | -| `jsonPath` | string | Yes | - | JSONPath expression to locate the field to decorate. Use `$.messages[0].content` for text decoration, or `$.messages` for chat decoration. | -| `append` | boolean | No | `false` | If `true`, decoration is appended to the content. If `false`, decoration is prepended (default). | - -## Decoration Modes - -### Mode 1: Text Prompt Decoration - -Text decoration is used when the JSONPath targets a string field (e.g., `$.messages[0].content`). The decoration can be: -- A simple string that gets prepended or appended to the content -- An array of decoration objects (their content fields are concatenated with newlines) - -**Configuration Example:** -```json -{ - "decoration": "Summarize the following content in a concise, neutral, and professional tone. Structure the summary using bullet points if appropriate.\n\n" -} -``` - -**Behavior:** -- Decoration string is prepended or appended to the target content field -- A space is automatically added between the decoration and original content - -### Mode 2: Chat Prompt Decoration - -Chat decoration is used when the JSONPath targets an array field (e.g., `$.messages`). The decoration must be an array of message objects: - -**Configuration Example:** -```json -{ - "decoration": [ - { - "role": "system", - "content": "You are a helpful hotel booking receptionist for the imaginary hotel 'Azure Horizon Resort'. Your job is to collect all the necessary booking details from guests." - } - ] -} -``` - -**Behavior:** -- Decoration messages are prepended or appended to the messages array -- Each decoration object must have `role` and `content` fields -- Multiple decoration messages can be added - -## JSONPath Support - -The decorator supports JSONPath expressions to target specific fields. Common examples: - -- `$.messages[0].content` - First message's content field (text decoration) -- `$.messages[-1].content` - Last message's content field (text decoration) -- `$.messages` - Entire messages array (chat decoration) -- `$.data.text` - Nested text field (text decoration) - -**Array Index Syntax:** -- Use `[0]` for first element, `[1]` for second, etc. -- Use `[-1]` for last element, `[-2]` for second-to-last, etc. - -## Examples - -### Example 1: Text Prompt Decoration - Summarization Directive - -Add a summarization instruction to user prompts: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: summarization-provider -spec: - displayName: Summarization Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - policies: - - name: prompt-decorator - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - promptDecoratorConfig: '{"decoration": "Summarize the following content in a concise, neutral, and professional tone. Structure the summary using bullet points if appropriate.\\n\\n"}' - jsonPath: "$.messages[0].content" - append: false -EOF -``` - -**Test the decorator:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the LLM provider configuration and use localhost to invoke. - -```bash -# Original request -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Large text block to summarize here..." - } - ] - }' - -# After decoration, the request sent to OpenAI will be: -# { -# "model": "gpt-4", -# "messages": [ -# { -# "role": "user", -# "content": "Summarize the following content in a concise, neutral, and professional tone. Structure the summary using bullet points if appropriate.\n\n Large text block to summarize here..." -# } -# ] -# } -``` - -### Example 2: Chat Prompt Decoration - System Persona - -Add a system message to define AI behavior: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: hotel-booking-provider -spec: - displayName: Hotel Booking Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - policies: - - name: prompt-decorator - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - promptDecoratorConfig: '{"decoration": [{"role": "system", "content": "You are a helpful hotel booking receptionist for Azure Horizon Resort. Collect booking details: name, NIC, check-in time, staying duration (nights), and room type (single, double, suite). Ask one detail at a time in a friendly tone."}]}' - jsonPath: "$.messages" - append: false -EOF -``` - -**Test the decorator:** - -```bash -# Original request -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Hi, I would like to book a room." - } - ] - }' - -# After decoration, the request sent to OpenAI will be: -# { -# "model": "gpt-4", -# "messages": [ -# { -# "role": "system", -# "content": "You are a helpful hotel booking receptionist for Azure Horizon Resort. Collect booking details: name, NIC, check-in time, staying duration (nights), and room type (single, double, suite). Ask one detail at a time in a friendly tone." -# }, -# { -# "role": "user", -# "content": "Hi, I would like to book a room." -# } -# ] -# } -``` - -### Example 3: Append Mode - Adding Suffix Instructions - -Append instructions to the end of user messages: - -```yaml -policies: - - name: prompt-decorator - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - promptDecoratorConfig: '{"decoration": "\\n\\nPlease respond in JSON format."}' - jsonPath: "$.messages[-1].content" - append: true -``` - -## Use Cases - -1. **Consistent Instructions**: Prepend standardized instructions or guidelines to all prompts to ensure consistent AI behavior. - -2. **System Personas**: Inject system messages to define AI personality, role, or behavior before user interactions. - -3. **Quality Enhancement**: Add formatting instructions (e.g., "respond in bullet points", "use professional tone") to improve response quality. - -4. **Context Addition**: Prepend contextual information or background details to enrich prompts. - -5. **Multi-turn Conversations**: Add system messages at the beginning of chat conversations to set conversation rules. - -6. **Compliance**: Append compliance-related instructions or disclaimers to prompts. - -7. **Output Formatting**: Add instructions for specific output formats (JSON, markdown, structured text) to prompts. - -## Configuration Reference - -### Text Decoration Configuration - -```json -{ - "decoration": "Your decoration string here" -} -``` - -- Simple string that will be prepended or appended to the target content -- A space is automatically added between decoration and original content - -### Chat Decoration Configuration - -```json -{ - "decoration": [ - { - "role": "system", - "content": "Your system message content" - } - ] -} -``` - -- Array of message objects -- Each object must have `role` (e.g., "system", "user", "assistant") and `content` fields -- Messages are prepended or appended to the messages array in the order specified - -## Error Response - -When the policy encounters an error (e.g., invalid JSONPath, missing fields), it returns an HTTP 500 status code with the following structure: - -```json -{ - "type": "PROMPT_DECORATOR_ERROR", - "message": "Error description here" -} -``` - -## Notes - -- The policy only processes request bodies. -- For text decoration, a space is automatically added between the decoration and original content. -- JSONPath expressions must correctly identify the target field. Invalid paths will result in errors. -- When decorating message arrays, ensure the target field is actually an array of message objects. -- The `append: false` (default) means decoration is prepended. Set `append: true` to append decoration. -- Decoration objects in chat mode must have both `role` and `content` fields; both are required. -- Negative array indices (e.g., `[-1]` for last element) are supported in JSONPath expressions. -- When using text decoration with an array of decoration objects, their content fields are concatenated with newlines (`\n`). diff --git a/en/docs/ai-gateway/next/llm-proxy/prompt-management/prompt-template.md b/en/docs/ai-gateway/next/llm-proxy/prompt-management/prompt-template.md deleted file mode 100644 index f9df7da7a..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/prompt-management/prompt-template.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -title: "Prompt Template" -description: "Replace 'template://' URI patterns in LLM request payloads with predefined, parameterized prompt templates." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/prompt-management/prompt-template/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/prompt-management/prompt-template.md -tags: - - ai-gateway - - llm - - prompt-management -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# Prompt Template - -## Overview - -The Prompt Template policy enables dynamic prompt transformation by replacing `template://` URI patterns in JSON payloads with predefined templates. Template placeholders are resolved using parameters passed in the URI query string, allowing you to standardize and reuse prompts across different API calls. This is particularly useful for AI/LLM APIs where consistent prompt formatting improves response quality and maintainability. - -## Features - -- Pattern-based template matching using `template://` URI format -- Parameter substitution with `[[parameter-name]]` placeholder syntax -- Multiple templates per policy configuration -- JSON-safe string replacement and escaping -- Processes entire JSON payload as string to find and replace patterns - -## Configuration - -### Parameters - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `promptTemplateConfig` | string | Yes | - | JSON string containing an array of template objects. Each template must have a `name` and `prompt` field. Example: `[{"name": "translate", "prompt": "Translate from [[from]] to [[to]]: [[text]]"}]` | - -### Template Configuration Format - -The `promptTemplateConfig` must be a JSON array of template objects: - -```json -[ - { - "name": "template-name", - "prompt": "Template prompt with [[placeholder]] syntax" - } -] -``` - -Each template object contains: -- **name**: Unique identifier for the template (used in `template://` URIs) -- **prompt**: The template string with `[[parameter-name]]` placeholders that will be replaced - -## Template Syntax - -### Template URI Format - -Templates are referenced in JSON payloads using the following URI format: - -``` -template://?=&= -``` - -Example: -``` -template://translate?from=english&to=spanish&text=Hello world -``` - -### Placeholder Syntax - -Within template prompts, use double square brackets to define placeholders: - -``` -[[parameter-name]] -``` - -During resolution, placeholders are replaced with values from the URI query parameters. Parameter names are case-sensitive and must match exactly between the placeholder and the URI parameter. - -Example template: -``` -Translate the following text from [[from]] to [[to]]: [[text]] -``` - -When called with `template://translate?from=english&to=spanish&text=Hello`, the resolved prompt would be: -``` -Translate the following text from english to spanish: Hello -``` - -## Examples - -### Example 1: Translation Template - -Deploy an LLM provider with a translation prompt template: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: translation-provider -spec: - displayName: Translation Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - policies: - - name: prompt-template - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - promptTemplateConfig: '[{"name": "translate", "prompt": "Translate the following text from [[from]] to [[to]]: [[text]]"}]' -EOF -``` - -**Test the template:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the LLM provider configuration and use localhost to invoke. - -```bash -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "template://translate?from=english&to=spanish&text=Hello world" - } - ] - }' -``` - -The policy will transform the request to: - -```json -{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Translate the following text from english to spanish: Hello world" - } - ] -} -``` - -### Example 2: Summarization Template - -Create a template for summarizing content with configurable length: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: summarization-provider -spec: - displayName: Summarization Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - policies: - - name: prompt-template - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - promptTemplateConfig: '[{"name": "summarize", "prompt": "Summarize the following content in [[length]] words: [[content]]"}]' -EOF -``` - -**Test with template:** - -```bash -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "template://summarize?length=50&content=Artificial intelligence is a branch of computer science that aims to create intelligent machines capable of performing tasks that typically require human intelligence." - } - ] - }' -``` - -### Example 3: Multiple Templates - -Configure multiple templates in a single policy: - -```yaml -policies: - - name: prompt-template - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - promptTemplateConfig: | - [ - { - "name": "translate", - "prompt": "Translate from [[from]] to [[to]]: [[text]]" - }, - { - "name": "summarize", - "prompt": "Summarize in [[length]] words: [[content]]" - }, - { - "name": "explain", - "prompt": "Explain [[topic]] to a [[audience]] audience: [[question]]" - } - ] -``` - -## Use Cases - -1. **Standardized Prompts**: Ensure consistent prompt formatting across different API consumers by centralizing prompt definitions. - -2. **Reusable Templates**: Create library of common prompts (translation, summarization, explanation) that can be reused across multiple APIs. - -3. **Parameterized Prompts**: Allow dynamic content insertion while maintaining consistent prompt structure and quality. - -4. **Multi-language Support**: Use templates with language parameters to standardize prompts for different locales. - -5. **Prompt Versioning**: Update prompt templates centrally without requiring changes to client applications. - -## Template Pattern Matching - -The policy uses regex pattern matching to find `template://` URIs in the JSON payload: - -- **Pattern**: `template://[a-zA-Z0-9_-]+\?[^\s"']*` -- **Location**: Searches the entire JSON payload as a string -- **Replacement**: Each matched pattern is replaced with the resolved template string (JSON-escaped) - -### Pattern Details - -- Template names can contain letters, numbers, underscores, and hyphens -- Query parameters can contain any characters except spaces, quotes, or single quotes -- Multiple template:// patterns can exist in a single payload -- Each pattern is resolved independently - -## Error Handling - -If a template:// pattern references a template name that doesn't exist in the configuration, the pattern is left unchanged (no replacement occurs). This allows for graceful handling of missing templates. - -When template resolution fails (e.g., invalid JSON escaping), the specific pattern is skipped and other patterns continue to be processed. - -## Notes - -- Template names are case-sensitive and must match exactly between the URI reference and the configuration. -- Parameter names in placeholders `[[param]]` are case-sensitive and must match query parameter names exactly. -- Query parameter values are URL-decoded before being inserted into templates. -- The resolved template string is JSON-escaped (special characters like quotes, newlines are escaped) before replacement. -- The policy processes the entire JSON payload as a string, so templates can be used anywhere in the JSON structure. -- Multiple `template://` patterns can appear in a single payload and will all be processed. diff --git a/en/docs/ai-gateway/next/llm-proxy/semantic-caching.md b/en/docs/ai-gateway/next/llm-proxy/semantic-caching.md deleted file mode 100644 index 1c4018527..000000000 --- a/en/docs/ai-gateway/next/llm-proxy/semantic-caching.md +++ /dev/null @@ -1,305 +0,0 @@ ---- -title: "Semantic Caching" -description: "Cache LLM responses using vector similarity search so semantically similar queries are served from cache, reducing cost and latency." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/semantic-caching/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/semantic-caching.md -tags: - - ai-gateway - - llm - - caching -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# Semantic Caching - -## Overview - -The Semantic Cache policy enables intelligent response caching for LLM (Large Language Model) APIs using vector similarity search. Unlike traditional key-based caching, semantic caching understands the meaning of requests and can serve cached responses for semantically similar queries, even when the exact wording differs. This dramatically improves performance and reduces costs by avoiding redundant API calls to upstream LLM services. - -The policy uses embedding models to convert request text into high-dimensional vectors, then performs similarity searches in a vector database to find previously cached responses. If a similar request is found within the configured similarity threshold, the cached response is returned immediately without calling the upstream service. - -## Features - -- **Vector-based similarity matching**: Uses embeddings to find semantically similar requests, not just exact matches -- **Multiple embedding provider support**: Works with OpenAI, Mistral, and Azure OpenAI embedding services -- **Multiple vector database support**: Supports Redis and Milvus as vector storage backends -- **Configurable similarity threshold**: Control cache hit sensitivity (0.0 to 1.0) -- **JSONPath extraction**: Extract specific fields from request body for embedding generation -- **Automatic cache management**: Stores successful responses (200) automatically after upstream calls -- **Immediate response on cache hit**: Returns cached response with `X-Cache-Status: HIT` header without upstream call -- **TTL support**: Configurable time-to-live for cache entries - -## How It Works - -### Request Phase - -1. **Text Extraction**: Extracts text from the request body using JSONPath (if configured) or uses the entire request body -2. **Embedding Generation**: Generates a vector embedding from the extracted text using the configured embedding provider -3. **Cache Lookup**: Searches the vector database for semantically similar cached responses using cosine similarity -4. **Threshold Check**: If a similar embedding is found with similarity >= similarityThreshold, returns the cached response immediately -5. **Cache Miss**: If no similar response is found, the request proceeds to the upstream service - -### Response Phase - -1. **Success Check**: Only processes responses with 200 status codes -2. **Embedding Retrieval**: Retrieves the embedding generated during the request phase from metadata -3. **Response Storage**: Stores the response payload along with its embedding in the vector database -4. **TTL Application**: Applies the configured TTL to the cache entry - -## Configuration - -### Policy Parameters - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `similarityThreshold` | number | Yes | - | Similarity threshold for cache hits (0.0 to 1.0). Higher values require more similarity. For example, 0.9 means 90% similarity required. Recommended: 0.85-0.95 for strict matching, 0.70-0.85 for more flexible matching. | -| `jsonPath` | string | No | `""` | JSONPath expression to extract text from request body for embedding generation. If empty, uses the entire request body. Example: `"$.messages[0].content"` to extract the first message's content. | - -### System Parameters (Required) - -These parameters are typically configured at the gateway level and automatically injected, or you can override those values from the params section in the api artifact definition file as well: - -#### Embedding Provider Configuration - -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `embeddingProvider` | string | Yes | Embedding provider type. Must be one of: `OPENAI`, `MISTRAL`, `AZURE_OPENAI` | -| `embeddingEndpoint` | string | Yes | Endpoint URL for the embedding service. Examples: OpenAI: `https://api.openai.com/v1/embeddings`, Mistral: `https://api.mistral.ai/v1/embeddings`, Azure OpenAI: Your Azure OpenAI endpoint URL | -| `embeddingModel` | string | Conditional | - | Embedding model name. **Required for OPENAI and MISTRAL**, not required for AZURE_OPENAI (deployment name is in endpoint URL). Examples: OpenAI: `text-embedding-ada-002` or `text-embedding-3-small`, Mistral: `mistral-embed` | -| `embeddingDimension` | integer | Yes | Dimension of embedding vectors. Common values: 1536 (OpenAI ada-002), 1024 (Mistral). Must match the model's output dimension. | -| `apiKey` | string | Yes | API key for the embedding service authentication. The authentication header is automatically set to `api-key` for Azure OpenAI and `Authorization` for other providers. | - -#### Vector Database Configuration - -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `vectorStoreProvider` | string | Yes | Vector database provider. Must be one of: `REDIS`, `MILVUS` | -| `dbHost` | string | Yes | Vector database host address | -| `dbPort` | integer | Yes | Vector database port number | -| `username` | string | No | Database username for authentication (if required) | -| `password` | string | No | Database password for authentication (if required) | -| `database` | string | No | Database name or index number (for Redis) | -| `ttl` | integer | No | Time-to-live for cache entries in seconds. Default is 3600 (1 hour). Set to 0 for no expiration. | - - -### Configuring System Parameters in config.toml - -System parameters can be configured globally in the gateway's `config.toml` file. These values serve as defaults for all Semantic Cache policy instances and can be overridden per-policy in the API configuration if needed. - -#### Location in config.toml - -Add the following configuration section to your `config.toml` file: - -```toml -embedding_provider = "MISTRAL" # Supported: MISTRAL, OPENAI, AZURE_OPENAI -embedding_provider_endpoint = "https://api.mistral.ai/v1/embeddings" -embedding_provider_model = "mistral-embed" -embedding_provider_dimension = 1024 -embedding_provider_api_key = "" - -vector_db_provider = "REDIS" # Supported: REDIS, MILVUS -vector_db_provider_host = "redis" -vector_db_provider_port = 6379 -vector_db_provider_database = "0" -vector_db_provider_username = "default" -vector_db_provider_password = "default" -vector_db_provider_ttl = 3600 -``` - -## JSONPath Support - -The policy supports JSONPath expressions to extract specific text from request bodies before generating embeddings. This is useful for: -- Extracting message content from chat completion requests -- Focusing on specific prompt fields while ignoring metadata -- Handling structured JSON payloads - -### Common JSONPath Examples - -- `$.messages[0].content` - First message's content in chat completions -- `$.messages[-1].content` - Last message's content -- `$.prompt` - Extract prompt field from completions API -- `$.input` - Extract input field from embeddings API -- `$` - Entire request body (default if jsonPath is not specified) - -## Examples - -### Example 1: OpenAI Embeddings with Redis - -Deploy an LLM provider with semantic caching using OpenAI embeddings and Redis vector store: - -```bash -curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1 -kind: LlmProvider -metadata: - name: cached-chat-provider -spec: - displayName: OpenAI Cached Provider - version: v1.0 - template: openai - vhost: openai - upstream: - url: "https://api.openai.com/v1" - auth: - type: api-key - header: Authorization - value: Bearer - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - policies: - - name: semantic-cache - version: v1 - paths: - - path: /chat/completions - methods: [POST] - params: - similarityThreshold: 0.85 - jsonPath: "$.messages[0].content" -EOF -``` - -**Test the semantic cache:** - -**Note**: Ensure that "openai" is mapped to the appropriate IP address (e.g., 127.0.0.1) in your `/etc/hosts` file, or remove the vhost from the LLM provider configuration and use localhost to invoke. - -```bash -# First request - cache miss, will call upstream -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Explain quantum computing in simple terms" - } - ] - }' - -# Second request with similar but different wording - cache hit! -curl -X POST http://openai:8080/chat/completions \ - -H "Content-Type: application/json" \ - -H "Host: openai" \ - -d '{ - "model": "gpt-4", - "messages": [ - { - "role": "user", - "content": "Can you describe quantum computing using simple language?" - } - ] - }' -# Response will include: X-Cache-Status: HIT -``` - -## Use Cases - -1. **Cost Reduction**: Reduce API costs by serving cached responses for similar queries, especially valuable for expensive LLM API calls. - -2. **Latency Improvement**: Return cached responses in milliseconds instead of waiting for LLM processing (typically 1-5 seconds), significantly improving user experience. - -3. **Rate Limit Management**: Reduce the number of upstream API calls, helping stay within rate limits and avoiding throttling. - -4. **Consistency**: Ensure users receive consistent responses for semantically equivalent queries, improving predictability. - -5. **Offline Resilience**: Serve cached responses even when upstream services experience temporary outages. - -6. **A/B Testing**: Compare cached vs. fresh responses to evaluate prompt engineering improvements. - -7. **Development/Testing**: Speed up development cycles by reusing cached responses during testing. - -8. **High-Traffic Applications**: Handle high volumes of similar queries efficiently without overwhelming upstream services. - -## Similarity Threshold Guidelines - -The `similarityThreshold` parameter controls how similar requests must be to trigger a cache hit: - -- **0.95-1.0**: Very strict matching. Only near-identical requests will hit cache. Use for exact-match scenarios. -- **0.85-0.94**: Recommended for most use cases. Catches semantically equivalent requests with some wording variation. -- **0.75-0.84**: More flexible matching. Useful for broader conceptual similarity. -- **0.60-0.74**: Very flexible. May return cached responses for loosely related queries. -- **Below 0.60**: Not recommended. Risk of returning irrelevant cached responses. - -**Recommendation**: Start with 0.85 and adjust based on your use case. Monitor cache hit rates and response relevance to fine-tune. - -## Cache Behavior - -### Cache Hit - -When a similar request is found: -- Returns cached response immediately (no upstream call) -- Adds `X-Cache-Status: HIT` header to response -- Status code: 200 (from cached response) -- Response time: Typically < 50ms (vs. 1-5 seconds for LLM APIs) - -### Cache Miss - -When no similar request is found: -- Request proceeds to upstream service normally -- Response is cached after successful upstream call (200 status) -- Subsequent similar requests may hit the cache - -### Cache Storage - -- Only successful responses (200 status code) are cached -- Responses are stored with their embeddings in the vector database -- TTL is applied to all cache entries -- Each route/API maintains a separate cache namespace to avoid cross-contamination - -## Error Handling - -The policy is designed to be resilient: - -- **Embedding Generation Failure**: If embedding generation fails, the request proceeds to upstream (cache is skipped) -- **Vector Database Unavailable**: If the vector database is unreachable, requests proceed to upstream -- **Cache Storage Failure**: If storing a response fails, the response is still returned to the client (cache operation is non-blocking) -- **Invalid JSONPath**: If JSONPath extraction fails, the entire request body is used for embedding generation - -These behaviors ensure that caching failures do not block legitimate requests. The system gracefully degrades to direct upstream calls when caching is unavailable. - -## Performance Considerations - -1. **Embedding Generation Latency**: Generating embeddings adds ~100-500ms to request processing. This is offset by cache hits that save 1-5 seconds. - -2. **Vector Database Performance**: - - Redis with RedisSearch: Fast queries, good for smaller datasets (< 1M vectors) - - Milvus: Optimized for large-scale vector search, better for > 1M vectors - -3. **Cache Hit Rate**: Aim for 20-40% cache hit rate for cost-effective caching. Below 10% may not justify the overhead. - -4. **Embedding Dimension**: Higher dimensions (e.g., 1536) provide better accuracy but increase storage and search time. Choose based on your quality requirements. - -5. **Index Creation**: Vector database indexes are created automatically on first use. This may take a few seconds for large datasets. - -## Notes - -- The policy requires both request and response phases to function properly (generates embeddings in request phase, stores responses in response phase). - -- Embedding generation adds latency to each request (~100-500ms). This overhead is typically offset by the performance gains from cache hits. - -- Cache entries are scoped per route/API to prevent cross-contamination between different APIs or routes. - -- Only responses with 200 status code are cached. Errors and non-200 responses are never cached. - -- The similarity search uses cosine similarity to compare embeddings. This is optimal for semantic similarity matching. - -- Vector database indexes are created automatically when the policy is first used. Ensure your vector database has sufficient resources. - -- The policy maintains provider instances per route for efficiency. Configuration changes require policy reinitialization. - -- TTL of 0 means no expiration. Use with caution as it may lead to unbounded cache growth. - -- JSONPath extraction is optional. If not specified, the entire request body (as string) is used for embedding generation. - -- The policy stores embeddings in metadata between request and response phases. Ensure metadata persistence is enabled in your gateway configuration. - -- For production deployments, monitor cache hit rates, embedding generation latency, and vector database performance metrics to optimize configuration. diff --git a/en/docs/ai-gateway/next/mcp-proxy/quick-start-guide.md b/en/docs/ai-gateway/next/mcp-proxy/create-an-mcp-proxy.md similarity index 89% rename from en/docs/ai-gateway/next/mcp-proxy/quick-start-guide.md rename to en/docs/ai-gateway/next/mcp-proxy/create-an-mcp-proxy.md index 5cbf4bb0f..9c7ef8781 100644 --- a/en/docs/ai-gateway/next/mcp-proxy/quick-start-guide.md +++ b/en/docs/ai-gateway/next/mcp-proxy/create-an-mcp-proxy.md @@ -1,33 +1,22 @@ --- -title: "MCP Proxy Quick Start Guide" +title: "Create an MCP proxy" description: "Run API Platform AI Gateway with Docker Compose, configure an MCP proxy, and route your first MCP traffic through the gateway." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/quick-start-guide/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/quick-start-guide.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/create-an-mcp-proxy/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/create-an-mcp-proxy.md tags: - ai-gateway - mcp - quickstart author: WSO2 API Platform Documentation Team -last_updated: 2026-08-05 +last_updated: 2026-08-11 content_type: "quickstart" --- -## Quick Start +# Create an MCP proxy -!!! info "Watch the video walkthrough" - [Check out this quick start on YouTube](https://youtu.be/xt6RIax__dU?rel=0) or watch below. +## Quick start - - -### Using Docker Compose (Recommended) +### Using Docker Compose (recommended) ### Prerequisites @@ -156,7 +145,7 @@ http://localhost:8080/everything/mcp The gateway syncs the artifacts you deploy on it up to [AI Workspace](../../../ai-workspace/next/overview.md), the control plane for AI traffic across your organization. The `everything-mcp-v1.0` proxy you deployed above appears there without being re-declared, in the `default` project named in its `project-id` annotation. See [Manage Gateway-deployed AI artifacts in AI Workspace](../../../ai-workspace/next/sync-gateway-created-artifacts.md). -## Stopping the Gateway +## Stopping the gateway Stop and remove the MCP backend first. diff --git a/en/docs/ai-gateway/next/mcp-proxy/overview.md b/en/docs/ai-gateway/next/mcp-proxy/overview.md new file mode 100644 index 000000000..253320642 --- /dev/null +++ b/en/docs/ai-gateway/next/mcp-proxy/overview.md @@ -0,0 +1,48 @@ +--- +title: "MCP proxy" +description: "Route Model Context Protocol traffic through the AI Gateway with an MCP proxy, and apply authentication, authorization, and access control to that traffic." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/overview/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/overview.md +tags: + - ai-gateway + - mcp + - mcp-proxy +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-12 +content_type: "concept" +--- + +# MCP proxy + +An MCP Proxy routes Model Context Protocol traffic to MCP servers. MCP is a protocol that enables AI assistants to interact with external tools and data sources. With MCP Proxies, you can: + +- Expose MCP servers through a centralized gateway +- Apply authentication and access control to MCP traffic +- Manage multiple MCP servers from a single control plane + +## In this section + +This section contains the following page: + +| Page | What it covers | +|------|----------------| +| [Create an MCP proxy](create-an-mcp-proxy.md) | Run API Platform AI Gateway with Docker Compose, configure an MCP proxy, and route your first MCP traffic through the gateway. | + +## Policies + +MCP proxy policies are documented in the [Policy Hub](https://wso2.com/api-platform/policy-hub), the versioned reference for every API Platform policy. For policy categories and how policies chain, see the [Policy Hub overview](../../../policy-hub/overview.md). + +| Policy | What it does | +|--------|--------------| +| [MCP Authentication](https://wso2.com/api-platform/policy-hub/policies/mcp-auth) | Secures MCP server traffic per the MCP specification authorization profile | +| [MCP Authorization](https://wso2.com/api-platform/policy-hub/policies/mcp-authz) | Validates access to MCP tools, resources, and prompts using JWT claims or OAuth scopes | +| [MCP Access Control](https://wso2.com/api-platform/policy-hub/policies/mcp-acl-list) | Controls which tools, resources, and prompts a caller can reach using allow/deny lists | +| [MCP Rewrite](https://wso2.com/api-platform/policy-hub/policies/mcp-rewrite) | Defines user-facing tool names and maps them to backend capability names | +| [MCP Rate Limit](https://wso2.com/api-platform/policy-hub/policies/mcp-ratelimit) | Applies rate limits to MCP traffic per tool, resource, prompt, or JSON-RPC method | +| [Semantic Tool Filtering](https://wso2.com/api-platform/policy-hub/policies/semantic-tool-filtering) | Filters MCP tools to only those semantically relevant to the user query | + +## Related guides + +- [Build an AI agent that uses aggregated MCP tools from multiple APIs](../../../guides/ai-and-mcp/build-ai-agent-with-multiple-mcp-servers.md) — connects an agent to three independently governed MCP servers through the gateway. +- [Convert a REST API into an MCP tool for Claude Desktop](../../../guides/ai-and-mcp/convert-rest-api-to-mcp-server.md) — exposes a REST API as a governed MCP server and enforces OAuth2 at the gateway. +- [Find and connect to an enterprise MCP server from the MCP Hub](../../../guides/ai-and-mcp/find-and-connect-to-an-enterprise-mcp-server-from-the-mcp-hub.md) — discovers MCP servers in the MCP Hub and connects a client to one. diff --git a/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-acl-list.md b/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-acl-list.md deleted file mode 100644 index 0e3a2fcad..000000000 --- a/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-acl-list.md +++ /dev/null @@ -1,175 +0,0 @@ ---- -title: "MCP ACL List" -description: "Control access to MCP tools, resources, and prompts using allow or deny mode with exceptions in API Platform AI Gateway." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-acl-list/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-acl-list.md -tags: - - ai-gateway - - mcp - - policies -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# MCP ACL List - -## Overview - -The MCP ACL List policy provides access control for Model Context Protocol (MCP) tools, resources, and prompts using an allow/deny mode with exceptions. This policy filters list responses and enforces access rules on request paths based on configured mode and exceptions. Unlike the [MCP Rewrite policy](./mcp-rewrite.md), this policy does not rewrite capability names or modify list entry contents—it purely controls visibility and access. - -The policy operates on three types of MCP capabilities: tools, resources, and prompts. For each type, you can specify a mode (allow or deny) and a list of exceptions. Requests for capabilities not matching the access control rules are rejected with an appropriate error. - -## Features - -- **Tool-Level Access Control**: Allow or deny access to specific tools using allow/deny mode with exceptions. -- **Resource-Level Access Control**: Control access to specific resources (identified by URI) using flexible ACL rules. -- **Prompt-Level Access Control**: Manage access to specific prompts using configurable access modes. -- **Flexible ACL Modes**: Support both allow-with-exceptions and deny-with-exceptions patterns. -- **List Filtering**: Filter list responses to only include capabilities that match the access control rules. -- **Request Path Enforcement**: Enforce the same allow/deny rules on request paths, rejecting access to denied capabilities. - -## Configuration - -The MCP ACL List policy uses a single-level configuration model where all parameters are configured per-MCP-API/route in the API definition YAML. - -### User Parameters (API Definition) - -These parameters are configured per MCP Proxy by the API developer: - -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `tools` | object | No | ACL configuration for tools with `mode` (allow or deny) and optional `exceptions` list. | -| `tools.mode` | string | Yes | ACL mode for tools: "allow" (allow all except exceptions) or "deny" (deny all except exceptions). | -| `tools.exceptions` | array | No | List of tool names that are exceptions to the mode (1-256 characters each). | -| `resources` | object | No | ACL configuration for resources with `mode` (allow or deny) and optional `exceptions` list. | -| `resources.mode` | string | Yes | ACL mode for resources: "allow" (allow all except exceptions) or "deny" (deny all except exceptions). | -| `resources.exceptions` | array | No | List of resource URIs that are exceptions to the mode (1-2048 characters each). | -| `prompts` | object | No | ACL configuration for prompts with `mode` (allow or deny) and optional `exceptions` list. | -| `prompts.mode` | string | Yes | ACL mode for prompts: "allow" (allow all except exceptions) or "deny" (deny all except exceptions). | -| `prompts.exceptions` | array | No | List of prompt names that are exceptions to the mode (1-256 characters each). | - -## Access Control Logic - -For each capability type (tools, resources, prompts): - -- **Missing capability config**: All capabilities of that type are allowed (no restrictions). -- **mode: allow, exceptions: [...]**: Allow all capabilities except those listed in exceptions. -- **mode: deny, exceptions: [...]**: Deny all capabilities except those listed in exceptions. - -## MCP Proxy Definition Examples - -### Example 1: Deny Specific Tools - -Deny access to certain tools while allowing all others: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-acl-list - version: v0.1.0 - params: - tools: - mode: allow - exceptions: - - delete-all - - drop-database - tools: - ... -``` - -### Example 2: Allow Only Specific Resources - -Allow access to only whitelisted resources: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-acl-list - version: v0.1.0 - params: - resources: - mode: deny - exceptions: - - file:///public/documents - - file:///public/images - resources: - ... -``` - -### Example 3: Mixed Access Control - -Apply different access control rules to different capability types: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-acl-list - version: v0.1.0 - params: - tools: - mode: allow - exceptions: - - admin-only-tool - - deprecated-tool - resources: - mode: allow - exceptions: - - file:///internal-resources - prompts: - mode: deny - exceptions: - - standard-prompt - - approved-prompt - tools: - ... -``` - -## Use Cases - -1. **Sensitive Operation Blocking**: Deny access to tools or resources that perform sensitive operations (e.g., delete, modify system configuration). -2. **Public API Restriction**: Allow only specific public resources while denying access to internal resources. -3. **Role-Based Access**: Combine this policy with authentication/authorization policies to implement role-based access control. -4. **Gradual Feature Rollout**: Deny access to beta or experimental tools while they are being tested. -5. **Compliance and Security**: Enforce compliance policies by denying access to resources or tools that are not approved for a specific tenant or environment. -6. **Cost Control**: Deny access to expensive or resource-intensive operations. - -## Comparison with MCP Rewrite Policy - -| Aspect | MCP ACL List | MCP Rewrite | -|--------|--------------|-------------| -| **Primary Purpose** | Access control via allow/deny | Capability name mapping | -| **Rewrites Names** | No | Yes | -| **Filters Lists** | Yes | Yes | -| **Enforces Request Paths** | Yes | Yes | -| **Configuration Complexity** | Simple (mode + exceptions) | Detailed (names, descriptions, targets) | -| **Metadata Modification** | No | Yes | - -Both policies can be used together: use MCP ACL List for access control and MCP Rewrite for name mapping. diff --git a/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-authentication.md b/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-authentication.md deleted file mode 100644 index 2b97837f8..000000000 --- a/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-authentication.md +++ /dev/null @@ -1,175 +0,0 @@ ---- -title: "MCP Authentication" -description: "Secure MCP server traffic by validating JWT access tokens and serving OAuth protected resource metadata in AI Gateway." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-authentication/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-authentication.md -tags: - - ai-gateway - - mcp - - security -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# MCP Authentication - -## Overview - -The MCP Authentication policy is designed to secure traffic to Model Context Protocol (MCP) servers. The Gateway acts as a resource server, protecting MCP resources by validating access tokens presented in requests. This policy leverages the underlying JWT Authentication mechanism for token validation and additionally handles MCP-specific requirements such as serving protected resource metadata. This policy supports the auth requirements mentioned in the [MCP Specification](https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization#introduction). - -## Features - -- **Access Token Validation**: Validates JWT access tokens using configured key managers. Please refer the [JWT Authentication Policy](https://wso2.com/api-platform/policy-hub/policies/jwt-auth) for more information on how the key validation works. -- **Protected Resource Metadata**: Intercepts `GET /.well-known/oauth-protected-resource` requests to return resource metadata, including authorization servers and supported scopes. -- **Standardized Error Handling**: Returns `WWW-Authenticate` headers with `resource_metadata` on authentication failures. -- **Configurable Validation**: Supports issuer, audience, scope, and custom claim validation. -- **Claim Mapping**: Maps token claims to downstream headers. - -## Configuration - -The MCP Authentication policy uses a two-level configuration model: - -- **System Parameters**: Configured by the administrator in `config.toml` under `policy_configurations.mcpauth_v0` or `policy_configurations.jwtauth_v0` depending on the parameter. -- **User Parameters**: Configured per MCP proxy in the configuration yaml. - -### System Parameters (config.toml) - -These parameters are set by the administrator and apply globally to all MCP authentication policies: - -| Parameter | Type | Required | Path | Description | -|-----------|------|----------|----------|-------------| -| `keymanagers` | array | Yes | jwtauth_v0 | List of key manager definitions. Each entry must include a unique `name` and either `jwks` (for remote JWKS or local certificates) configuration. | -| `gatewayhost` | string | No | mcpauth_v0 | The outward-facing gateway host name which will be used when deriving the values related to protected resource metadata in headers and body. The gateway will fall back to this if there are no vhosts defined in the MCP proxy configuration. | - -#### Key Manager Configuration - -Each key manager in the `keymanagers` array supports the following structure: - -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `name` | string | Yes | Unique name for this key manager (used in user-level `issuers` configuration). | -| `issuer` | string | No | Optional issuer (iss) value associated with keys from this provider. | -| `jwks.remote.uri` | string | Conditional | JWKS endpoint URL. Required if using remote JWKS. | -| `jwks.remote.certificatePath` | string | No | Path to CA certificate file for validating self-signed JWKS endpoints. | -| `jwks.remote.skipTlsVerify` | boolean | No | If true, skip TLS certificate verification. Use with caution. | -| `jwks.local.inline` | string | Conditional | Inline PEM-encoded certificate or public key. | -| `jwks.local.certificatePath` | string | Conditional | Path to certificate or public key file. | - -> **Note**: Either `jwks.remote` or `jwks.local` must be specified, but not both. - -### User Parameters (API Definition) - -These parameters are configured per-API/route by the API developer: - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `issuers` | array | No | - | List of issuer names (referencing entries in `system.keymanagers`). This list is sent as `authorization_servers` in the protected resource metadata response. If omitted, all configured key managers are used. | -| `requiredScopes` | array | No | - | List of scopes that should be included in the token. These are also advertised in the protected resource metadata. | -| `audiences` | array | No | - | List of acceptable audience values; token must contain at least one. | -| `requiredClaims` | object | No | - | Map of claimName → expectedValue for custom claim validation. | -| `claimMappings` | object | No | - | Map of claimName → downstream header name to expose claims for downstream services. | - -## System Configuration Example - -Add the following to your `gateway/configs/config.toml` file under `policy_configurations`: - -```toml -[policy_configurations.mcpauth_v0] -gatewayhost = "gw.example.com" - -[policy_configurations.jwtauth_v0] -jwkscachettl = "5m" -jwksfetchtimeout = "5s" -jwksfetchretrycount = 3 -jwksfetchretryinterval = "2s" -allowedalgorithms = ["RS256", "ES256"] -leeway = "30s" -authheaderscheme = "Bearer" -headername = "Authorization" -onfailurestatuscode = 401 -errormessageformat = "json" -errormessage = "Authentication failed." -validateissuer = true - -[[policy_configurations.jwtauth_v0.keymanagers]] -name = "PrimaryIDP" -issuer = "https://idp.example.com/oauth2/token" - -[policy_configurations.jwtauth_v0.keymanagers.jwks.remote] -uri = "https://idp.example.com/oauth2/jwks" -skipTlsVerify = false - -[[policy_configurations.jwtauth_v0.keymanagers]] -name = "SecondaryIDP" -issuer = "https://auth.example.org/oauth2/token" - -[policy_configurations.jwtauth_v0.keymanagers.jwks.remote] -uri = "https://auth.example.org/oauth2/jwks" -skipTlsVerify = false -``` - -## MCP Proxy Definition Examples - -### Example 1: Basic MCP Authentication - -Apply MCP authentication to an API using a specific key manager: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - vhost: mcp1.gw.example.com - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-auth - version: v0.1.1 - params: - issuers: - - PrimaryIDP - tools: - ... -``` - -### Example 2: Scope and Audience Validation - -Require specific scopes and audiences: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - vhost: mcp1.gw.example.com - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-auth - version: v0.1.1 - params: - issuers: - - PrimaryIDP - audiences: - - https://mcp-api.example.com - requiredScopes: - - mcp:read - - mcp:write - tools: - ... -``` - -## Use Cases - -1. **MCP Server Security**: Protect Model Context Protocol servers by requiring valid access tokens from trusted identity providers. -2. **Resource Discovery**: Enable MCP clients to discover authorization requirements (authorization servers and scopes) via the standard `.well-known/oauth-protected-resource` endpoint. -3. **Multi-Provider Support**: Allow MCP clients to authenticate using tokens from different identity providers (e.g., different organizations or tenants). diff --git a/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-authorization.md b/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-authorization.md deleted file mode 100644 index 4c977ce6e..000000000 --- a/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-authorization.md +++ /dev/null @@ -1,271 +0,0 @@ ---- -title: "MCP Authorization" -description: "Define fine-grained access control rules for MCP tools, resources, and prompts based on JWT claims and scopes." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-authorization/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-authorization.md -tags: - - ai-gateway - - mcp - - security -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# MCP Authorization - -## Overview - -The MCP Authorization policy provides fine-grained access control for Model Context Protocol (MCP) server resources. It enables API administrators to define authorization rules based on user claims and scopes extracted from validated JWT tokens, controlling access to specific MCP tools, resources, prompts, and JSON-RPC methods. - -> **Prerequisite**: The MCP Authorization policy requires the [MCP Authentication policy](./mcp-authentication.md) to be applied first. The MCP Authentication policy validates and extracts JWT claims that are used by the authorization policy for access control decisions. - -## Features - -- **Tool-Level Access Control**: Restrict access to specific MCP tools based on user claims and scopes -- **Resource-Level Access Control**: Control access to specific MCP resources based on authorization rules -- **Prompt-Level Access Control**: Manage access to specific MCP prompts -- **JSON-RPC Method-Level Access Control**: Apply authorization rules at the JSON-RPC method level (e.g., `tools/call`, `resources/read`, `prompts/get`) for fine-grained control. Only methods under `tools/`, `resources/`, and `prompts/` are evaluated. -- **Flexible Rule-Based Authorization**: Define multiple authorization rules with attribute matching (exact or wildcard) -- **Claim-Based Validation**: Validate custom claims (e.g., department, role, team) in user tokens -- **Scope-Based Validation**: Require specific OAuth scopes for accessing protected resources -- **Wildcard Matching**: Use wildcard patterns ("*") to create default rules for all resources of a type - -## Configuration - -The MCP Authorization policy uses a single-level configuration model where all parameters are configured per-MCP-API/route in the API definition YAML. - -### User Parameters (API Definition) - -These parameters are configured per MCP Proxy by the API developer: - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `rules` | array | Yes | - | List of authorization rules that define access control policies for MCP resources. | -| `rules[].attribute` | object | Yes | - | The MCP resource attribute to which the authorization rule applies. | -| `rules[].attribute.type` | string | Yes | - | Type of MCP resource: "tool", "resource", "prompt", "method". | -| `rules[].attribute.name` | string | No | - | Name or identifier of the resource. Use "*" for wildcard matching (applies to all resources of the specified type). Examples: "list_files" for tools, "file:///some_resource" for resources, "weather_summary" for prompts, "tools/call" for methods. | -| `rules[].requiredScopes` | array | No | - | List of OAuth scopes required to access this resource. The token must contain all of the specified scopes. | -| `rules[].requiredClaims` | object | No | - | Map of claim names to expected values. All specified claims must be present in the token with matching values. | - -## MCP Proxy Definition Examples - -### Example 1: Basic Tool Access Control - -Restrict access to specific tools based on scopes: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - vhost: mcp1.gw.example.com - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-auth - version: v0.1.1 - params: - issuers: - - PrimaryIDP - - name: mcp-authz - version: v0.1.0 - params: - rules: - - attribute: - type: tool - name: list_files - requiredScopes: - - mcp:tool:read - - attribute: - type: tool - name: create_file - requiredScopes: - - mcp:tool:write - - attribute: - type: tool - name: "*" - requiredScopes: - - mcp:tool:execute - tools: - ... -``` - -### Example 2: Claim-Based Resource Access - -Control resource access based on user claims: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - vhost: mcp1.gw.example.com - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-auth - version: v0.1.1 - params: - issuers: - - PrimaryIDP - - name: mcp-authz - version: v0.1.0 - params: - rules: - - attribute: - type: resource - name: "file:///private/main" - requiredClaims: - department: "engineering" - requiredScopes: - - mcp:resource:read - - attribute: - type: resource - name: "file:///public/main" - requiredScopes: - - mcp:resource:read - tools: - ... -``` - -### Example 3: Role-Based Prompt Access - -Restrict prompt access based on user roles: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - vhost: mcp1.gw.example.com - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-auth - version: v0.1.1 - params: - issuers: - - PrimaryIDP - - name: mcp-authz - version: v0.1.0 - params: - rules: - - attribute: - type: prompt - name: "admin_summary" - requiredClaims: - role: "admin" - requiredScopes: - - mcp:prompt:admin - - attribute: - type: prompt - name: "*" - requiredScopes: - - mcp:prompt:read - tools: - ... -``` - -### Example 4: Multi-Level Authorization - -Combine different resource types with varying access requirements: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - vhost: mcp1.gw.example.com - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-auth - version: v0.1.1 - params: - issuers: - - PrimaryIDP - requiredScopes: - - mcp:access - - name: mcp-authz - version: v0.1.0 - params: - rules: - # Restrictive tool access - - attribute: - type: tool - name: "execute_command" - requiredClaims: - department: "platform" - role: "admin" - requiredScopes: - - mcp:tool:execute:admin - # General tool access - - attribute: - type: tool - name: "*" - requiredScopes: - - mcp:tool:execute - # Resource access for finance department - - attribute: - type: resource - name: "file:///finance/*" - requiredClaims: - department: "finance" - requiredScopes: - - mcp:resource:read:finance - # Public resources - - attribute: - type: resource - name: "*" - requiredScopes: - - mcp:resource:read - tools: - ... -``` - -## Authorization Decision Examples - -**Scenario 1**: User with scope `mcp:tool:read` attempts to call `list_files` tool -- Rule: `attribute.type="tool", attribute.name="list_files", requiredScopes=["mcp:tool:read"]` -- Result: ✅ Access Granted - -**Scenario 2**: User with scope `mcp:tool:execute` (no write scope) attempts to call `create_file` tool -- Rule: `attribute.type="tool", attribute.name="create_file", requiredScopes=["mcp:tool:write"]` -- Result: ❌ Access Denied (insufficient scopes) - -**Scenario 3**: User with claim `department="engineering"` attempts to read resource `file:///private/code` -- Rule: `attribute.type="resource", attribute.name="file:///private/code", requiredClaims={department="engineering"}` -- Result: ✅ Access Granted - -**Scenario 4**: User with claim `department="finance"` (no engineering) attempts to read resource `file:///private/code` -- Rule: `attribute.type="resource", attribute.name="file:///private/code", requiredClaims={department="engineering"}` -- Result: ❌ Access Denied (claim mismatch) - -## Error Handling - -When authorization fails, the policy returns: -- **HTTP Status**: `403 Forbidden` -- **Response Body**: JSON error response with a reason message -- **WWW-Authenticate Header**: Contains information about required scopes for the denied resource - -## Related Policies - -- [MCP Authentication Policy](./mcp-authentication.md) - Validates JWT tokens and is a prerequisite for MCP Authorization -- [JWT Authentication Policy](https://wso2.com/api-platform/policy-hub/policies/jwt-auth) - Base JWT token validation mechanism diff --git a/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-rewrite.md b/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-rewrite.md deleted file mode 100644 index f06278d56..000000000 --- a/en/docs/ai-gateway/next/mcp-proxy/policies/mcp-rewrite.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -title: "MCP Rewrite" -description: "Expose user-facing names for MCP tools, resources, and prompts while mapping them to different backend capability names." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-rewrite/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-rewrite.md -tags: - - ai-gateway - - mcp - - policies -author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 -content_type: "reference" ---- - -# MCP Rewrite - -## Overview - -The MCP Rewrite policy enables API administrators to expose user-facing names for Model Context Protocol (MCP) tools, resources, and prompts while mapping them to different backend capability names. This policy supports three types of MCP capabilities: tools, resources, and prompts. For each capability type, you can define a list of user-facing capabilities with optional mappings to backend names, and optionally specify additional metadata fields to be returned in list responses. - -When a list is provided for a capability type, only the configured capabilities are included in list responses. Requests for unlisted capabilities are rejected with an appropriate error. The policy rewrites request payloads to use backend capability names when configured, and rewrites list responses to return user-facing values. - -## Features - -- **Tool Rewriting**: Define user-facing tool names and map them to backend tool names with custom schemas and descriptions. -- **Resource Rewriting**: Define user-facing resource identifiers and map them to backend resource identifiers with custom descriptions. -- **Prompt Rewriting**: Define user-facing prompt names and map them to backend prompt names with custom metadata. -- **Flexible Metadata**: Include additional fields (beyond `name`, `description`, `target`, etc.) in capability definitions for custom metadata in list responses. -- **Optional Mapping**: Omit the `target` field to expose capabilities as-is without mapping to a different backend name. - -## Configuration - -The MCP Rewrite policy uses a single-level configuration model where all parameters are configured per-MCP-API/route in the API definition YAML. - -### User Parameters (API Definition) - -These parameters are configured per MCP Proxy by the API developer: - -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `tools` | array | No | List of tools to expose and optionally rewrite. Each entry must include `name` and `description`, and should include `inputSchema`. When provided (non-empty), only these tools are included in tools/list responses. | -| `tools[].name` | string | Yes | User-facing tool name exposed to clients (1-256 characters). | -| `tools[].description` | string | Yes | User-facing tool description returned in tools/list. | -| `tools[].inputSchema` | string | Yes | Tool input schema returned in tools/list. | -| `tools[].outputSchema` | string | No | Tool output schema returned in tools/list. | -| `tools[].target` | string | No | Backend tool name to use when forwarding requests. If omitted, the `name` is used. | -| `resources` | array | No | List of resources to expose and optionally rewrite. Each entry must include `name` and `uri`. When provided (non-empty), only these resources are included in resources/list responses. | -| `resources[].name` | string | Yes | User-facing resource identifier exposed to clients (1-1024 characters). | -| `resources[].uri` | string | Yes | User-facing resource URI returned in resources/list (1-2048 characters). | -| `resources[].description` | string | No | User-facing resource description returned in resources/list. | -| `resources[].target` | string | No | Backend resource identifier (URI) to use when forwarding requests. If omitted, the `uri` is used. | -| `prompts` | array | No | List of prompts to expose and optionally rewrite. Each entry must include `name`. When provided (non-empty), only these prompts are included in prompts/list responses. | -| `prompts[].name` | string | Yes | User-facing prompt name exposed to clients (1-256 characters). | -| `prompts[].description` | string | No | User-facing prompt description returned in prompts/list. | -| `prompts[].target` | string | No | Backend prompt name to use when forwarding requests. If omitted, the `name` is used. | - -> **Note**: Additional custom fields can be included in `tools`, `resources`, and `prompts` definitions and will be returned in the corresponding list responses. - -## MCP Proxy Definition Examples - -### Example 1: Basic Tool Rewriting - -Expose tools with different names than the backend: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-rewrite - version: v0.1.0 - params: - tools: - - name: list-files - description: List files in a directory - inputSchema: '{"type": "object", "properties": {"path": {"type": "string"}}}' - target: backend_list_files - - name: read-file - description: Read file contents - inputSchema: '{"type": "object", "properties": {"path": {"type": "string"}}}' - target: backend_read_file - tools: - ... -``` - -### Example 2: Resource Rewriting with URI Mapping - -Expose resources with user-friendly URIs mapped to backend resources: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-rewrite - version: v0.1.0 - params: - resources: - - name: user-docs - uri: file:///user-documentation - description: User documentation files - target: file:///internal/docs/users - - name: api-specs - uri: file:///api-specifications - description: API specification files - target: file:///internal/specs/api - resources: - ... -``` - -### Example 3: Prompt and Tool Rewriting Combined - -Rewrite prompts and tools with metadata: - -```yaml -apiVersion: gateway.api-platform.wso2.com/v1 -kind: Mcp -metadata: - name: mcp-server-api-v1.0 -spec: - displayName: mcp-server-api - version: v1.0 - context: /mcpserver - upstream: - url: https://mcp-backend:8080 - policies: - - name: mcp-rewrite - version: v0.1.0 - params: - tools: - - name: create-document - description: Create a new document - inputSchema: '{"type": "object", "properties": {"title": {"type": "string"}}}' - target: create_doc - prompts: - - name: summarize - description: Summarize content - target: summarize_content - category: text-processing - tools: - ... -``` - -## Use Cases - -1. **Semantic Naming**: Use user-friendly names in the API while keeping backend names internal or legacy. -2. **AI Readiness**: Redefine name, description, input schema, etc. in a way that is friendly to the AI agents. diff --git a/en/docs/ai-gateway/next/analytics/analytics-header-filter.md b/en/docs/ai-gateway/next/monitor-traffic/analytics-header-filter.md similarity index 96% rename from en/docs/ai-gateway/next/analytics/analytics-header-filter.md rename to en/docs/ai-gateway/next/monitor-traffic/analytics-header-filter.md index 91d619a6c..411ab3058 100644 --- a/en/docs/ai-gateway/next/analytics/analytics-header-filter.md +++ b/en/docs/ai-gateway/next/monitor-traffic/analytics-header-filter.md @@ -1,14 +1,14 @@ --- title: "Analytics Header Filter" description: "Control which request and response headers are sent to analytics backends using allow or deny mode in API Platform AI Gateway." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/analytics/analytics-header-filter/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/analytics/analytics-header-filter.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/analytics-header-filter/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/analytics-header-filter.md tags: - ai-gateway - analytics - policies author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 +last_updated: 2026-08-11 content_type: "reference" --- diff --git a/en/docs/ai-gateway/next/observability/logging.md b/en/docs/ai-gateway/next/monitor-traffic/logging.md similarity index 98% rename from en/docs/ai-gateway/next/observability/logging.md rename to en/docs/ai-gateway/next/monitor-traffic/logging.md index 8565c1a76..29cf95a35 100644 --- a/en/docs/ai-gateway/next/observability/logging.md +++ b/en/docs/ai-gateway/next/monitor-traffic/logging.md @@ -1,14 +1,14 @@ --- title: "Gateway Logging" description: "Configure centralized log collection for API Platform AI Gateway using Fluent Bit, OpenSearch, and alternative logging stacks." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/observability/logging/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/observability/logging.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/logging/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/logging.md tags: - ai-gateway - observability - devops author: WSO2 API Platform Documentation Team -last_updated: 2026-06-16 +last_updated: 2026-08-11 content_type: "how-to" --- diff --git a/en/docs/ai-gateway/next/analytics/moesif-analytics.md b/en/docs/ai-gateway/next/monitor-traffic/moesif-analytics.md similarity index 97% rename from en/docs/ai-gateway/next/analytics/moesif-analytics.md rename to en/docs/ai-gateway/next/monitor-traffic/moesif-analytics.md index bff1d93c9..a6f949a3e 100644 --- a/en/docs/ai-gateway/next/analytics/moesif-analytics.md +++ b/en/docs/ai-gateway/next/monitor-traffic/moesif-analytics.md @@ -1,14 +1,14 @@ --- title: "Moesif Analytics" description: "Configure Moesif in API Platform AI Gateway to capture and publish API request and response data." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/analytics/moesif-analytics/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/analytics/moesif-analytics.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/moesif-analytics/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/moesif-analytics.md tags: - ai-gateway - analytics - observability author: WSO2 API Platform Documentation Team -last_updated: 2026-06-16 +last_updated: 2026-08-11 content_type: "how-to" --- diff --git a/en/docs/ai-gateway/next/observability/tracing.md b/en/docs/ai-gateway/next/monitor-traffic/tracing.md similarity index 99% rename from en/docs/ai-gateway/next/observability/tracing.md rename to en/docs/ai-gateway/next/monitor-traffic/tracing.md index 9353ff60a..863ac0dc3 100644 --- a/en/docs/ai-gateway/next/observability/tracing.md +++ b/en/docs/ai-gateway/next/monitor-traffic/tracing.md @@ -1,14 +1,14 @@ --- title: "Gateway Tracing" description: "Configure distributed tracing for API Platform AI Gateway using OpenTelemetry and Jaeger, with support for cloud-native tracing backends." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/observability/tracing/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/observability/tracing.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/tracing/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/tracing.md tags: - ai-gateway - observability - devops author: WSO2 API Platform Documentation Team -last_updated: 2026-06-16 +last_updated: 2026-08-11 content_type: "how-to" --- diff --git a/en/docs/ai-gateway/next/overview.md b/en/docs/ai-gateway/next/overview.md index 7f72dcf7f..eb575d00b 100644 --- a/en/docs/ai-gateway/next/overview.md +++ b/en/docs/ai-gateway/next/overview.md @@ -8,7 +8,7 @@ tags: - llm - mcp author: WSO2 API Platform Documentation Team -last_updated: 2026-08-11 +last_updated: 2026-08-12 content_type: "concept" --- @@ -16,114 +16,21 @@ content_type: "concept" A gateway for managing and securing AI traffic, including Large Language Model (LLM) APIs and Model Context Protocol (MCP) servers. -## Quick Start +## Why use the AI Gateway -- [Quick Start Guide](quick-start-guide.md)—Set up the gateway, verify the controller admin health endpoint, and route traffic to LLM providers like OpenAI -- [MCP Quick Start Guide](mcp-proxy/quick-start-guide.md)—Set up the gateway, verify the controller admin health endpoint, and route traffic to MCP servers +Run the AI Gateway when AI traffic needs the controls you already apply to your APIs. With it, you can: -## Key Concepts +- Apply guardrails that validate, filter, or transform content before it reaches a model or a client. See [Guardrails overview](control-content/overview.md). +- Serve one OpenAI-compatible endpoint that routes requests to multiple LLM providers. See [Route across multiple providers](expose-llms/multi-provider-routing.md). +- Expose MCP servers through a central gateway, and apply authentication and access control to MCP traffic. See [Create an MCP proxy](mcp-proxy/create-an-mcp-proxy.md). +- Collect logs, traces, and analytics for the traffic the gateway handles. See [Logging](monitor-traffic/logging.md). +- Run the gateway on its own, or register it with AI Workspace to govern the gateways across your organization. See [Connect to AI Workspace](run-the-gateway/production-deployment/control-plane-connection.md). -### LLM Provider Template +## Who it is for -An LLM Provider Template defines the characteristics and behaviors specific to an AI service provider, such as OpenAI, Azure OpenAI, or other LLM platforms. It describes how the gateway should interpret and extract usage and operational metadata, including prompt, completion, total, and remaining token information, as well as request and response model metadata. +Two roles share the gateway. A platform administrator configures LLM providers, the credentials they use, and the policies that apply organization-wide. An AI developer creates LLM proxies on top of those providers, and adds the policies a single application needs. -Following templates are shipped out-of-the-box +## Where to go next -- OpenAI -- Azure OpenAI -- Anthropic -- AWS Bedrock -- Azure AI Foundry -- Gemini - -### LLM Provider - -An LLM Provider represents a connection to an AI backend service such as OpenAI, Azure OpenAI, or other LLM APIs. Platform administrators configure LLM Providers to define: - -- The LLM Provider Template -- The upstream LLM service URL -- Authentication credentials (API keys, tokens) -- Access control rules for which endpoints are exposed -- Budget control policies, such as token-based rate limiting -- Organization-wide policies such as guardrails - -Once configured, the LLM Provider allows traffic to flow through the gateway to the AI backend. - -To connect the gateway to AWS Bedrock, see [Configure an AWS Bedrock LLM Provider](llm-proxy/configure-aws-bedrock-provider.md). The guide covers both Bedrock bearer API keys and AWS Signature Version 4 (SigV4) authentication. - -### LLM Proxy - -An LLM Proxy allows developers to create custom API endpoints that consume an LLM Provider, while inheriting administrator-enforced access control, budgeting and organization-wide policies defined at the provider level. Each proxy gets its own URL context (e.g., `/assistant`) and can have its own policies applied. This enables: - -- Multiple AI applications to share a single LLM Provider -- A single OpenAI-compatible endpoint to route requests to multiple LLM providers. See [Multi-Provider Routing for LLM Proxies](./llm-proxy/multi-provider-routing.md). -- Per-application policies such as prompt management and guardrails -- Separation between platform administration and application development - -### MCP Proxy - -An MCP Proxy routes Model Context Protocol traffic to MCP servers. MCP is a protocol that enables AI assistants to interact with external tools and data sources. With MCP Proxies, you can: - -- Expose MCP servers through a centralized gateway -- Apply authentication and access control to MCP traffic -- Manage multiple MCP servers from a single control plane - -### Streaming - -When an upstream service streams its response, the gateway relays it to the client chunk by chunk instead of buffering the whole response. This holds for LLM providers and LLM proxies, and needs no configuration. On MCP proxies, request bodies stream, but response bodies stay buffered. See [Real-time AI streaming](./streaming-responses.md). - -## Default Ports - -| Port | Service | Description | -|------|---------|-------------| -| 8080 | Router | HTTP traffic | -| 8443 | Router | HTTPS traffic | -| 9090 | Gateway-Controller | REST API | -| 9094 | Gateway-Controller Admin | Health and admin endpoints | - -## Architecture - -``` - ┌─────────────────┐ - │ LLM Providers │ - │ (OpenAI, etc.) │ - └────────▲────────┘ - │ -┌──────────┐ ┌──────────────┐ │ -│ AI Apps │───▶│ AI Gateway │────┤ -└──────────┘ └──────────────┘ │ - │ - ┌────────▼────────┐ - │ MCP Servers │ - └─────────────────┘ -``` - -**How it works:** - -1. Administrators verify the Gateway-Controller admin health endpoint and configure LLM Providers and MCP Proxies via the Gateway-Controller API -2. Developers create LLM Proxies to build AI applications on top of available providers -3. The gateway routes traffic, applies policies, and manages authentication - -## AI Guardrails - -AI Guardrails allow you to enforce safety, content, and compliance policies on AI traffic flowing through the AI Gateway. They can be applied at the LLM Provider level (organization-wide), at the LLM Proxy level (per-application), or on MCP Proxies. - -The complete and up-to-date guardrail catalogue — with configuration references and examples — is maintained in the gateway-controllers repository: [gateway-controllers documentation](https://github.com/wso2/gateway-controllers/blob/main/docs/README.md) - -You can extend the AI Gateway with custom guardrail policies by building a custom gateway image using the `ap` CLI. See [Customizing the Gateway by Adding and Removing Policies](../../tools/cli/customizing-gateway-policies.md). - -## Documentation - -The following table lists the AI Gateway documentation sections and what each one covers: - -| Section | Description | -|---------|-------------| -| [LLM](quick-start-guide.md) | LLM provider configuration, guardrails, prompt management, and semantic caching | -| [MCP](mcp-proxy/quick-start-guide.md) | MCP proxy setup and policies | -| [Real-time AI streaming](streaming-responses.md) | Streamed responses across providers and proxies, and how policies and analytics behave | -| [Observability](observability/logging.md) | Logging and tracing configuration | -| [Analytics](analytics/moesif-analytics.md) | Analytics integrations (Moesif) | -| [Policies and Guardrails](https://github.com/wso2/gateway-controllers/blob/main/docs/README.md) | Gateway policies and guardrails for AI traffic control | -| [Gateway Controller Management API](./gateway-controller-management-api/overview.md) | REST API reference for managing LLM providers, LLM proxies, MCP proxies, certificates, and secrets | -| [Production deployment](./deployment/production-deployment/overview.md) | High-availability Kubernetes deployment with Helm, an external database, replicated workloads, and AI workload tuning | -| [AI Workspace](../../ai-workspace/next/overview.md) | The control plane for governing LLM providers, proxies, and policies across every gateway you run | +- To install the gateway and route a first request through it, see [Quick Start Guide](quick-start-guide.md). +- To learn which artifacts a request passes through, see [How it works](how-it-works.md). diff --git a/en/docs/ai-gateway/next/quick-start-guide.md b/en/docs/ai-gateway/next/quick-start-guide.md index acd2f553d..eeddddfdb 100644 --- a/en/docs/ai-gateway/next/quick-start-guide.md +++ b/en/docs/ai-gateway/next/quick-start-guide.md @@ -126,87 +126,291 @@ The commands below use version `1.2.0`. Substitute the API Platform AI Gateway r Stop the conflicting service if you don't need it. If you need to keep it running, change the host-side value of the relevant `ports:` mapping in `docker-compose.yaml`. Then use the remapped host port in the verification and test commands on this page. !!! tip "Customizing configuration" - The setup script (`setup.sh`, or `setup.ps1` on Windows) writes `api-platform.env`, which is loaded into the containers via Docker Compose `env_file`. To change the storage backend, connect to a control plane, or tune other settings, edit that file (or the `config.toml` interpolation tokens directly). See [Gateway Configuration and Environment Interpolation](./setup/configuration.md). - -## Deploy an OpenAI LLM provider configuration - -The API Platform Gateway supports the OpenAI LLM provider. As a platform administrator, replace `` with your OpenAI API key and run the following command to deploy a sample OpenAI LLM provider. - -=== "Linux / macOS" - - ```bash - curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -H "Content-Type: application/yaml" \ - -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ - --data-binary @- <<'EOF' - apiVersion: gateway.api-platform.wso2.com/v1 - kind: LlmProvider - metadata: - name: openai-provider - spec: - displayName: OpenAI Provider - version: v1.0 - template: openai - context: /openai/latest - upstream: - url: https://api.openai.com/v1 - auth: - type: api-key - header: Authorization - value: - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - - path: /models - methods: [GET] - - path: /models/{modelId} - methods: [GET] - EOF - ``` - -=== "Windows (PowerShell)" - - Save the provider definition to `openai-provider.yaml`: - - ```powershell - @' - apiVersion: gateway.api-platform.wso2.com/v1 - kind: LlmProvider - metadata: - name: openai-provider - spec: - displayName: OpenAI Provider - version: v1.0 - template: openai - context: /openai/latest - upstream: - url: https://api.openai.com/v1 - auth: - type: api-key - header: Authorization - value: - accessControl: - mode: deny_all - exceptions: - - path: /chat/completions - methods: [POST] - - path: /models - methods: [GET] - - path: /models/{modelId} - methods: [GET] - '@ | Set-Content -Path openai-provider.yaml -Encoding utf8 - ``` - - Then post it: - - ```powershell - curl.exe -X POST http://localhost:9090/api/management/v1/llm-providers ` - -H "Content-Type: application/yaml" ` - -u "${env:ADMIN_USERNAME}:${env:ADMIN_PASSWORD}" ` - --data-binary "@openai-provider.yaml" - ``` + The setup script (`setup.sh`, or `setup.ps1` on Windows) writes `api-platform.env`, which is loaded into the containers via Docker Compose `env_file`. To change the storage backend, connect to a control plane, or tune other settings, edit that file (or the `config.toml` interpolation tokens directly). See [Gateway Configuration and Environment Interpolation](./run-the-gateway/configuration.md). + +## Deploy an LLM provider configuration + +As a platform administrator, deploy an LLM provider for the vendor whose API key you hold. Select a provider below; each tab carries the complete definition for that provider. + +=== "OpenAI" + + Replace *``* with your OpenAI API key, keeping the `Bearer ` prefix. + + === "Linux / macOS" + + ```bash + curl -X POST http://localhost:9090/api/management/v1/llm-providers \ + -H "Content-Type: application/yaml" \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + --data-binary @- <<'EOF' + apiVersion: gateway.api-platform.wso2.com/v1 + kind: LlmProvider + metadata: + name: openai-provider + spec: + displayName: OpenAI Provider + version: v1.0 + template: openai + context: /openai/latest + upstream: + url: https://api.openai.com/v1 + auth: + type: api-key + header: Authorization + value: Bearer + accessControl: + mode: deny_all + exceptions: + - path: /chat/completions + methods: [POST] + - path: /models + methods: [GET] + - path: /models/{modelId} + methods: [GET] + EOF + ``` + + === "Windows (PowerShell)" + + Save the provider definition to `openai-provider.yaml`: + + ```powershell + @' + apiVersion: gateway.api-platform.wso2.com/v1 + kind: LlmProvider + metadata: + name: openai-provider + spec: + displayName: OpenAI Provider + version: v1.0 + template: openai + context: /openai/latest + upstream: + url: https://api.openai.com/v1 + auth: + type: api-key + header: Authorization + value: Bearer + accessControl: + mode: deny_all + exceptions: + - path: /chat/completions + methods: [POST] + - path: /models + methods: [GET] + - path: /models/{modelId} + methods: [GET] + '@ | Set-Content -Path openai-provider.yaml -Encoding utf8 + ``` + + Then post it: + + ```powershell + curl.exe -X POST http://localhost:9090/api/management/v1/llm-providers ` + -H "Content-Type: application/yaml" ` + -u "${env:ADMIN_USERNAME}:${env:ADMIN_PASSWORD}" ` + --data-binary "@openai-provider.yaml" + ``` + +=== "Anthropic" + + Replace *``* with your Anthropic API key. Anthropic reads the key from an `x-api-key` header, so the value carries no prefix. + + === "Linux / macOS" + + ```bash + curl -X POST http://localhost:9090/api/management/v1/llm-providers \ + -H "Content-Type: application/yaml" \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + --data-binary @- <<'EOF' + apiVersion: gateway.api-platform.wso2.com/v1 + kind: LlmProvider + metadata: + name: anthropic-provider + spec: + displayName: Anthropic Provider + version: v1.0 + template: anthropic + context: /providers/anthropic + upstream: + url: https://api.anthropic.com + auth: + type: api-key + header: x-api-key + value: + accessControl: + mode: deny_all + exceptions: + - path: /v1/messages + methods: [POST] + EOF + ``` + + === "Windows (PowerShell)" + + Save the provider definition to `anthropic-provider.yaml`: + + ```powershell + @' + apiVersion: gateway.api-platform.wso2.com/v1 + kind: LlmProvider + metadata: + name: anthropic-provider + spec: + displayName: Anthropic Provider + version: v1.0 + template: anthropic + context: /providers/anthropic + upstream: + url: https://api.anthropic.com + auth: + type: api-key + header: x-api-key + value: + accessControl: + mode: deny_all + exceptions: + - path: /v1/messages + methods: [POST] + '@ | Set-Content -Path anthropic-provider.yaml -Encoding utf8 + ``` + + Then post it: + + ```powershell + curl.exe -X POST http://localhost:9090/api/management/v1/llm-providers ` + -H "Content-Type: application/yaml" ` + -u "${env:ADMIN_USERNAME}:${env:ADMIN_PASSWORD}" ` + --data-binary "@anthropic-provider.yaml" + ``` + +=== "AWS Bedrock" + + Replace *``* with the AWS Region hosting your model, and *``* with an AWS Bedrock API key. For SigV4 authentication and the IAM permissions Bedrock needs, see [AWS Bedrock](./connect-llm-providers/supported-providers/aws-bedrock.md). + + === "Linux / macOS" + + Store the Bedrock API key as a gateway secret: + + ```bash + export AWS_REGION="" + export AWS_BEARER_TOKEN_BEDROCK="" + + curl --fail-with-body -X POST \ + http://localhost:9090/api/management/v1/secrets \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + -H "Content-Type: application/yaml" \ + --data-binary @- <ResourceStatus +## ResourceStatus @@ -33,7 +33,7 @@ content_type: "reference" Server-managed lifecycle information for a resource -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -43,14 +43,14 @@ Server-managed lifecycle information for a resource |updatedAt|string(date-time)|false|none|Timestamp when the resource was last updated (UTC)| |deployedAt|string(date-time)|false|none|Timestamp when the resource was last deployed (omitted when undeployed)| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |state|deployed| |state|undeployed| -

RestAPIRequest

+## RestAPIRequest @@ -124,7 +124,7 @@ Server-managed lifecycle information for a resource ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -133,14 +133,14 @@ Server-managed lifecycle information for a resource |metadata|[Metadata](#schemametadata)|true|none|none| |spec|[APIConfigData](#schemaapiconfigdata)|true|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |apiVersion|gateway.api-platform.wso2.com/v1| |kind|RestApi| -

RestAPI

+## RestAPI @@ -221,7 +221,7 @@ Server-managed lifecycle information for a resource ``` -### Properties +#### Properties allOf @@ -236,7 +236,7 @@ and |*anonymous*|object|false|none|none| |status|[ResourceStatus](#schemaresourcestatus)|false|read-only|Server-managed lifecycle fields. Populated on responses.| -

Metadata

+## Metadata @@ -258,7 +258,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -268,7 +268,7 @@ and |annotations|object|false|none|Annotations are arbitrary non-identifying metadata. Use domain-prefixed keys.| |**additionalProperties**|string|false|none|none| -

APIConfigData

+## APIConfigData @@ -364,7 +364,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -384,14 +384,14 @@ and |operations|[[Operation](#schemaoperation)]|true|none|List of HTTP operations/routes| |deploymentState|string|false|none|Desired deployment state - 'deployed' (default) or 'undeployed'. When set to 'undeployed', the API is removed from router traffic but configuration, API keys, and policies are preserved for potential redeployment.| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |deploymentState|deployed| |deploymentState|undeployed| -

UpstreamDefinition

+## UpstreamDefinition @@ -417,7 +417,7 @@ and Reusable upstream configuration with optional timeout and load balancing settings -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -428,7 +428,7 @@ Reusable upstream configuration with optional timeout and load balancing setting |url|string(uri)|true|none|Backend URL (host and port only, path comes from basePath)| |weight|integer|false|none|Relative weight for load balancing across multiple upstream targets. Reserved for future multi-target load balancing; not applied yet (only the first target is currently used).| -

UpstreamTimeout

+## UpstreamTimeout @@ -444,13 +444,13 @@ Reusable upstream configuration with optional timeout and load balancing setting Timeout configuration for upstream requests -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |connect|string|false|none|Connection timeout duration (e.g., "5s", "500ms")| -

Resilience

+## Resilience @@ -467,14 +467,14 @@ Timeout configuration for upstream requests Backend/route timeout configuration. Maps to Envoy RouteAction timeouts. Can be set at the API level (applies to all routes) and/or the operation level (applies to that operation's route). When set at both levels, the operation-level value takes precedence. When unset, the gateway's global route timeout defaults apply. -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |timeout|string|false|none|Maximum time for the entire route (request to upstream response). "0s" disables the timeout.| |idleTimeout|string|false|none|Per-route stream idle timeout (overrides the listener stream idle timeout for this route). "0s" disables the timeout.| -

Upstream

+## Upstream @@ -492,7 +492,7 @@ Backend/route timeout configuration. Maps to Envoy RouteAction timeouts. Can be Upstream backend configuration (single target or reference) -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -512,14 +512,14 @@ xor |---|---|---|---|---| |*anonymous*|object|false|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |hostRewrite|auto| |hostRewrite|manual| -

Operation

+## Operation @@ -562,7 +562,7 @@ xor An operation is matched either by the simple top-level method+path form, or by the richer 'match' block (method + path + headers). When 'match' is present it is authoritative and the top-level method/path are ignored. At least one form must be provided. -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -572,7 +572,7 @@ An operation is matched either by the simple top-level method+path form, or by t |policies|[[Policy](#schemapolicy)]|false|none|List of policies applied only to this operation (overrides or adds to API-level policies)| |resilience|[Resilience](#schemaresilience)|false|none|Backend/route timeout configuration. Maps to Envoy RouteAction timeouts. Can be set at the API level (applies to all routes) and/or the operation level (applies to that operation's route). When set at both levels, the operation-level value takes precedence. When unset, the gateway's global route timeout defaults apply.| -

OperationMethod

+## OperationMethod @@ -586,13 +586,13 @@ An operation is matched either by the simple top-level method+path form, or by t HTTP method -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |*anonymous*|string|false|none|HTTP method| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -604,7 +604,7 @@ HTTP method |*anonymous*|HEAD| |*anonymous*|OPTIONS| -

OperationMatch

+## OperationMatch @@ -631,7 +631,7 @@ HTTP method Request matching criteria for an operation. Extensible with query params, cookies, etc. -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -639,7 +639,7 @@ Request matching criteria for an operation. Extensible with query params, cookie |path|[OperationPathMatch](#schemaoperationpathmatch)|true|none|none| |headers|[[OperationHeaderMatch](#schemaoperationheadermatch)]|false|none|Header matchers ANDed with the path match for Envoy route selection| -

OperationPathMatch

+## OperationPathMatch @@ -654,21 +654,21 @@ Request matching criteria for an operation. Extensible with query params, cookie ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |value|string|true|none|Route path with optional {param} placeholders| |type|string|false|none|Path matching semantics for the operation route| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |type|Exact| |type|PathPrefix| -

OperationHeaderMatch

+## OperationHeaderMatch @@ -684,7 +684,7 @@ Request matching criteria for an operation. Extensible with query params, cookie ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -692,14 +692,14 @@ Request matching criteria for an operation. Extensible with query params, cookie |value|string|true|none|Header value to match| |type|string|false|none|Header match type| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |type|Exact| |type|RegularExpression| -

Policy

+## Policy @@ -716,7 +716,7 @@ Request matching criteria for an operation. Extensible with query params, cookie ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -725,7 +725,7 @@ Request matching criteria for an operation. Extensible with query params, cookie |executionCondition|string|false|none|Expression controlling conditional execution of the policy| |params|object|false|none|Arbitrary parameters for the policy (free-form key/value structure)| -

Channel

+## Channel @@ -750,7 +750,7 @@ Request matching criteria for an operation. Extensible with query params, cookie Channel (topic/event stream) definition for async APIs. -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -758,13 +758,13 @@ Channel (topic/event stream) definition for async APIs. |method|string|true|none|Operation method type.| |policies|[[Policy](#schemapolicy)]|false|none|List of policies applied only to this channel (overrides or adds to API-level policies)| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |method|SUB| -

APIKeyCreationRequest

+## APIKeyCreationRequest @@ -778,7 +778,7 @@ Channel (topic/event stream) definition for async APIs. ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -792,7 +792,7 @@ Channel (topic/event stream) definition for async APIs. |externalRefId|string|false|none|External reference ID for the API key.
This field is optional and used for tracing purposes only.
The gateway generates its own internal ID for tracking.| |issuer|string|false|none|Identifies the portal that created this key. If provided, only api keys generated from
the same portal will be accepted. If not provided, there is no portal restriction.| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -803,7 +803,7 @@ Channel (topic/event stream) definition for async APIs. |unit|weeks| |unit|months| -

APIKeyCreationResponse

+## APIKeyCreationResponse @@ -830,7 +830,7 @@ Channel (topic/event stream) definition for async APIs. ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -839,7 +839,7 @@ Channel (topic/event stream) definition for async APIs. |remainingApiKeyQuota|integer|false|none|Remaining API key quota for the user| |apiKey|[APIKey](#schemaapikey)|false|none|Details of an API key| -

APIKey

+## APIKey @@ -863,7 +863,7 @@ Channel (topic/event stream) definition for async APIs. Details of an API key -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -878,7 +878,7 @@ Details of an API key |source|string|true|none|Source of the API key (local or external)| |externalRefId|string|false|none|External reference ID for the API key| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -888,7 +888,7 @@ Details of an API key |source|local| |source|external| -

APIKeyRegenerationRequest

+## APIKeyRegenerationRequest @@ -900,7 +900,7 @@ Details of an API key ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -909,7 +909,7 @@ Details of an API key |duration|integer|true|none|Duration value for expiration| |expiresAt|string(date-time)|false|none|Expiration timestamp| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -920,7 +920,7 @@ Details of an API key |unit|weeks| |unit|months| -

APIKeyUpdateRequest

+## APIKeyUpdateRequest @@ -934,11 +934,11 @@ Details of an API key ``` -### Properties +#### Properties *None* -

APIKeyRevocationResponse

+## APIKeyRevocationResponse @@ -953,14 +953,14 @@ Details of an API key ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |status|string|true|none|none| |message|string|true|none|none| -

SubscriptionPlanCreateRequest

+## SubscriptionPlanCreateRequest @@ -980,7 +980,7 @@ Details of an API key ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -992,7 +992,7 @@ Details of an API key |expiryTime|string(date-time)|false|none|none| |status|string|false|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -1003,7 +1003,7 @@ Details of an API key |status|ACTIVE| |status|INACTIVE| -

SubscriptionPlanUpdateRequest

+## SubscriptionPlanUpdateRequest @@ -1023,7 +1023,7 @@ Details of an API key ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1035,7 +1035,7 @@ Details of an API key |expiryTime|string(date-time)|false|none|none| |status|string|false|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -1046,7 +1046,7 @@ Details of an API key |status|ACTIVE| |status|INACTIVE| -

SubscriptionPlanResponse

+## SubscriptionPlanResponse @@ -1070,7 +1070,7 @@ Details of an API key ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1086,14 +1086,14 @@ Details of an API key |createdAt|string(date-time)|false|none|none| |updatedAt|string(date-time)|false|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |status|ACTIVE| |status|INACTIVE| -

SubscriptionPlanListResponse

+## SubscriptionPlanListResponse @@ -1122,14 +1122,14 @@ Details of an API key ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |subscriptionPlans|[[SubscriptionPlanResponse](#schemasubscriptionplanresponse)]|false|none|none| |count|integer|false|none|none| -

SubscriptionCreateRequest

+## SubscriptionCreateRequest @@ -1149,7 +1149,7 @@ Details of an API key ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1161,7 +1161,7 @@ Details of an API key |billingSubscriptionId|string|false|none|Billing subscription identifier (optional, for analytics tracking).| |status|string|false|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -1169,7 +1169,7 @@ Details of an API key |status|INACTIVE| |status|REVOKED| -

SubscriptionUpdateRequest

+## SubscriptionUpdateRequest @@ -1183,13 +1183,13 @@ Details of an API key ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |status|string|false|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -1197,7 +1197,7 @@ Details of an API key |status|INACTIVE| |status|REVOKED| -

SubscriptionResponse

+## SubscriptionResponse @@ -1221,7 +1221,7 @@ Details of an API key ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1237,7 +1237,7 @@ Details of an API key |createdAt|string(date-time)|false|none|none| |updatedAt|string(date-time)|false|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -1245,7 +1245,7 @@ Details of an API key |status|INACTIVE| |status|REVOKED| -

SubscriptionListResponse

+## SubscriptionListResponse @@ -1274,14 +1274,14 @@ Details of an API key ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |subscriptions|[[SubscriptionResponse](#schemasubscriptionresponse)]|false|none|none| |count|integer|false|none|none| -

MCPProxyConfigurationRequest

+## MCPProxyConfigurationRequest @@ -1311,7 +1311,7 @@ Details of an API key ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1320,14 +1320,14 @@ Details of an API key |metadata|[Metadata](#schemametadata)|true|none|none| |spec|[MCPProxyConfigData](#schemamcpproxyconfigdata)|true|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |apiVersion|gateway.api-platform.wso2.com/v1| |kind|Mcp| -

MCPProxyConfiguration

+## MCPProxyConfiguration @@ -1364,7 +1364,7 @@ Details of an API key ``` -### Properties +#### Properties allOf @@ -1379,7 +1379,7 @@ and |*anonymous*|object|false|none|none| |status|[ResourceStatus](#schemaresourcestatus)|false|read-only|Server-managed lifecycle fields. Populated on responses.| -

MCPProxyConfigData

+## MCPProxyConfigData @@ -1469,7 +1469,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1504,14 +1504,14 @@ continued |deploymentState|string|false|none|Desired deployment state - 'deployed' (default) or 'undeployed'. When set to 'undeployed', the MCP Proxy is removed from router traffic but configuration and policies are preserved for potential redeployment.| |resilience|[Resilience](#schemaresilience)|false|none|API-level backend/route timeout configuration. Applies to the traffic-forwarding routes generated for this MCP proxy (GET/POST/DELETE on the MCP resource path). Supported at the API level only. Because MCP transports are long-lived streams, the route timeout defaults to disabled ("0s") for MCP unless a timeout is set here (unlike REST/LLM, which fall back to the gateway's global route timeout); the idle timeout remains the liveness guard.| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |deploymentState|deployed| |deploymentState|undeployed| -

MCPTool

+## MCPTool @@ -1529,7 +1529,7 @@ continued ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1539,7 +1539,7 @@ continued |inputSchema|string|true|none|JSON Schema defining expected parameters| |outputSchema|string|false|none|Optional JSON Schema defining expected output structure| -

MCPResource

+## MCPResource @@ -1558,7 +1558,7 @@ continued ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1569,7 +1569,7 @@ continued |mimeType|string|false|none|Optional MIME type| |size|integer|false|none|Optional size in bytes| -

MCPPrompt

+## MCPPrompt @@ -1593,7 +1593,7 @@ continued ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1606,7 +1606,7 @@ continued |required|boolean|false|none|Whether the argument is required| |title|string|false|none|Optional human-readable title of the argument| -

ErrorResponse

+## ErrorResponse @@ -1627,7 +1627,7 @@ continued ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1635,7 +1635,7 @@ continued |message|string|true|none|High-level error description| |errors|[[ValidationError](#schemavalidationerror)]|false|none|Detailed validation errors| -

ValidationError

+## ValidationError @@ -1650,14 +1650,14 @@ continued ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |field|string|false|none|Field that failed validation| |message|string|false|none|Human-readable error message| -

LLMProviderTemplateRequest

+## LLMProviderTemplateRequest @@ -1702,7 +1702,7 @@ continued ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1711,14 +1711,14 @@ continued |metadata|[Metadata](#schemametadata)|true|none|none| |spec|[LLMProviderTemplateData](#schemallmprovidertemplatedata)|true|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |apiVersion|gateway.api-platform.wso2.com/v1| |kind|LlmProviderTemplate| -

LLMProviderTemplate

+## LLMProviderTemplate @@ -1768,7 +1768,7 @@ continued ``` -### Properties +#### Properties allOf @@ -1783,7 +1783,7 @@ and |*anonymous*|object|false|none|none| |status|[ResourceStatus](#schemaresourcestatus)|false|read-only|Server-managed lifecycle fields. Populated on responses.| -

LLMProviderTemplateData

+## LLMProviderTemplateData @@ -1855,7 +1855,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1871,7 +1871,7 @@ and |responseModel|[ExtractionIdentifier](#schemaextractionidentifier)|false|none|none| |resourceMappings|[LLMProviderTemplateResourceMappings](#schemallmprovidertemplateresourcemappings)|false|none|none| -

LLMProviderTemplateResourceMappings

+## LLMProviderTemplateResourceMappings @@ -1913,13 +1913,13 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |resources|[[LLMProviderTemplateResourceMapping](#schemallmprovidertemplateresourcemapping)]|false|none|none| -

LLMProviderTemplateResourceMapping

+## LLMProviderTemplateResourceMapping @@ -1957,7 +1957,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -1969,7 +1969,7 @@ and |requestModel|[ExtractionIdentifier](#schemaextractionidentifier)|false|none|none| |responseModel|[ExtractionIdentifier](#schemaextractionidentifier)|false|none|none| -

ExtractionIdentifier

+## ExtractionIdentifier @@ -1984,14 +1984,14 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |location|string|true|none|Where to find the token information| |identifier|string|true|none|JSONPath expression or header name to identify the token value| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -2000,7 +2000,7 @@ and |location|queryParam| |location|pathParam| -

LLMProviderConfigurationRequest

+## LLMProviderConfigurationRequest @@ -2055,7 +2055,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2064,14 +2064,14 @@ and |metadata|[Metadata](#schemametadata)|true|none|none| |spec|[LLMProviderConfigData](#schemallmproviderconfigdata)|true|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |apiVersion|gateway.api-platform.wso2.com/v1| |kind|LlmProvider| -

LLMProviderConfiguration

+## LLMProviderConfiguration @@ -2132,7 +2132,7 @@ and ``` -### Properties +#### Properties allOf @@ -2147,7 +2147,7 @@ and |*anonymous*|object|false|none|none| |status|[ResourceStatus](#schemaresourcestatus)|false|read-only|Server-managed lifecycle fields. Populated on responses.| -

LLMProviderConfigData

+## LLMProviderConfigData @@ -2245,7 +2245,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2280,14 +2280,14 @@ continued |deploymentState|string|false|none|Desired deployment state - 'deployed' (default) or 'undeployed'. When set to 'undeployed', the LLM Provider is removed from router traffic but configuration and policies are preserved for potential redeployment.| |resilience|[Resilience](#schemaresilience)|false|none|API-level backend/route timeout configuration. Applies to all routes generated for this LLM Provider (the routes that forward traffic upstream). Supported at the API level only - LLM routes are synthesized by the gateway, so there is no operation-level override.| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |deploymentState|deployed| |deploymentState|undeployed| -

UpstreamAuth

+## UpstreamAuth @@ -2305,7 +2305,7 @@ continued ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2314,7 +2314,7 @@ continued |header|string|false|none|none| |value|string|false|write-only|Upstream credential. Write-only: accepted on create/update and never returned by the management API on a read, for any role. Supply either a literal value or a secret reference (e.g. a `secret` template expression); either way the field is omitted from management API response bodies. An update that omits it inherits the stored value; set `type: none` to remove auth.| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -2322,7 +2322,7 @@ continued |type|other| |type|none| -

LLMUpstreamAuth

+## LLMUpstreamAuth @@ -2338,7 +2338,7 @@ continued ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2346,7 +2346,7 @@ continued |header|string|false|none|none| |value|string|false|write-only|Upstream credential. Write-only: accepted on create/update and never returned by the management API on a read, for any role. An update that omits it inherits the stored value; set `type: none` to remove auth.| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| @@ -2354,7 +2354,7 @@ continued |type|other| |type|none| -

LLMProxyProvider

+## LLMProxyProvider @@ -2373,14 +2373,14 @@ continued ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |id|string|true|none|Unique id of a deployed llm provider| |auth|[LLMUpstreamAuth](#schemallmupstreamauth)|false|none|none| -

LLMProxyAdditionalProvider

+## LLMProxyAdditionalProvider @@ -2407,7 +2407,7 @@ continued Additional LLM provider attached to this proxy as a selectable upstream. Policies route to it by referring to the `as` name (defaults to `id`). Optional auth config is used by the proxy when calling a protected LlmProvider over the internal loopback route. -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2416,7 +2416,7 @@ Additional LLM provider attached to this proxy as a selectable upstream. Policie |auth|[LLMUpstreamAuth](#schemallmupstreamauth)|false|none|none| |transformer|[LLMProxyTransformer](#schemallmproxytransformer)|false|none|Request/response translator applied when this provider is the selected upstream. The proxy injects the translator as a conditional policy whose execution condition matches this provider, so it runs only when the provider is selected. The provider's `as` name (defaults to `id`) is passed to the translator as its target upstream.| -

LLMProxyTransformer

+## LLMProxyTransformer @@ -2434,7 +2434,7 @@ Additional LLM provider attached to this proxy as a selectable upstream. Policie Request/response translator applied when this provider is the selected upstream. The proxy injects the translator as a conditional policy whose execution condition matches this provider, so it runs only when the provider is selected. The provider's `as` name (defaults to `id`) is passed to the translator as its target upstream. -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2442,7 +2442,7 @@ Request/response translator applied when this provider is the selected upstream. |version|string|true|none|Major-only translator policy version (for example v1). The Gateway Controller resolves it to the installed full version.| |params|object|false|none|Translator-specific parameters (for example model, apiVersion).| -

LLMAccessControl

+## LLMAccessControl @@ -2464,21 +2464,21 @@ Request/response translator applied when this provider is the selected upstream. ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |mode|string|true|none|Access control mode| |exceptions|[[RouteException](#schemarouteexception)]|false|none|Path exceptions to the access control mode| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |mode|allow_all| |mode|deny_all| -

RouteException

+## RouteException @@ -2495,14 +2495,14 @@ Request/response translator applied when this provider is the selected upstream. ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |path|string|true|none|Path pattern| |methods|[string]|true|none|HTTP methods| -

LLMPolicy

+## LLMPolicy @@ -2526,7 +2526,7 @@ Request/response translator applied when this provider is the selected upstream. ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2534,7 +2534,7 @@ Request/response translator applied when this provider is the selected upstream. |version|string|true|none|none| |paths|[[LLMPolicyPath](#schemallmpolicypath)]|true|none|none| -

LLMPolicyPath

+## LLMPolicyPath @@ -2552,7 +2552,7 @@ Request/response translator applied when this provider is the selected upstream. ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2560,7 +2560,7 @@ Request/response translator applied when this provider is the selected upstream. |methods|[string]|true|none|none| |params|object|true|none|JSON Schema describing the parameters accepted by this policy. This itself is a JSON Schema document.| -

OperationPolicy

+## OperationPolicy @@ -2585,7 +2585,7 @@ Request/response translator applied when this provider is the selected upstream. ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2594,7 +2594,7 @@ Request/response translator applied when this provider is the selected upstream. |executionCondition|string|false|none|Expression controlling conditional execution of the policy| |paths|[[OperationPolicyPath](#schemaoperationpolicypath)]|true|none|none| -

OperationPolicyPath

+## OperationPolicyPath @@ -2612,7 +2612,7 @@ Request/response translator applied when this provider is the selected upstream. ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2620,7 +2620,7 @@ Request/response translator applied when this provider is the selected upstream. |methods|[string]|true|none|none| |params|object|true|none|JSON Schema describing the parameters accepted by this policy. This itself is a JSON Schema document.| -

LLMProxyConfigurationRequest

+## LLMProxyConfigurationRequest @@ -2647,7 +2647,7 @@ Request/response translator applied when this provider is the selected upstream. ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2656,14 +2656,14 @@ Request/response translator applied when this provider is the selected upstream. |metadata|[Metadata](#schemametadata)|true|none|none| |spec|[LLMProxyConfigData](#schemallmproxyconfigdata)|true|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |apiVersion|gateway.api-platform.wso2.com/v1| |kind|LlmProxy| -

LLMProxyConfiguration

+## LLMProxyConfiguration @@ -2697,7 +2697,7 @@ Request/response translator applied when this provider is the selected upstream. ``` -### Properties +#### Properties allOf @@ -2712,7 +2712,7 @@ and |*anonymous*|object|false|none|none| |status|[ResourceStatus](#schemaresourcestatus)|false|read-only|Server-managed lifecycle fields. Populated on responses.| -

LLMProxyConfigData

+## LLMProxyConfigData @@ -2797,7 +2797,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2813,14 +2813,14 @@ and |deploymentState|string|false|none|Desired deployment state - 'deployed' (default) or 'undeployed'. When set to 'undeployed', the LLM Proxy is removed from router traffic but configuration and policies are preserved for potential redeployment.| |resilience|[Resilience](#schemaresilience)|false|none|API-level backend/route timeout configuration. Applies to all routes generated for this LLM Proxy (the routes that forward traffic upstream). Supported at the API level only - LLM routes are synthesized by the gateway, so there is no operation-level override.| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |deploymentState|deployed| |deploymentState|undeployed| -

SecretConfigurationRequest

+## SecretConfigurationRequest @@ -2843,7 +2843,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2852,14 +2852,14 @@ and |metadata|[Metadata](#schemametadata)|true|none|none| |spec|[SecretConfigData](#schemasecretconfigdata)|true|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |apiVersion|gateway.api-platform.wso2.com/v1| |kind|Secret| -

SecretConfiguration

+## SecretConfiguration @@ -2892,7 +2892,7 @@ For actual HTTP response shapes, use `SecretConfigurationResponseCreateUpdate`, `SecretConfigurationResponseRetrieved`, or `SecretListItem` (see `SecretResourceServiceStatus` for the id/timestamp-only status on secret APIs). -### Properties +#### Properties allOf @@ -2907,7 +2907,7 @@ and |*anonymous*|object|false|none|none| |status|[ResourceStatus](#schemaresourcestatus)|false|read-only|Server-managed lifecycle fields. Populated on responses.| -

SecretConfigData

+## SecretConfigData @@ -2923,7 +2923,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2931,7 +2931,7 @@ and |description|string|false|none|Description of the secret| |value|string(password)|true|none|Secret value (stored encrypted)| -

SecretConfigListData

+## SecretConfigListData @@ -2946,14 +2946,14 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |displayName|string|true|none|Human-readable secret name (must be URL-friendly - only letters, numbers, spaces, hyphens, underscores, and dots allowed)| |description|string|false|none|Description of the secret, if the server includes it| -

SecretListItem

+## SecretListItem @@ -2979,7 +2979,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -2989,14 +2989,14 @@ and |spec|[SecretConfigListData](#schemasecretconfiglistdata)|true|none|none| |status|[ResourceStatus](#schemaresourcestatus)|false|read-only|Server-managed lifecycle fields. Omitted in list items may vary; the secret value is never included here.| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |apiVersion|gateway.api-platform.wso2.com/v1| |kind|Secret| -

CertificateUploadRequest

+## CertificateUploadRequest @@ -3011,14 +3011,14 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| |name|string|true|none|Unique name for the certificate. Must be unique across all certificates.| |certificate|string|true|none|PEM-encoded X.509 certificate(s). Can contain multiple certificates.| -

CertificateResponse

+## CertificateResponse @@ -3039,7 +3039,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -3052,14 +3052,14 @@ and |message|string|false|none|Success or informational message| |status|string|false|none|none| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |status|success| |status|error| -

CertificateListResponse

+## CertificateListResponse @@ -3087,7 +3087,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -3096,7 +3096,7 @@ and |totalBytes|integer|false|none|Total bytes of all certificate files| |status|string|false|none|none| -

APIKeyListResponse

+## APIKeyListResponse @@ -3124,7 +3124,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -3132,7 +3132,7 @@ and |totalCount|integer|false|none|Total number of API keys| |status|string|false|none|none| -

SecretListResponse

+## SecretListResponse @@ -3164,7 +3164,7 @@ and ``` -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -3172,7 +3172,7 @@ and |count|integer|false|none|Total number of secrets| |secrets|[[SecretListItem](#schemasecretlistitem)]|false|none|List of secrets. For security, the spec.value field is omitted for every item in the list; retrieve a single secret by id to obtain the decrypted value.| -

SecretResourceServiceStatus

+## SecretResourceServiceStatus @@ -3191,7 +3191,7 @@ and Id and optional timestamps. Not the full ResourceStatus model (no `state` or `deployedAt`). -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -3199,7 +3199,7 @@ Id and optional timestamps. Not the full ResourceStatus model (no `state` or |createdAt|string(date-time)|false|none|none| |updatedAt|string(date-time)|false|none|none| -

SecretConfigurationResponseCreateUpdate

+## SecretConfigurationResponseCreateUpdate @@ -3228,7 +3228,7 @@ Id and optional timestamps. Not the full ResourceStatus model (no `state` or POST/PUT /secrets response. `spec.value` is not returned; see SecretConfigurationRequest for create/update request bodies. -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -3238,14 +3238,14 @@ POST/PUT /secrets response. `spec.value` is not returned; see SecretConfiguratio |spec|[SecretConfigListData](#schemasecretconfiglistdata)|true|none|none| |status|[SecretResourceServiceStatus](#schemasecretresourceservicestatus)|true|none|Id and optional timestamps. Not the full ResourceStatus model (no `state` or
`deployedAt`).| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| |apiVersion|gateway.api-platform.wso2.com/v1| |kind|Secret| -

SecretConfigurationResponseRetrieved

+## SecretConfigurationResponseRetrieved @@ -3275,7 +3275,7 @@ POST/PUT /secrets response. `spec.value` is not returned; see SecretConfiguratio GET /secrets/{id} response including decrypted `spec.value`. -### Properties +#### Properties |Name|Type|Required|Restrictions|Description| |---|---|---|---|---| @@ -3285,7 +3285,7 @@ GET /secrets/{id} response including decrypted `spec.value`. |spec|[SecretConfigData](#schemasecretconfigdata)|true|none|none| |status|[SecretResourceServiceStatus](#schemasecretresourceservicestatus)|true|none|Id and optional timestamps. Not the full ResourceStatus model (no `state` or
`deployedAt`).| -#### Enumerated Values +##### Enumerated Values |Property|Value| |---|---| diff --git a/en/docs/ai-gateway/next/gateway-controller-management-api/secrets-management.md b/en/docs/ai-gateway/next/reference/management-api/secrets-management.md similarity index 97% rename from en/docs/ai-gateway/next/gateway-controller-management-api/secrets-management.md rename to en/docs/ai-gateway/next/reference/management-api/secrets-management.md index 0eb703065..5c2d4d9f8 100644 --- a/en/docs/ai-gateway/next/gateway-controller-management-api/secrets-management.md +++ b/en/docs/ai-gateway/next/reference/management-api/secrets-management.md @@ -1,14 +1,14 @@ --- title: "Management API: Secrets Management" description: "REST API reference for creating, listing, retrieving, updating, and deleting secrets in API Platform Gateway." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/gateway-controller-management-api/secrets-management/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/gateway-controller-management-api/secrets-management.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/reference/management-api/secrets-management/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/reference/management-api/secrets-management.md tags: - ai-gateway - management-api - security author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 +last_updated: 2026-08-11 content_type: "reference" --- diff --git a/en/docs/ai-gateway/next/setup/configuration.md b/en/docs/ai-gateway/next/run-the-gateway/configuration.md similarity index 96% rename from en/docs/ai-gateway/next/setup/configuration.md rename to en/docs/ai-gateway/next/run-the-gateway/configuration.md index d74c0d490..de6df61c0 100644 --- a/en/docs/ai-gateway/next/setup/configuration.md +++ b/en/docs/ai-gateway/next/run-the-gateway/configuration.md @@ -1,14 +1,14 @@ --- title: "Gateway Configuration and Environment Interpolation" description: "How the API Platform Gateway loads its config.toml, injects environment values through interpolation tokens, and bootstraps required keys and certificates with the setup script." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/setup/configuration/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/setup/configuration.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/configuration/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/configuration.md tags: - api-gateway - configuration - interpolation author: WSO2 API Platform Documentation Team -last_updated: 2026-07-22 +last_updated: 2026-08-11 content_type: "reference" --- @@ -92,7 +92,7 @@ APIP_GW_CONTROLLER_CONTROLPLANE_HOST=your-control-plane-host:9443 APIP_GW_CONTROLLER_CONTROLPLANE_TOKEN= ``` -For Kubernetes/Helm deployments, the chart renders `config.toml` into a ConfigMap and injects the runtime secrets (control plane token, database password) as interpolation tokens backed by Kubernetes Secrets — see [Security Hardening](../deployment/production-deployment/security-hardening.md) and [Database Configuration](../deployment/production-deployment/database-configuration.md). +For Kubernetes/Helm deployments, the chart renders `config.toml` into a ConfigMap and injects the runtime secrets (control plane token, database password) as interpolation tokens backed by Kubernetes Secrets — see [Security Hardening](./production-deployment/security-hardening.md) and [Database Configuration](./production-deployment/database-configuration.md). ## No development / demo mode diff --git a/en/docs/ai-gateway/next/setup/database-setup.md b/en/docs/ai-gateway/next/run-the-gateway/database-setup.md similarity index 98% rename from en/docs/ai-gateway/next/setup/database-setup.md rename to en/docs/ai-gateway/next/run-the-gateway/database-setup.md index c8266dec3..35edca014 100644 --- a/en/docs/ai-gateway/next/setup/database-setup.md +++ b/en/docs/ai-gateway/next/run-the-gateway/database-setup.md @@ -1,8 +1,8 @@ --- title: "Setting Up the Database" description: "Create the database and apply the Gateway Controller schema for PostgreSQL or SQL Server before starting the API Platform AI Gateway." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/setup/database-setup/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/setup/database-setup.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/database-setup/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/database-setup.md tags: - ai-gateway - configuration @@ -10,7 +10,7 @@ tags: - sqlserver - devops author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 +last_updated: 2026-08-11 content_type: "how-to" --- diff --git a/en/docs/ai-gateway/next/deployment/deployment-modes/immutable-gateway.md b/en/docs/ai-gateway/next/run-the-gateway/immutable-gateway.md similarity index 93% rename from en/docs/ai-gateway/next/deployment/deployment-modes/immutable-gateway.md rename to en/docs/ai-gateway/next/run-the-gateway/immutable-gateway.md index adddeacfd..bc7e8d3de 100644 --- a/en/docs/ai-gateway/next/deployment/deployment-modes/immutable-gateway.md +++ b/en/docs/ai-gateway/next/run-the-gateway/immutable-gateway.md @@ -1,14 +1,14 @@ --- title: "Immutable Gateway" description: "Run API Platform AI Gateway in immutable mode, loading LLM and MCP configurations from files at startup for GitOps workflows." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/immutable-gateway/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/immutable-gateway.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/immutable-gateway/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/immutable-gateway.md tags: - ai-gateway - deployment - gitops author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 +last_updated: 2026-08-11 content_type: "how-to" --- @@ -62,7 +62,7 @@ By default, the gateway controller loads artifacts from `/etc/api-platform-gatew ``` {% endraw %} - Then set `APIP_GW_IMMUTABLE_GATEWAY_ENABLED=true`. See [Gateway Configuration and Environment Interpolation](../../setup/configuration.md). + Then set `APIP_GW_IMMUTABLE_GATEWAY_ENABLED=true`. See [Gateway Configuration and Environment Interpolation](./configuration.md). ## Artifact format @@ -79,7 +79,7 @@ Artifact files support Go template expressions for injecting dynamic values. Tem | `{{ env "KEY" | redact }}` | Yes | Sensitive env vars (tokens, API keys) | {% endraw %} -Use `| redact` for sensitive values to hide them from config dumps. A `| default "value"` pipe is available for fallback values. See [Gateway Artifact Templating](../../../../api-gateway/1.1.0/setup/artifact-templating.md) for the full function reference. +Use `| redact` for sensitive values to hide them from config dumps. A `| default "value"` pipe is available for fallback values. See [Gateway Artifact Templating](../../../api-gateway/next/setup/artifact-templating.md) for the full function reference. ### Sample: Reading List API diff --git a/en/docs/ai-gateway/next/deployment/deployment-modes/kubernetes/gateway-operator.md b/en/docs/ai-gateway/next/run-the-gateway/kubernetes/gateway-operator.md similarity index 98% rename from en/docs/ai-gateway/next/deployment/deployment-modes/kubernetes/gateway-operator.md rename to en/docs/ai-gateway/next/run-the-gateway/kubernetes/gateway-operator.md index 26adee308..69c087215 100644 --- a/en/docs/ai-gateway/next/deployment/deployment-modes/kubernetes/gateway-operator.md +++ b/en/docs/ai-gateway/next/run-the-gateway/kubernetes/gateway-operator.md @@ -1,14 +1,14 @@ --- title: "Kubernetes Operator for API Platform Gateway" description: "Deploy API Platform AI Gateway on Kubernetes using the Gateway Operator with platform CRDs or the Kubernetes Gateway API." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/kubernetes/gateway-operator/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/kubernetes/gateway-operator.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/kubernetes/gateway-operator/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/kubernetes/gateway-operator.md tags: - ai-gateway - kubernetes - deployment author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 +last_updated: 2026-08-11 content_type: "how-to" --- diff --git a/en/docs/ai-gateway/next/deployment/deployment-modes/kubernetes/kubernetes-standalone.md b/en/docs/ai-gateway/next/run-the-gateway/kubernetes/kubernetes-standalone.md similarity index 96% rename from en/docs/ai-gateway/next/deployment/deployment-modes/kubernetes/kubernetes-standalone.md rename to en/docs/ai-gateway/next/run-the-gateway/kubernetes/kubernetes-standalone.md index 6b517e668..542145bea 100644 --- a/en/docs/ai-gateway/next/deployment/deployment-modes/kubernetes/kubernetes-standalone.md +++ b/en/docs/ai-gateway/next/run-the-gateway/kubernetes/kubernetes-standalone.md @@ -1,14 +1,14 @@ --- title: "Kubernetes Standalone Mode" description: "Install and manage API Platform AI Gateway on Kubernetes using the standalone Helm chart without the Gateway Operator." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/kubernetes/kubernetes-standalone/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/kubernetes/kubernetes-standalone.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/kubernetes/kubernetes-standalone/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/kubernetes/kubernetes-standalone.md tags: - ai-gateway - kubernetes - deployment author: WSO2 API Platform Documentation Team -last_updated: 2026-08-07 +last_updated: 2026-08-11 content_type: "how-to" --- diff --git a/en/docs/ai-gateway/next/deployment/deployment-modes/kubernetes/overview.md b/en/docs/ai-gateway/next/run-the-gateway/kubernetes/overview.md similarity index 94% rename from en/docs/ai-gateway/next/deployment/deployment-modes/kubernetes/overview.md rename to en/docs/ai-gateway/next/run-the-gateway/kubernetes/overview.md index f72471cc6..f89936a94 100644 --- a/en/docs/ai-gateway/next/deployment/deployment-modes/kubernetes/overview.md +++ b/en/docs/ai-gateway/next/run-the-gateway/kubernetes/overview.md @@ -1,14 +1,14 @@ --- title: "AI Gateway Kubernetes Deployment Modes" description: "Choose between standalone and operator-managed Kubernetes deployment modes for API Platform AI Gateway." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/kubernetes/overview/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/kubernetes/overview.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/kubernetes/overview/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/kubernetes/overview.md tags: - ai-gateway - kubernetes - deployment author: WSO2 API Platform Documentation Team -last_updated: 2026-06-19 +last_updated: 2026-08-11 content_type: "overview" --- diff --git a/en/docs/ai-gateway/next/deployment/production-deployment/ai-workload-tuning.md b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/ai-workload-tuning.md similarity index 85% rename from en/docs/ai-gateway/next/deployment/production-deployment/ai-workload-tuning.md rename to en/docs/ai-gateway/next/run-the-gateway/production-deployment/ai-workload-tuning.md index 6929d27f7..b9f5fbfa6 100644 --- a/en/docs/ai-gateway/next/deployment/production-deployment/ai-workload-tuning.md +++ b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/ai-workload-tuning.md @@ -1,8 +1,8 @@ --- title: "Tune the Gateway for AI Traffic" description: "Tune API Platform AI Gateway for LLM and MCP traffic: streaming timeouts, body buffers, guardrail limits, cost pricing data, and semantic cache backing." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/ai-workload-tuning/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/ai-workload-tuning.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/ai-workload-tuning/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/ai-workload-tuning.md tags: - ai-gateway - production @@ -10,7 +10,7 @@ tags: - streaming - guardrails author: WSO2 API Platform Documentation Team -last_updated: 2026-08-09 +last_updated: 2026-08-11 content_type: "how-to" --- @@ -40,7 +40,7 @@ gateway: The two idle timeouts are what keep a streaming response alive. `route_idle_timeout_ms` bounds the gap between chunks arriving from the provider, and `stream_idle_timeout` bounds the same gap on the client side. Set both longer than the longest pause you expect between tokens, not longer than the response as a whole. -Individual proxies can override the route timeouts through their `resilience` block, which is useful when one model is much slower than the rest. For the full precedence order between the `resilience` block and these defaults, see [Timeouts](../../resiliency/timeouts.md). +Individual proxies can override the route timeouts through their `resilience` block, which is useful when one model is much slower than the rest. For the full precedence order between the `resilience` block and these defaults, see [Timeouts](../../control-cost-and-traffic/timeouts.md). !!! note Raise the matching timeouts on whatever sits in front of the gateway as well. An ingress controller or load balancer with a 60-second read timeout cuts the response off regardless of what the gateway allows. @@ -81,7 +81,7 @@ gateway: timeout: 30s ``` -- `python_executor.timeout` bounds a single policy execution. Guardrails that call an external service spend most of that budget on the network call. [Azure Content Safety](../../llm-proxy/guardrails/azure-content-safety.md), [AWS Bedrock guardrails](../../llm-proxy/guardrails/aws-bedrock-guardrail.md), and a [semantic prompt guardrail](../../llm-proxy/guardrails/semantic-prompt-guard.md) that generates embeddings all behave this way. Raise the timeout when the guardrail service is slow or distant. Keep it below `route_timeout_ms`, so the route timeout stays the outer bound. +- `python_executor.timeout` bounds a single policy execution. Guardrails that call an external service spend most of that budget on the network call. [Azure Content Safety](https://wso2.com/api-platform/policy-hub/policies/azure-content-safety-content-moderation), [AWS Bedrock guardrails](https://wso2.com/api-platform/policy-hub/policies/aws-bedrock-guardrail), and a [semantic prompt guardrail](https://wso2.com/api-platform/policy-hub/policies/semantic-prompt-guard) that generates embeddings all behave this way. Raise the timeout when the guardrail service is slow or distant. Keep it below `route_timeout_ms`, so the route timeout stays the outer bound. - `route_cache_action: RETAIN` keeps the route cache warm across requests. Leave it at `RETAIN`. Guardrails that reach an external service add that service's latency and its failure modes to every request. Deploy the guardrail service in the same region as the gateway, and check what your chosen guardrail does when the service is unreachable before you rely on it in production. @@ -129,7 +129,7 @@ Cost figures are only as accurate as this file. Treat it as data to review on a ## Provision infrastructure for semantic caching -[Semantic caching](../../llm-proxy/semantic-caching.md) depends on two external services that the gateway chart doesn't deploy, so provision both before you enable the policy: +[Semantic caching](https://wso2.com/api-platform/policy-hub/policies/semantic-cache) depends on two external services that the gateway chart doesn't deploy, so provision both before you enable the policy: - **A vector database** — Redis or Milvus — that stores the cached responses and their embeddings. Size it for your retention window, place it in the same region as the gateway, and secure it: it holds prompt and completion content in full. - **An embedding provider** — OpenAI, Mistral, or Azure OpenAI — called on every request that reaches the policy. Its latency is added to every cache miss, and its API key is stored as a gateway secret, encrypted with the key from [Security hardening](./security-hardening.md). @@ -154,7 +154,7 @@ gateway: ``` !!! warning "Prompts and completions are sensitive" - Debug-level logging can record request and response bodies, which on an AI Gateway means user prompts and model completions. Before you enable debug-level logging in production, confirm that your log retention and access controls suit that content. If you forward traffic data to an external analytics service, control what leaves the gateway with the [analytics header filter](../../analytics/analytics-header-filter.md). For where the logs go and how to read them, see [Gateway logging](../../observability/logging.md). + Debug-level logging can record request and response bodies, which on an AI Gateway means user prompts and model completions. Before you enable debug-level logging in production, confirm that your log retention and access controls suit that content. If you forward traffic data to an external analytics service, control what leaves the gateway with the [analytics header filter](../../monitor-traffic/analytics-header-filter.md). For where the logs go and how to read them, see [Gateway logging](../../monitor-traffic/logging.md). --- diff --git a/en/docs/ai-gateway/next/deployment/production-deployment/control-plane-connection.md b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/control-plane-connection.md similarity index 96% rename from en/docs/ai-gateway/next/deployment/production-deployment/control-plane-connection.md rename to en/docs/ai-gateway/next/run-the-gateway/production-deployment/control-plane-connection.md index 7e9304170..d91700247 100644 --- a/en/docs/ai-gateway/next/deployment/production-deployment/control-plane-connection.md +++ b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/control-plane-connection.md @@ -1,15 +1,15 @@ --- title: "Connect to AI Workspace" description: "Register a production AI Gateway with AI Workspace: the registration token as a Kubernetes Secret, the control plane address, TLS trust, and sync behavior." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/control-plane-connection/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/control-plane-connection.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/control-plane-connection/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/control-plane-connection.md tags: - ai-gateway - production - ai-workspace - control-plane author: WSO2 API Platform Documentation Team -last_updated: 2026-08-09 +last_updated: 2026-08-11 content_type: "how-to" --- diff --git a/en/docs/ai-gateway/next/deployment/production-deployment/database-configuration.md b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/database-configuration.md similarity index 94% rename from en/docs/ai-gateway/next/deployment/production-deployment/database-configuration.md rename to en/docs/ai-gateway/next/run-the-gateway/production-deployment/database-configuration.md index cfb4c2ba6..ee504dc32 100644 --- a/en/docs/ai-gateway/next/deployment/production-deployment/database-configuration.md +++ b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/database-configuration.md @@ -1,8 +1,8 @@ --- title: "Database Configuration" description: "Point AI Gateway controller replicas at a shared PostgreSQL or SQL Server database, inject the password from a Secret, and tune the connection pool." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/database-configuration/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/database-configuration.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/database-configuration/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/database-configuration.md tags: - ai-gateway - production @@ -10,7 +10,7 @@ tags: - sqlserver - high-availability author: WSO2 API Platform Documentation Team -last_updated: 2026-08-09 +last_updated: 2026-08-11 content_type: "how-to" --- @@ -20,7 +20,7 @@ An external database is what makes controller high availability possible. It rep ## Before you begin -Create the database, the gateway account, and the schema first. That procedure, including where the schema scripts live and how to restrict runtime privileges, is covered in [Setting up the database](../../setup/database-setup.md). The Helm chart has no bootstrap job, so run the schema scripts from a CI job, a bastion host, or a temporary pod with network access to the database. +Create the database, the gateway account, and the schema first. That procedure, including where the schema scripts live and how to restrict runtime privileges, is covered in [Setting up the database](../database-setup.md). The Helm chart has no bootstrap job, so run the schema scripts from a CI job, a bastion host, or a temporary pod with network access to the database. Come back here once the schema exists. This page covers only the chart configuration. diff --git a/en/docs/ai-gateway/next/deployment/production-deployment/deploy-and-verify.md b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/deploy-and-verify.md similarity index 96% rename from en/docs/ai-gateway/next/deployment/production-deployment/deploy-and-verify.md rename to en/docs/ai-gateway/next/run-the-gateway/production-deployment/deploy-and-verify.md index 5dfb378b2..42b9bae37 100644 --- a/en/docs/ai-gateway/next/deployment/production-deployment/deploy-and-verify.md +++ b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/deploy-and-verify.md @@ -1,15 +1,15 @@ --- title: "Deploy and Verify" description: "Install the API Platform AI Gateway Helm chart, confirm the controller and runtime are healthy, route a live LLM request, and run upgrades and rollbacks." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/deploy-and-verify/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/deploy-and-verify.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/deploy-and-verify/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/deploy-and-verify.md tags: - ai-gateway - production - helm - kubernetes author: WSO2 API Platform Documentation Team -last_updated: 2026-08-09 +last_updated: 2026-08-11 content_type: "how-to" --- @@ -198,7 +198,7 @@ curl -N -X POST "https://ai-gateway.example.com/assistant/chat/completions" \ Chunks should arrive progressively. If the whole response arrives at once, one layer in the path is holding it until the response completes. Check these in order: - The ingress controller or reverse proxy in front of the gateway. Response buffering there hides the stream from the client. -- A gateway policy that needs the complete body before it can run. See [Real-time AI streaming](../../streaming-responses.md). +- A gateway policy that needs the complete body before it can run. See [Real-time AI streaming](../../expose-llms/streaming-responses.md). - The provider or the model. Not every model streams every request. If the response cuts off partway, revisit the timeouts in [Tune the gateway for AI traffic](./ai-workload-tuning.md#raise-the-timeouts-for-long-completions). diff --git a/en/docs/ai-gateway/next/deployment/production-deployment/overview.md b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/overview.md similarity index 96% rename from en/docs/ai-gateway/next/deployment/production-deployment/overview.md rename to en/docs/ai-gateway/next/run-the-gateway/production-deployment/overview.md index 38ce9519d..e2ec611f6 100644 --- a/en/docs/ai-gateway/next/deployment/production-deployment/overview.md +++ b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/overview.md @@ -1,8 +1,8 @@ --- title: "Production Deployment Overview" description: "Plan a high-availability production deployment of API Platform AI Gateway on Kubernetes with Helm, an external database, and replicated workloads." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/overview/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/overview.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/overview/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/overview.md tags: - ai-gateway - production @@ -10,7 +10,7 @@ tags: - kubernetes - high-availability author: WSO2 API Platform Documentation Team -last_updated: 2026-08-09 +last_updated: 2026-08-11 content_type: "concept" --- @@ -23,7 +23,7 @@ This section is for platform engineers and site reliability engineers who run AP - Replicated workloads. - The AI-specific tuning that large language model (LLM) and Model Context Protocol (MCP) traffic needs. -For a single-host evaluation, follow the [quick start guide](../../quick-start-guide.md) instead. For the other ways to run the gateway, see [Immutable Gateway](../deployment-modes/immutable-gateway.md) and [Kubernetes deployment modes](../deployment-modes/kubernetes/overview.md). +For a single-host evaluation, follow the [quick start guide](../../quick-start-guide.md) instead. For the other ways to run the gateway, see [Immutable Gateway](../immutable-gateway.md) and [Kubernetes deployment modes](../kubernetes/overview.md). !!! info "Version-pinned instructions" Every command and configuration key on these pages is taken from the `ai-gateway/v1.2.0` release tag and Helm chart `1.2.0`. Chart fields change between releases, so follow the pages for the AI Gateway version you deploy. diff --git a/en/docs/ai-gateway/next/deployment/production-deployment/resources-and-scaling.md b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/resources-and-scaling.md similarity index 94% rename from en/docs/ai-gateway/next/deployment/production-deployment/resources-and-scaling.md rename to en/docs/ai-gateway/next/run-the-gateway/production-deployment/resources-and-scaling.md index 87bf9b5a1..cfdd5863d 100644 --- a/en/docs/ai-gateway/next/deployment/production-deployment/resources-and-scaling.md +++ b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/resources-and-scaling.md @@ -1,15 +1,15 @@ --- title: "Resources and Scaling" description: "Size CPU and memory for the AI Gateway controller and runtime, spread replicas with anti-affinity, and configure autoscaling and pod disruption budgets." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/resources-and-scaling/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/resources-and-scaling.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/resources-and-scaling/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/resources-and-scaling.md tags: - ai-gateway - production - kubernetes - scaling author: WSO2 API Platform Documentation Team -last_updated: 2026-08-09 +last_updated: 2026-08-11 content_type: "how-to" --- @@ -19,7 +19,7 @@ content_type: "how-to" Set requests and limits on both components in production. Without them, a traffic spike on the gateway can starve other workloads on the node. -The allocations below match the ones used in the [AI Gateway performance tests](../../performance/overview.md), so the published throughput and latency figures describe what these settings deliver. +The allocations below match the ones used in the [AI Gateway performance tests](../sizing-and-performance/overview.md), so the published throughput and latency figures describe what these settings deliver. **Gateway Controller:** diff --git a/en/docs/ai-gateway/next/deployment/production-deployment/security-hardening.md b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/security-hardening.md similarity index 97% rename from en/docs/ai-gateway/next/deployment/production-deployment/security-hardening.md rename to en/docs/ai-gateway/next/run-the-gateway/production-deployment/security-hardening.md index ced5bae94..ee1e80a72 100644 --- a/en/docs/ai-gateway/next/deployment/production-deployment/security-hardening.md +++ b/en/docs/ai-gateway/next/run-the-gateway/production-deployment/security-hardening.md @@ -1,8 +1,8 @@ --- title: "Security Hardening" description: "Harden API Platform AI Gateway before production: AES-256 at-rest encryption keys, TLS for the listener and upstreams, and management API authentication." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/security-hardening/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/security-hardening.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/security-hardening/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/security-hardening.md tags: - ai-gateway - production @@ -10,7 +10,7 @@ tags: - encryption - tls author: WSO2 API Platform Documentation Team -last_updated: 2026-08-09 +last_updated: 2026-08-11 content_type: "how-to" --- diff --git a/en/docs/ai-gateway/next/performance/ai-gateway-runtime-with-four-cpus.md b/en/docs/ai-gateway/next/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-four-cpus.md similarity index 70% rename from en/docs/ai-gateway/next/performance/ai-gateway-runtime-with-four-cpus.md rename to en/docs/ai-gateway/next/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-four-cpus.md index 724677e52..7cab6a9b4 100644 --- a/en/docs/ai-gateway/next/performance/ai-gateway-runtime-with-four-cpus.md +++ b/en/docs/ai-gateway/next/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-four-cpus.md @@ -1,15 +1,15 @@ --- title: "AI Gateway Runtime with Four CPUs - Performance Results" description: "View API Platform AI Gateway performance benchmark results with a four-CPU gateway runtime, including throughput, average response time, and percentile response times for Auth No Guardrails, PII Masking, and Advanced Guardrails scenarios." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/performance/ai-gateway-runtime-with-four-cpus/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/performance/ai-gateway-runtime-with-four-cpus.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-four-cpus/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-four-cpus.md tags: - ai-gateway - performance - benchmarks - gateway-runtime author: WSO2 API Platform Documentation Team -last_updated: 2026-07-15 +last_updated: 2026-08-11 content_type: "reference" --- @@ -26,7 +26,7 @@ The table below displays the resource allocations for the AI Gateway components The graph below shows how AI Gateway throughput changes as concurrent users increase for AI API Auth No Guardrails, AI API PII Masking, and AI API Advanced Guardrails. -![Line chart of AI Gateway throughput in requests per second versus concurrent users at 100, 500, 800, and 1000 users, comparing AI API Auth No Guardrails, AI API PII Masking, and AI API Advanced Guardrails on a four-CPU gateway runtime](../../../assets/img/ai-gateway/performance/four-core-Throughput-vs-Concurrent-Users.svg){ width="900" } +![Line chart of AI Gateway throughput in requests per second versus concurrent users on a four-CPU gateway runtime](../../../../assets/img/ai-gateway/performance/four-core-Throughput-vs-Concurrent-Users.svg){ width="900" } **Key observations:** @@ -38,7 +38,7 @@ The graph below shows how AI Gateway throughput changes as concurrent users incr The graph below shows how average response time changes for the same AI API scenarios as concurrent users increase. The backend delay was configured to 10 ms for these tests. -![Line chart of average response time in milliseconds versus concurrent users at 100, 500, 800, and 1000 users, comparing AI API Auth No Guardrails, AI API PII Masking, and AI API Advanced Guardrails on a four-CPU gateway runtime](../../../assets/img/ai-gateway/performance/four-core-Average-Response-Time-vs-Concurrent-Users.svg){ width="900" } +![Line chart of average response time in milliseconds versus concurrent users on a four-CPU gateway runtime](../../../../assets/img/ai-gateway/performance/four-core-Average-Response-Time-vs-Concurrent-Users.svg){ width="900" } **Key observations:** @@ -49,7 +49,7 @@ The graph below shows how average response time changes for the same AI API scen The graphs below show the 90th, 95th, and 99th percentile response times at 10 ms backend delay. Percentile values indicate the response time below which that percentage of requests completed, for example, the 99th percentile is the response time exceeded by only 1% of requests. -![Line chart of 90th, 95th, and 99th percentile response times in milliseconds versus concurrent users for AI API Auth No Guardrails on a four-CPU gateway runtime](../../../assets/img/ai-gateway/performance/4-core-Percentile-Response-Times-aiapi.svg){ width="900" } +![Line chart of 90th, 95th, and 99th percentile response times for AI API Auth No Guardrails on a four-CPU gateway runtime](../../../../assets/img/ai-gateway/performance/4-core-Percentile-Response-Times-aiapi.svg){ width="900" } **Key observations:** @@ -57,7 +57,7 @@ The graphs below show the 90th, 95th, and 99th percentile response times at 10 m - The four-CPU configuration yields lower percentile values at high concurrency than the two-CPU configuration. - Percentile growth mainly reflects load on the gateway and the fixed backend delay, without content-level guardrail processing. -![Line chart of 90th, 95th, and 99th percentile response times in milliseconds versus concurrent users for AI API PII Masking on a four-CPU gateway runtime](../../../assets/img/ai-gateway/performance/4-core-Percentile-Response-Times-pii-masking.svg){ width="900" } +![Line chart of 90th, 95th, and 99th percentile response times for AI API PII Masking on a four-CPU gateway runtime](../../../../assets/img/ai-gateway/performance/4-core-Percentile-Response-Times-pii-masking.svg){ width="900" } **Key observations:** @@ -65,7 +65,7 @@ The graphs below show the 90th, 95th, and 99th percentile response times at 10 m - Compared with Auth No Guardrails, percentile values are higher at each concurrency level due to request and response masking. - Compared with the two-CPU PII Masking results, the four-CPU configuration keeps percentiles lower under similar concurrency. -![Line chart of 90th, 95th, and 99th percentile response times in milliseconds versus concurrent users for AI API Advanced Guardrails on a four-CPU gateway runtime](../../../assets/img/ai-gateway/performance/4-core-Percentile-Response-Times-advanced.svg){ width="900" } +![Line chart of 90th, 95th, and 99th percentile response times for AI API Advanced Guardrails on a four-CPU gateway runtime](../../../../assets/img/ai-gateway/performance/4-core-Percentile-Response-Times-advanced.svg){ width="900" } **Key observations:** diff --git a/en/docs/ai-gateway/next/performance/ai-gateway-runtime-with-two-cpus.md b/en/docs/ai-gateway/next/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-two-cpus.md similarity index 68% rename from en/docs/ai-gateway/next/performance/ai-gateway-runtime-with-two-cpus.md rename to en/docs/ai-gateway/next/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-two-cpus.md index 34da46d2b..d10556394 100644 --- a/en/docs/ai-gateway/next/performance/ai-gateway-runtime-with-two-cpus.md +++ b/en/docs/ai-gateway/next/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-two-cpus.md @@ -1,15 +1,15 @@ --- title: "AI Gateway Runtime with Two CPUs - Performance Results" description: "View API Platform AI Gateway performance benchmark results with a two-CPU gateway runtime, including throughput, average response time, and percentile response times for Auth No Guardrails, PII Masking, and Advanced Guardrails scenarios." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/performance/ai-gateway-runtime-with-two-cpus/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/performance/ai-gateway-runtime-with-two-cpus.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-two-cpus/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-two-cpus.md tags: - ai-gateway - performance - benchmarks - gateway-runtime author: WSO2 API Platform Documentation Team -last_updated: 2026-07-15 +last_updated: 2026-08-11 content_type: "reference" --- @@ -26,7 +26,7 @@ The table below displays the resource allocations for the AI Gateway components The graph below shows how AI Gateway throughput changes as concurrent users increase for AI API Auth No Guardrails, AI API PII Masking, and AI API Advanced Guardrails. -![Line chart of AI Gateway throughput in requests per second versus concurrent users at 100, 500, and 1000 users, comparing AI API Auth No Guardrails, AI API PII Masking, and AI API Advanced Guardrails on a two-CPU gateway runtime](../../../assets/img/ai-gateway/performance/two-core-throughput-vs-concurrent-users.svg){ width="900" } +![Line chart of AI Gateway throughput in requests per second versus concurrent users on a two-CPU gateway runtime](../../../../assets/img/ai-gateway/performance/two-core-throughput-vs-concurrent-users.svg){ width="900" } **Key observations:** @@ -37,7 +37,7 @@ The graph below shows how AI Gateway throughput changes as concurrent users incr The graph below shows how average response time changes for the same AI API scenarios as concurrent users increase. The backend delay was configured to 10 ms for these tests. -![Line chart of average response time in milliseconds versus concurrent users at 100, 500, and 1000 users, comparing AI API Auth No Guardrails, AI API PII Masking, and AI API Advanced Guardrails on a two-CPU gateway runtime](../../../assets/img/ai-gateway/performance/two-core-average-response-time-vs-concurrent-users.svg){ width="900" } +![Line chart of average response time in milliseconds versus concurrent users on a two-CPU gateway runtime](../../../../assets/img/ai-gateway/performance/two-core-average-response-time-vs-concurrent-users.svg){ width="900" } **Key observations:** @@ -48,7 +48,7 @@ The graph below shows how average response time changes for the same AI API scen The graphs below show the 90th, 95th, and 99th percentile response times at 10 ms backend delay. Percentile values indicate the response time below which that percentage of requests completed, for example, the 99th percentile is the response time exceeded by only 1% of requests. -![Line chart of 90th, 95th, and 99th percentile response times in milliseconds versus concurrent users for AI API Auth No Guardrails on a two-CPU gateway runtime](../../../assets/img/ai-gateway/performance/2-core-Percentile-Response-Times-ai-api.svg){ width="900" } +![Line chart of 90th, 95th, and 99th percentile response times for AI API Auth No Guardrails on a two-CPU gateway runtime](../../../../assets/img/ai-gateway/performance/2-core-Percentile-Response-Times-ai-api.svg){ width="900" } **Key observations:** @@ -56,14 +56,14 @@ The graphs below show the 90th, 95th, and 99th percentile response times at 10 m - Higher concurrency widens the spread between lower and upper percentiles. - Because this scenario uses API key authentication without content guardrails, percentile growth mainly reflects gateway load and the backend delay. -![Line chart of 90th, 95th, and 99th percentile response times in milliseconds versus concurrent users for AI API PII Masking on a two-CPU gateway runtime](../../../assets/img/ai-gateway/performance/2-core-Percentile-Response-AI-api-pii-masking.svg){ width="900" } +![Line chart of 90th, 95th, and 99th percentile response times for AI API PII Masking on a two-CPU gateway runtime](../../../../assets/img/ai-gateway/performance/2-core-Percentile-Response-AI-api-pii-masking.svg){ width="900" } **Key observations:** - Percentile trends follow the same upward pattern as concurrent users increase across the test range. - Compared with Auth No Guardrails, percentile values sit higher at each concurrency level because of message inspection and masking. -![Line chart of 90th, 95th, and 99th percentile response times in milliseconds versus concurrent users for AI API Advanced Guardrails on a two-CPU gateway runtime](../../../assets/img/ai-gateway/performance/2-core-Percentile-Response-Times-ai-api-advanced.svg){ width="900" } +![Line chart of 90th, 95th, and 99th percentile response times for AI API Advanced Guardrails on a two-CPU gateway runtime](../../../../assets/img/ai-gateway/performance/2-core-Percentile-Response-Times-ai-api-advanced.svg){ width="900" } **Key observations:** diff --git a/en/docs/ai-gateway/next/performance/overview.md b/en/docs/ai-gateway/next/run-the-gateway/sizing-and-performance/overview.md similarity index 89% rename from en/docs/ai-gateway/next/performance/overview.md rename to en/docs/ai-gateway/next/run-the-gateway/sizing-and-performance/overview.md index 481ee0c6d..890bb1030 100644 --- a/en/docs/ai-gateway/next/performance/overview.md +++ b/en/docs/ai-gateway/next/run-the-gateway/sizing-and-performance/overview.md @@ -1,15 +1,15 @@ --- title: "API Platform AI Gateway Performance" description: "Review WSO2 API Platform AI Gateway performance test methodology, deployment architecture, metrics, and benchmark results for two-CPU and four-CPU gateway runtime configurations." -canonical_url: https://wso2.com/api-platform/docs/ai-gateway/performance/overview/ -md_url: https://wso2.com/api-platform/docs/ai-gateway/performance/overview.md +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/sizing-and-performance/overview/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/sizing-and-performance/overview.md tags: - ai-gateway - performance - benchmarks - guardrails author: WSO2 API Platform Documentation Team -last_updated: 2026-07-15 +last_updated: 2026-08-11 content_type: "overview" --- @@ -34,7 +34,7 @@ The following key metrics were used to evaluate AI Gateway performance: The diagram below shows the deployment architecture used for the performance tests documented here. -![Performance test deployment architecture showing Apache JMeter client and servers, Netty HTTP backend, and WSO2 API Platform AI Gateway on separate EC2 instances connected in a test topology](../../../assets/img/ai-gateway/performance/deployment-used-for-test.png){ width="900" } +![Performance test topology with a JMeter client and servers, a Netty HTTP backend, and AI Gateway on separate EC2 instances](../../../../assets/img/ai-gateway/performance/deployment-used-for-test.png){ width="900" } | Component | EC2 Instance Type | vCPU | Memory (GiB) | | ---------------------------- | ----------------- | :--: | :----------: | diff --git a/en/docs/ai-workspace/1.0.0/policies/overview.md b/en/docs/ai-workspace/1.0.0/policies/overview.md index 1da4d913c..aace0b5d7 100644 --- a/en/docs/ai-workspace/1.0.0/policies/overview.md +++ b/en/docs/ai-workspace/1.0.0/policies/overview.md @@ -88,7 +88,7 @@ These policies shape how requests are routed and composed: ### Provider transformation policies -These policies translate an OpenAI Chat Completions request into another provider's API shape, and translate the response back. Pair them with the [LLM header router](https://wso2.com/api-platform/policy-hub/policies/llm-header-router) to route one endpoint across several providers. Use one on its own to point a single OpenAI-shaped endpoint at a different provider. For an end-to-end example configured on the gateway, see [Multi-provider routing](../../../ai-gateway/next/llm-proxy/multi-provider-routing.md). +These policies translate an OpenAI Chat Completions request into another provider's API shape, and translate the response back. Pair them with the [LLM header router](https://wso2.com/api-platform/policy-hub/policies/llm-header-router) to route one endpoint across several providers. Use one on its own to point a single OpenAI-shaped endpoint at a different provider. For an end-to-end example configured on the gateway, see [Multi-provider routing](../../../ai-gateway/next/expose-llms/multi-provider-routing.md). | Policy | Target provider | |--------|-----------------| diff --git a/en/docs/ai-workspace/next/policies/overview.md b/en/docs/ai-workspace/next/policies/overview.md index ea71300de..6df4f6567 100644 --- a/en/docs/ai-workspace/next/policies/overview.md +++ b/en/docs/ai-workspace/next/policies/overview.md @@ -88,7 +88,7 @@ These policies shape how requests are routed and composed: ### Provider transformation policies -These policies translate an OpenAI Chat Completions request into another provider's API shape, and translate the response back. Pair them with the [LLM header router](https://wso2.com/api-platform/policy-hub/policies/llm-header-router) to route one endpoint across several providers, or use one on its own to point a single OpenAI-shaped endpoint at a different provider. For an end-to-end example configured on the gateway, see [Multi-provider routing](../../../ai-gateway/next/llm-proxy/multi-provider-routing.md). +These policies translate an OpenAI Chat Completions request into another provider's API shape, and translate the response back. Pair them with the [LLM header router](https://wso2.com/api-platform/policy-hub/policies/llm-header-router) to route one endpoint across several providers, or use one on its own to point a single OpenAI-shaped endpoint at a different provider. For an end-to-end example configured on the gateway, see [Multi-provider routing](../../../ai-gateway/next/expose-llms/multi-provider-routing.md). | Policy | Target provider | |--------|-----------------| diff --git a/en/docs/ai-workspace/next/sync-gateway-created-artifacts.md b/en/docs/ai-workspace/next/sync-gateway-created-artifacts.md index f3d942a29..bb2f3948b 100644 --- a/en/docs/ai-workspace/next/sync-gateway-created-artifacts.md +++ b/en/docs/ai-workspace/next/sync-gateway-created-artifacts.md @@ -306,7 +306,7 @@ You can create artifacts on a gateway while it's disconnected, and they reconcil ## Immutable gateways -Some gateways run in **immutable** mode, where artifacts are loaded from on-disk configuration at startup rather than created through the management API (see [Immutable Gateway](../../ai-gateway/next/deployment/deployment-modes/immutable-gateway.md)). +Some gateways run in **immutable** mode, where artifacts are loaded from on-disk configuration at startup rather than created through the management API (see [Immutable Gateway](../../ai-gateway/next/run-the-gateway/immutable-gateway.md)). The sync behaves exactly the same for these gateways: artifacts loaded from files are synced up to AI Workspace just like ones created through the management API, with the same read-only copies and the same automatic reconciliation — no extra configuration. An immutable, file-driven gateway is still fully visible in AI Workspace. diff --git a/en/docs/llms.txt b/en/docs/llms.txt index 1af47f9de..652583c9f 100644 --- a/en/docs/llms.txt +++ b/en/docs/llms.txt @@ -310,43 +310,38 @@ - [AI Gateway Overview](https://wso2.com/api-platform/docs/ai-gateway/overview.md): Overview of the standalone AI Gateway: LLM proxy, MCP proxy, deployment, and observability - [AI Gateway Quick Start](https://wso2.com/api-platform/docs/ai-gateway/quick-start-guide.md): Deploy the standalone AI Gateway and route your first LLM request -- [LLM Provider Templates](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/llm-templates.md): Provider templates for OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Gemini, and Azure AI Foundry -- [Semantic Caching](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/semantic-caching.md): Cache LLM responses by semantic similarity in the standalone gateway -- [Model Round Robin](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/load-balancing/model-round-robin.md): Distribute LLM traffic evenly across multiple providers or model endpoints -- [Model Weighted Round Robin](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/load-balancing/model-weighted-round-robin.md): Distribute LLM traffic across providers with configurable traffic weights -- [Prompt Decorator](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/prompt-management/prompt-decorator.md): Prepend or append system-level instructions to every LLM request at the gateway level -- [Prompt Template](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/prompt-management/prompt-template.md): Replace 'template://' URI references in LLM request payloads with predefined, parameterized prompt templates -- [AWS Bedrock Guardrail](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/aws-bedrock-guardrail.md): AWS Bedrock Guardrail for enterprise-grade content filtering, topic detection, and PII protection -- [Azure Content Safety](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/azure-content-safety.md): Azure Content Safety integration for detecting hate speech, sexual content, violence, and self-harm -- [Content Length Guardrail](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/content-length.md): Enforce byte-length limits on LLM inputs and outputs -- [JSON Schema Guardrail](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/json-schema.md): Validate LLM request and response body content against a JSON schema definition to enforce structured data formats -- [PII Masking Regex Guardrail](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/pii-masking-regex.md): Mask personally identifiable information using configurable regex patterns -- [Regex Guardrail](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/regex.md): Block or allow content matching a regular expression pattern -- [Semantic Prompt Guard](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/semantic-prompt-guard.md): Block semantically similar prompts using embedding-based similarity matching against allow/deny phrase lists -- [Sentence Count Guardrail](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/sentence-count.md): Enforce min/max sentence count on LLM inputs and outputs -- [URL Guardrail](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/url.md): Validate and block URLs in LLM prompts or completions -- [Word Count Guardrail](https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/guardrails/word-count.md): Enforce min/max word count on LLM inputs and outputs -- [MCP Proxy Quick Start](https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/quick-start-guide.md): Set up a standalone MCP proxy and invoke it through an MCP client -- [MCP Access Control List](https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-acl-list.md): Control access to MCP tools, resources, and prompts using allow or deny mode with exceptions -- [MCP Authentication](https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-authentication.md): Enforce authentication on inbound MCP traffic -- [MCP Authorization](https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-authorization.md): Define fine-grained access control rules for MCP tools, resources, and prompts based on JWT claims and scopes -- [MCP Rewrite](https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/policies/mcp-rewrite.md): Expose user-facing names for MCP tools, resources, and prompts while mapping them to different backend capability names -- [AI Gateway Immutable Deployment](https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/immutable-gateway.md): Run the standalone AI Gateway in immutable mode, loading all configuration from files at startup -- [AI Gateway Kubernetes Overview](https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/kubernetes/overview.md): Overview of Kubernetes deployment for the standalone AI Gateway -- [AI Gateway Kubernetes Standalone](https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/kubernetes/kubernetes-standalone.md): Deploy the standalone AI Gateway on Kubernetes without an operator -- [AI Gateway Kubernetes Operator](https://wso2.com/api-platform/docs/ai-gateway/deployment/deployment-modes/kubernetes/gateway-operator.md): Deploy the standalone AI Gateway using the Kubernetes operator -- [AI Gateway Production Deployment Overview](https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/overview.md): Plan a high-availability production deployment of the standalone AI Gateway on Kubernetes with Helm, an external database, and replicated controller and runtime workloads -- [AI Gateway Security Hardening](https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/security-hardening.md): Harden the AI Gateway for production with AES-256 at-rest encryption keys, TLS for the listener and upstreams, and authentication on the management API -- [AI Gateway Database Configuration](https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/database-configuration.md): Point AI Gateway controller replicas at a shared PostgreSQL or SQL Server database, inject the password from a Secret, and tune the connection pool and EventHub -- [AI Gateway Resources and Scaling](https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/resources-and-scaling.md): Size CPU and memory for the AI Gateway controller and runtime, spread replicas with anti-affinity, and configure the Horizontal Pod Autoscaler and Pod Disruption Budget -- [Tune the AI Gateway for AI Traffic](https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/ai-workload-tuning.md): Production tuning for LLM and MCP workloads: streaming timeouts, large body buffers, guardrail execution limits, token cost pricing data, and semantic cache infrastructure -- [AI Gateway Deploy and Verify](https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/deploy-and-verify.md): Install the AI Gateway Helm chart, confirm the controller and runtime are healthy, route a live LLM request, and run upgrades and rollbacks -- [Connect an AI Gateway to AI Workspace](https://wso2.com/api-platform/docs/ai-gateway/deployment/production-deployment/control-plane-connection.md): Register a production AI Gateway with AI Workspace: the registration token as a Kubernetes Secret, the control plane address, TLS trust, and sync behavior -- [AI Gateway Logging](https://wso2.com/api-platform/docs/ai-gateway/observability/logging.md): Configure structured logging for the standalone AI Gateway -- [AI Gateway Tracing](https://wso2.com/api-platform/docs/ai-gateway/observability/tracing.md): Configure distributed tracing for the standalone AI Gateway -- [AI Gateway Moesif Analytics](https://wso2.com/api-platform/docs/ai-gateway/analytics/moesif-analytics.md): Integrate the standalone AI Gateway with Moesif for LLM usage analytics and cost tracking -- [AI Gateway Analytics Header Filter](https://wso2.com/api-platform/docs/ai-gateway/analytics/analytics-header-filter.md): Control which request and response headers are sent to analytics backends using allow or deny mode -- [AI Gateway Controller Authentication](https://wso2.com/api-platform/docs/ai-gateway/ai-gateway-rest-api/authentication.md): Configure Basic Auth or JWT/IDP authentication and role-based authorization for the AI Gateway Controller REST API +- [AI Gateway How It Works](https://wso2.com/api-platform/docs/ai-gateway/how-it-works.md): How the AI Gateway handles AI traffic: LLM proxies, LLM providers, MCP proxies, where policies attach, provider templates, and streaming responses. +- [Connect LLM Providers Overview](https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/overview.md): Connect the AI Gateway to an LLM backend: what an LLM Provider holds, who configures it, and the OpenAI, Anthropic, and AWS Bedrock connection guides. +- [OpenAI](https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/supported-providers/openai.md): Connect the AI Gateway to OpenAI: the upstream URL, API key authentication, the endpoints the provider exposes, and a request that tests the connection. +- [Anthropic](https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/supported-providers/anthropic.md): Connect the AI Gateway to the Anthropic Messages API: the upstream URL, x-api-key authentication, and the endpoint the provider exposes. +- [AWS Bedrock](https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/supported-providers/aws-bedrock.md): Connect API Platform AI Gateway to AWS Bedrock using a bearer API key or AWS Signature Version 4 authentication, then invoke a model through the gateway. +- [LLM Provider Templates](https://wso2.com/api-platform/docs/ai-gateway/connect-llm-providers/llm-templates.md): Provider templates for OpenAI, Azure OpenAI, Anthropic, Gemini, MistralAI, AWS Bedrock, and Azure AI Foundry +- [Expose LLMs to Applications Overview](https://wso2.com/api-platform/docs/ai-gateway/expose-llms/overview.md): Expose an LLM provider to applications through an LLM proxy: its own URL context, per-application policies, and the provider-level rules it inherits. +- [Control Cost and Traffic Overview](https://wso2.com/api-platform/docs/ai-gateway/control-cost-and-traffic/overview.md): Keep LLM spend and traffic volume predictable: distribute requests across model endpoints, cache equivalent prompts, and set timeouts on a resource. +- [AI Gateway Timeouts](https://wso2.com/api-platform/docs/ai-gateway/control-cost-and-traffic/timeouts.md): Configure gateway-level and API-level timeouts (connect, route, idle, and HTTP connection manager) to protect against slow or unreachable backends and slow clients. +- [Guardrails Overview](https://wso2.com/api-platform/docs/ai-gateway/control-content/overview.md): AI Gateway guardrails: LLM-aware policies for content filtering, safety, and compliance, with per-policy reference in the WSO2 API Platform Policy Hub. +- [Guardrail Execution Order](https://wso2.com/api-platform/docs/ai-gateway/control-content/execution-order.md): How guardrails execute across phases in the AI Gateway dual-hop model: the LLM Proxy chain runs before the LLM Provider chain on request, and in reverse on response. +- [MCP Proxy Overview](https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/overview.md): Route Model Context Protocol traffic through the AI Gateway with an MCP proxy, and apply authentication, authorization, and access control to that traffic. +- [MCP Proxy Quick Start](https://wso2.com/api-platform/docs/ai-gateway/mcp-proxy/create-an-mcp-proxy.md): Set up a standalone MCP proxy and invoke it through an MCP client +- [AI Gateway Immutable Deployment](https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/immutable-gateway.md): Run the standalone AI Gateway in immutable mode, loading all configuration from files at startup +- [AI Gateway Kubernetes Overview](https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/kubernetes/overview.md): Overview of Kubernetes deployment for the standalone AI Gateway +- [AI Gateway Kubernetes Standalone](https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/kubernetes/kubernetes-standalone.md): Deploy the standalone AI Gateway on Kubernetes without an operator +- [AI Gateway Kubernetes Operator](https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/kubernetes/gateway-operator.md): Deploy the standalone AI Gateway using the Kubernetes operator +- [AI Gateway Production Deployment Overview](https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/overview.md): Plan a high-availability production deployment of the standalone AI Gateway on Kubernetes with Helm, an external database, and replicated controller and runtime workloads +- [AI Gateway Security Hardening](https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/security-hardening.md): Harden the AI Gateway for production with AES-256 at-rest encryption keys, TLS for the listener and upstreams, and authentication on the management API +- [AI Gateway Database Configuration](https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/database-configuration.md): Point AI Gateway controller replicas at a shared PostgreSQL or SQL Server database, inject the password from a Secret, and tune the connection pool and EventHub +- [AI Gateway Resources and Scaling](https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/resources-and-scaling.md): Size CPU and memory for the AI Gateway controller and runtime, spread replicas with anti-affinity, and configure the Horizontal Pod Autoscaler and Pod Disruption Budget +- [Tune the AI Gateway for AI Traffic](https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/ai-workload-tuning.md): Production tuning for LLM and MCP workloads: streaming timeouts, large body buffers, guardrail execution limits, token cost pricing data, and semantic cache infrastructure +- [AI Gateway Deploy and Verify](https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/deploy-and-verify.md): Install the AI Gateway Helm chart, confirm the controller and runtime are healthy, route a live LLM request, and run upgrades and rollbacks +- [Connect an AI Gateway to AI Workspace](https://wso2.com/api-platform/docs/ai-gateway/run-the-gateway/production-deployment/control-plane-connection.md): Register a production AI Gateway with AI Workspace: the registration token as a Kubernetes Secret, the control plane address, TLS trust, and sync behavior +- [AI Gateway Logging](https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/logging.md): Configure structured logging for the standalone AI Gateway +- [AI Gateway Tracing](https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/tracing.md): Configure distributed tracing for the standalone AI Gateway +- [AI Gateway Moesif Analytics](https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/moesif-analytics.md): Integrate the standalone AI Gateway with Moesif for LLM usage analytics and cost tracking +- [AI Gateway Analytics Header Filter](https://wso2.com/api-platform/docs/ai-gateway/monitor-traffic/analytics-header-filter.md): Control which request and response headers are sent to analytics backends using allow or deny mode +- [Control Access Overview](https://wso2.com/api-platform/docs/ai-gateway/control-access/overview.md): Control access to the AI Gateway: client authentication on proxies and providers, authentication and role-based authorization on the management API, and provider-level rules for upstream endpoints. +- [Authenticate Clients](https://wso2.com/api-platform/docs/ai-gateway/control-access/authenticate-clients.md): Protect an LLM proxy or provider with the api-key-auth policy, issue consumer API keys through the management API, and manage the key lifecycle. +- [AI Gateway Controller Authentication](https://wso2.com/api-platform/docs/ai-gateway/control-access/secure-the-management-api.md): Configure Basic Auth or JWT/IDP authentication and role-based authorization for the AI Gateway Controller REST API +- [AI Gateway Default Ports](https://wso2.com/api-platform/docs/ai-gateway/reference/default-ports.md): Default ports the AI Gateway listens on: router HTTP and HTTPS traffic, the Gateway-Controller REST API, and the admin health endpoints. ## Policy Hub diff --git a/en/docs/next/index.md b/en/docs/next/index.md index 34d15dba7..d54bdbb7b 100644 --- a/en/docs/next/index.md +++ b/en/docs/next/index.md @@ -159,8 +159,8 @@ The following is a complete view of all components and where to find their docum | Run everything on my own infrastructure with a UI | [API Manager](../api-manager/overview.md) | | Run a lightweight API gateway with no UI | [API Gateway](../api-gateway/next/overview.md) | | Govern LLM traffic (rate limits, guardrails, cost control) | [AI Gateway Quick Start Guide](../ai-gateway/next/quick-start-guide.md) | -| Expose my APIs as MCP tools for AI agents | [AI Gateway MCP Proxy](../ai-gateway/next/mcp-proxy/quick-start-guide.md) | -| Govern MCP traffic (inbound + outbound) | [AI Gateway MCP Proxy](../ai-gateway/next/mcp-proxy/quick-start-guide.md) | +| Expose my APIs as MCP tools for AI agents | [AI Gateway MCP Proxy](../ai-gateway/next/mcp-proxy/create-an-mcp-proxy.md) | +| Govern MCP traffic (inbound + outbound) | [AI Gateway MCP Proxy](../ai-gateway/next/mcp-proxy/create-an-mcp-proxy.md) | | Manage LLM providers and AI policies at the organizational level | [AI Workspace](../ai-workspace/next/overview.md) | | Set up an API Portal & MCP Hub for API discovery | [API Portal](api-portal/overview.md) | | Monitor traffic and monetize my APIs | [Analytics](../analytics/overview.md) and [Monetization](../monetization/overview.md) | diff --git a/en/mkdocs.yml b/en/mkdocs.yml index ea1d711db..c0c145fcc 100644 --- a/en/mkdocs.yml +++ b/en/mkdocs.yml @@ -615,77 +615,81 @@ nav: - "next": - Overview: ai-gateway/next/overview.md - Quick Start Guide: ai-gateway/next/quick-start-guide.md - - Setup: - - Configuration & Interpolation: ai-gateway/next/setup/configuration.md - - Setting Up the Database: ai-gateway/next/setup/database-setup.md - - LLM Proxy: - - LLM Provider Templates: ai-gateway/next/llm-proxy/llm-templates.md - - Multi-Provider Routing: ai-gateway/next/llm-proxy/multi-provider-routing.md - - Guardrails: - - Overview: ai-gateway/next/llm-proxy/guardrails/overview.md - - Guardrail execution order: ai-gateway/next/llm-proxy/guardrails/execution-order.md - - AWS Bedrock Guardrail: ai-gateway/next/llm-proxy/guardrails/aws-bedrock-guardrail.md - - Azure Content Safety: ai-gateway/next/llm-proxy/guardrails/azure-content-safety.md - - Content Length Guardrail: ai-gateway/next/llm-proxy/guardrails/content-length.md - - JSON Schema Guardrail: ai-gateway/next/llm-proxy/guardrails/json-schema.md - - PII Masking Regex Guardrail: ai-gateway/next/llm-proxy/guardrails/pii-masking-regex.md - - Regex Guardrail: ai-gateway/next/llm-proxy/guardrails/regex.md - - Semantic Prompt Guardrail: ai-gateway/next/llm-proxy/guardrails/semantic-prompt-guard.md - - Sentence Count Guardrail: ai-gateway/next/llm-proxy/guardrails/sentence-count.md - - URL Guardrail: ai-gateway/next/llm-proxy/guardrails/url.md - - Word Count Guardrail: ai-gateway/next/llm-proxy/guardrails/word-count.md - - Load Balancing: - - Model Round Robin: ai-gateway/next/llm-proxy/load-balancing/model-round-robin.md - - Model Weighted Round Robin: ai-gateway/next/llm-proxy/load-balancing/model-weighted-round-robin.md - - Prompt Management: - - Prompt Decorator: ai-gateway/next/llm-proxy/prompt-management/prompt-decorator.md - - Prompt Template: ai-gateway/next/llm-proxy/prompt-management/prompt-template.md - - Semantic Caching: ai-gateway/next/llm-proxy/semantic-caching.md - - MCP Proxy: - - Quick Start Guide: ai-gateway/next/mcp-proxy/quick-start-guide.md - - Policies: - - MCP Access Control List: ai-gateway/next/mcp-proxy/policies/mcp-acl-list.md - - MCP Authentication: ai-gateway/next/mcp-proxy/policies/mcp-authentication.md - - MCP Authorization: ai-gateway/next/mcp-proxy/policies/mcp-authorization.md - - MCP Rewrite: ai-gateway/next/mcp-proxy/policies/mcp-rewrite.md - - Real-Time AI Streaming: ai-gateway/next/streaming-responses.md - - Deployment: - - Deployment Modes: - - Immutable Gateway: ai-gateway/next/deployment/deployment-modes/immutable-gateway.md + - How It Works: ai-gateway/next/how-it-works.md + - Run the Gateway: - Kubernetes: - - Overview: ai-gateway/next/deployment/deployment-modes/kubernetes/overview.md - - Standalone Mode: ai-gateway/next/deployment/deployment-modes/kubernetes/kubernetes-standalone.md - - Kubernetes Operator Mode: ai-gateway/next/deployment/deployment-modes/kubernetes/gateway-operator.md - - Production Deployment: - - Overview: ai-gateway/next/deployment/production-deployment/overview.md - - Security Hardening: ai-gateway/next/deployment/production-deployment/security-hardening.md - - Database Configuration: ai-gateway/next/deployment/production-deployment/database-configuration.md - - Resources & Scaling: ai-gateway/next/deployment/production-deployment/resources-and-scaling.md - - Tune for AI Traffic: ai-gateway/next/deployment/production-deployment/ai-workload-tuning.md - - Deploy & Verify: ai-gateway/next/deployment/production-deployment/deploy-and-verify.md - - Connect to AI Workspace: ai-gateway/next/deployment/production-deployment/control-plane-connection.md - - Resiliency: - - Timeouts: ai-gateway/next/resiliency/timeouts.md - - Observability: - - Logging: ai-gateway/next/observability/logging.md - - Tracing: ai-gateway/next/observability/tracing.md - - Analytics: - - Moesif Analytics: ai-gateway/next/analytics/moesif-analytics.md - - Analytics Header Filter: ai-gateway/next/analytics/analytics-header-filter.md - - Performance: - - Overview: ai-gateway/next/performance/overview.md - - AI Gateway runtime with two CPUs: ai-gateway/next/performance/ai-gateway-runtime-with-two-cpus.md - - AI Gateway runtime with four CPUs: ai-gateway/next/performance/ai-gateway-runtime-with-four-cpus.md - - Management API: - - Overview: ai-gateway/next/gateway-controller-management-api/overview.md - - Secure the Management API: ai-gateway/next/gateway-controller-management-api/authentication.md - - LLM Provider Template Management: ai-gateway/next/gateway-controller-management-api/llm-provider-template-management.md - - LLM Provider Management: ai-gateway/next/gateway-controller-management-api/llm-provider-management.md - - LLM Proxy Management: ai-gateway/next/gateway-controller-management-api/llm-proxy-management.md - - MCP Proxy Management: ai-gateway/next/gateway-controller-management-api/mcp-proxy-management.md - - Certificate Management: ai-gateway/next/gateway-controller-management-api/certificate-management.md - - Secrets Management: ai-gateway/next/gateway-controller-management-api/secrets-management.md - - Schemas: ai-gateway/next/gateway-controller-management-api/schemas.md + - Overview: ai-gateway/next/run-the-gateway/kubernetes/overview.md + - Standalone Mode: ai-gateway/next/run-the-gateway/kubernetes/kubernetes-standalone.md + - Kubernetes Operator Mode: ai-gateway/next/run-the-gateway/kubernetes/gateway-operator.md + - Immutable Gateway: ai-gateway/next/run-the-gateway/immutable-gateway.md + - Configuration & Interpolation: ai-gateway/next/run-the-gateway/configuration.md + - Database: ai-gateway/next/run-the-gateway/database-setup.md + - Production Deployment: + - Overview: ai-gateway/next/run-the-gateway/production-deployment/overview.md + - Security Hardening: ai-gateway/next/run-the-gateway/production-deployment/security-hardening.md + - Database Configuration: ai-gateway/next/run-the-gateway/production-deployment/database-configuration.md + - Resources & Scaling: ai-gateway/next/run-the-gateway/production-deployment/resources-and-scaling.md + - Tune for AI Traffic: ai-gateway/next/run-the-gateway/production-deployment/ai-workload-tuning.md + - Deploy & Verify: ai-gateway/next/run-the-gateway/production-deployment/deploy-and-verify.md + - Connect to AI Workspace: ai-gateway/next/run-the-gateway/production-deployment/control-plane-connection.md + - Sizing & Performance: + - Overview: ai-gateway/next/run-the-gateway/sizing-and-performance/overview.md + - AI Gateway runtime with two CPUs: ai-gateway/next/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-two-cpus.md + - AI Gateway runtime with four CPUs: ai-gateway/next/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-four-cpus.md + - Connect LLM Providers: + - Overview: ai-gateway/next/connect-llm-providers/overview.md + - Supported Providers: + - OpenAI: ai-gateway/next/connect-llm-providers/supported-providers/openai.md + - Anthropic: ai-gateway/next/connect-llm-providers/supported-providers/anthropic.md + - AWS Bedrock: ai-gateway/next/connect-llm-providers/supported-providers/aws-bedrock.md + - Provider Templates: ai-gateway/next/connect-llm-providers/llm-templates.md + - Expose LLMs to Applications: + - Overview: ai-gateway/next/expose-llms/overview.md + - Route Across Multiple Providers: ai-gateway/next/expose-llms/multi-provider-routing.md + - Stream Responses: ai-gateway/next/expose-llms/streaming-responses.md + - MCP Proxy: + - Overview: ai-gateway/next/mcp-proxy/overview.md + - Create an MCP Proxy: ai-gateway/next/mcp-proxy/create-an-mcp-proxy.md + - Control Access: + - Overview: ai-gateway/next/control-access/overview.md + - Authenticate Clients: ai-gateway/next/control-access/authenticate-clients.md + - Secure the Management API: ai-gateway/next/control-access/secure-the-management-api.md + - Control Content (Guardrails): + - Guardrails Overview: ai-gateway/next/control-content/overview.md + - Guardrail execution order: ai-gateway/next/control-content/execution-order.md + - Control Cost & Traffic: + - Overview: ai-gateway/next/control-cost-and-traffic/overview.md + - Timeouts: ai-gateway/next/control-cost-and-traffic/timeouts.md + - Monitor Traffic: + - Logging: ai-gateway/next/monitor-traffic/logging.md + - Tracing: ai-gateway/next/monitor-traffic/tracing.md + - Moesif Analytics: ai-gateway/next/monitor-traffic/moesif-analytics.md + - Analytics Header Filter: ai-gateway/next/monitor-traffic/analytics-header-filter.md + - Extend the Gateway: + - Customizing Gateway Policies: tools/cli/customizing-gateway-policies.md + - Guides: + - Set up a Governed Multi-Model LLM Proxy with Cost Controls and Failover: guides/ai-and-mcp/set-up-a-governed-multi-model-llm-proxy-with-cost-controls-and-failover.md + - Enforce Token-Based Rate Limiting on an LLM Proxy: guides/ai-and-mcp/enforce-token-based-rate-limiting-on-an-llm-proxy.md + - Enforce a Consistent AI Persona with the Prompt Decorator Policy: guides/ai-and-mcp/using-prompt-decorator-policy.md + - Build an AI App with Claude Code that Calls Governed Backend APIs: guides/ai-and-mcp/build-ai-app-with-claude-code.md + - Build an AI Agent That Uses Aggregated MCP Tools from Multiple APIs: guides/ai-and-mcp/build-ai-agent-with-multiple-mcp-servers.md + - Convert a REST API into an MCP Tool and Use It in Claude Desktop: guides/ai-and-mcp/convert-rest-api-to-mcp-server.md + - Find and Connect to an Enterprise MCP Server from the MCP Hub: guides/ai-and-mcp/find-and-connect-to-an-enterprise-mcp-server-from-the-mcp-hub.md + - Configure AI Coding Assistants: + - Configure Claude Code with AI Gateway: guides/ai-and-mcp/ai-coding-assistants/claude-code-configuration-with-ai-gateway.md + - Configure Google Gemini CLI with AI Gateway: guides/ai-and-mcp/ai-coding-assistants/gemini-cli-configuration-with-ai-gateway.md + - Configure OpenAI Codex CLI with AI Gateway: guides/ai-and-mcp/ai-coding-assistants/codex-configuration-with-ai-gateway.md + - Reference: + - Management REST API: + - Overview: ai-gateway/next/reference/management-api/overview.md + - LLM Provider Template Management: ai-gateway/next/reference/management-api/llm-provider-template-management.md + - LLM Provider Management: ai-gateway/next/reference/management-api/llm-provider-management.md + - LLM Proxy Management: ai-gateway/next/reference/management-api/llm-proxy-management.md + - MCP Proxy Management: ai-gateway/next/reference/management-api/mcp-proxy-management.md + - Certificate Management: ai-gateway/next/reference/management-api/certificate-management.md + - Secrets Management: ai-gateway/next/reference/management-api/secrets-management.md + - Schemas: ai-gateway/next/reference/management-api/schemas.md + - Default Ports: ai-gateway/next/reference/default-ports.md - About this release: ai-gateway/next/about-this-release.md - "1.2.0": - Overview: ai-gateway/1.2.0/overview.md @@ -1237,7 +1241,6 @@ nav: - Set up a Governed Multi-Model LLM Proxy with Cost Controls and Failover: guides/ai-and-mcp/set-up-a-governed-multi-model-llm-proxy-with-cost-controls-and-failover.md - Enforce Token-Based Rate Limiting on an LLM Proxy: guides/ai-and-mcp/enforce-token-based-rate-limiting-on-an-llm-proxy.md - Enforce a Consistent AI Persona with the Prompt Decorator Policy: guides/ai-and-mcp/using-prompt-decorator-policy.md - - Configure an AWS Bedrock LLM Provider: ai-gateway/next/llm-proxy/configure-aws-bedrock-provider.md - AI Coding Assistants: - Configure Claude Code with AI Gateway: guides/ai-and-mcp/ai-coding-assistants/claude-code-configuration-with-ai-gateway.md - Configure Google Gemini CLI with AI Gateway: guides/ai-and-mcp/ai-coding-assistants/gemini-cli-configuration-with-ai-gateway.md @@ -1341,10 +1344,10 @@ plugins: ai-gateway/1.2.0/deployment-modes/kubernetes/gateway-operator.md: ai-gateway/1.2.0/deployment/deployment-modes/kubernetes/gateway-operator.md ai-gateway/1.2.0/deployment-modes/kubernetes/kubernetes-standalone.md: ai-gateway/1.2.0/deployment/deployment-modes/kubernetes/kubernetes-standalone.md ai-gateway/1.2.0/deployment-modes/kubernetes/overview.md: ai-gateway/1.2.0/deployment/deployment-modes/kubernetes/overview.md - ai-gateway/next/deployment-modes/immutable-gateway.md: ai-gateway/next/deployment/deployment-modes/immutable-gateway.md - ai-gateway/next/deployment-modes/kubernetes/gateway-operator.md: ai-gateway/next/deployment/deployment-modes/kubernetes/gateway-operator.md - ai-gateway/next/deployment-modes/kubernetes/kubernetes-standalone.md: ai-gateway/next/deployment/deployment-modes/kubernetes/kubernetes-standalone.md - ai-gateway/next/deployment-modes/kubernetes/overview.md: ai-gateway/next/deployment/deployment-modes/kubernetes/overview.md + ai-gateway/next/deployment-modes/immutable-gateway.md: ai-gateway/next/run-the-gateway/immutable-gateway.md + ai-gateway/next/deployment-modes/kubernetes/gateway-operator.md: ai-gateway/next/run-the-gateway/kubernetes/gateway-operator.md + ai-gateway/next/deployment-modes/kubernetes/kubernetes-standalone.md: ai-gateway/next/run-the-gateway/kubernetes/kubernetes-standalone.md + ai-gateway/next/deployment-modes/kubernetes/overview.md: ai-gateway/next/run-the-gateway/kubernetes/overview.md # AI/API Gateway versioning restructuring (per-product Cloud/Self-Hosted variants) ai-gateway/ai-gateway-rest-api/authentication.md: ai-gateway/1.1.0/ai-gateway-rest-api/authentication.md ai-gateway/analytics/analytics-header-filter.md: ai-gateway/1.1.0/analytics/analytics-header-filter.md @@ -1430,6 +1433,89 @@ plugins: api-gateway/setup/upstream-timeouts.md: api-gateway/1.1.0/setup/upstream-timeouts.md # AI Gateway revamp phase 0: the two quick starts merged into one canonical page ai-gateway/next/llm-proxy/quick-start-guide.md: ai-gateway/next/quick-start-guide.md + # AI Gateway revamp phase 1: job-oriented nav + ai-gateway/next/setup/configuration.md: ai-gateway/next/run-the-gateway/configuration.md + ai-gateway/next/setup/database-setup.md: ai-gateway/next/run-the-gateway/database-setup.md + ai-gateway/next/deployment/deployment-modes/immutable-gateway.md: ai-gateway/next/run-the-gateway/immutable-gateway.md + ai-gateway/next/deployment/deployment-modes/kubernetes/overview.md: ai-gateway/next/run-the-gateway/kubernetes/overview.md + ai-gateway/next/deployment/deployment-modes/kubernetes/kubernetes-standalone.md: ai-gateway/next/run-the-gateway/kubernetes/kubernetes-standalone.md + ai-gateway/next/deployment/deployment-modes/kubernetes/gateway-operator.md: ai-gateway/next/run-the-gateway/kubernetes/gateway-operator.md + ai-gateway/next/deployment/production-deployment/overview.md: ai-gateway/next/run-the-gateway/production-deployment/overview.md + ai-gateway/next/deployment/production-deployment/security-hardening.md: ai-gateway/next/run-the-gateway/production-deployment/security-hardening.md + ai-gateway/next/deployment/production-deployment/database-configuration.md: ai-gateway/next/run-the-gateway/production-deployment/database-configuration.md + ai-gateway/next/deployment/production-deployment/resources-and-scaling.md: ai-gateway/next/run-the-gateway/production-deployment/resources-and-scaling.md + ai-gateway/next/deployment/production-deployment/ai-workload-tuning.md: ai-gateway/next/run-the-gateway/production-deployment/ai-workload-tuning.md + ai-gateway/next/deployment/production-deployment/deploy-and-verify.md: ai-gateway/next/run-the-gateway/production-deployment/deploy-and-verify.md + ai-gateway/next/deployment/production-deployment/control-plane-connection.md: ai-gateway/next/run-the-gateway/production-deployment/control-plane-connection.md + ai-gateway/next/performance/overview.md: ai-gateway/next/run-the-gateway/sizing-and-performance/overview.md + ai-gateway/next/performance/ai-gateway-runtime-with-two-cpus.md: ai-gateway/next/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-two-cpus.md + ai-gateway/next/performance/ai-gateway-runtime-with-four-cpus.md: ai-gateway/next/run-the-gateway/sizing-and-performance/ai-gateway-runtime-with-four-cpus.md + ai-gateway/next/llm-proxy/llm-templates.md: ai-gateway/next/connect-llm-providers/llm-templates.md + ai-gateway/next/llm-proxy/configure-aws-bedrock-provider.md: ai-gateway/next/connect-llm-providers/supported-providers/aws-bedrock.md + ai-gateway/next/connect-llm-providers/configure-aws-bedrock-provider.md: ai-gateway/next/connect-llm-providers/supported-providers/aws-bedrock.md + ai-gateway/next/llm-proxy/multi-provider-routing.md: ai-gateway/next/expose-llms/multi-provider-routing.md + ai-gateway/next/streaming-responses.md: ai-gateway/next/expose-llms/streaming-responses.md + ai-gateway/next/mcp-proxy/quick-start-guide.md: ai-gateway/next/mcp-proxy/create-an-mcp-proxy.md + # AI Gateway revamp phase 2: per-policy reference moved to Policy Hub + ai-gateway/next/mcp-proxy/mcp-authentication.md: https://wso2.com/api-platform/policy-hub/policies/mcp-auth + ai-gateway/next/mcp-proxy/mcp-authorization.md: https://wso2.com/api-platform/policy-hub/policies/mcp-authz + ai-gateway/next/mcp-proxy/mcp-acl-list.md: https://wso2.com/api-platform/policy-hub/policies/mcp-acl-list + ai-gateway/next/mcp-proxy/mcp-rewrite.md: https://wso2.com/api-platform/policy-hub/policies/mcp-rewrite + # Phase 1 paths, repointed past the deleted pages to the same Policy Hub targets + ai-gateway/next/mcp-proxy/policies/mcp-rewrite.md: https://wso2.com/api-platform/policy-hub/policies/mcp-rewrite + ai-gateway/next/mcp-proxy/policies/mcp-authentication.md: https://wso2.com/api-platform/policy-hub/policies/mcp-auth + ai-gateway/next/mcp-proxy/policies/mcp-authorization.md: https://wso2.com/api-platform/policy-hub/policies/mcp-authz + ai-gateway/next/mcp-proxy/policies/mcp-acl-list.md: https://wso2.com/api-platform/policy-hub/policies/mcp-acl-list + ai-gateway/next/gateway-controller-management-api/authentication.md: ai-gateway/next/control-access/secure-the-management-api.md + ai-gateway/next/llm-proxy/guardrails/overview.md: ai-gateway/next/control-content/overview.md + ai-gateway/next/llm-proxy/guardrails/execution-order.md: ai-gateway/next/control-content/execution-order.md + # AI Gateway revamp phase 2: per-policy reference moved to Policy Hub + ai-gateway/next/control-content/regex.md: https://wso2.com/api-platform/policy-hub/policies/regex-guardrail + ai-gateway/next/control-content/json-schema.md: https://wso2.com/api-platform/policy-hub/policies/json-schema-guardrail + ai-gateway/next/control-content/word-count.md: https://wso2.com/api-platform/policy-hub/policies/word-count-guardrail + ai-gateway/next/control-content/sentence-count.md: https://wso2.com/api-platform/policy-hub/policies/sentence-count-guardrail + ai-gateway/next/control-content/content-length.md: https://wso2.com/api-platform/policy-hub/policies/content-length-guardrail + ai-gateway/next/control-content/url.md: https://wso2.com/api-platform/policy-hub/policies/url-guardrail + ai-gateway/next/control-content/pii-masking-regex.md: https://wso2.com/api-platform/policy-hub/policies/pii-masking-regex + ai-gateway/next/control-content/semantic-prompt-guard.md: https://wso2.com/api-platform/policy-hub/policies/semantic-prompt-guard + ai-gateway/next/control-content/azure-content-safety.md: https://wso2.com/api-platform/policy-hub/policies/azure-content-safety-content-moderation + ai-gateway/next/control-content/aws-bedrock-guardrail.md: https://wso2.com/api-platform/policy-hub/policies/aws-bedrock-guardrail + ai-gateway/next/control-content/prompt-decorator.md: https://wso2.com/api-platform/policy-hub/policies/prompt-decorator + ai-gateway/next/control-content/prompt-template.md: https://wso2.com/api-platform/policy-hub/policies/prompt-template + # Phase 1 paths, repointed past the deleted pages to the same Policy Hub targets + ai-gateway/next/llm-proxy/guardrails/regex.md: https://wso2.com/api-platform/policy-hub/policies/regex-guardrail + ai-gateway/next/llm-proxy/guardrails/json-schema.md: https://wso2.com/api-platform/policy-hub/policies/json-schema-guardrail + ai-gateway/next/llm-proxy/guardrails/word-count.md: https://wso2.com/api-platform/policy-hub/policies/word-count-guardrail + ai-gateway/next/llm-proxy/guardrails/sentence-count.md: https://wso2.com/api-platform/policy-hub/policies/sentence-count-guardrail + ai-gateway/next/llm-proxy/guardrails/content-length.md: https://wso2.com/api-platform/policy-hub/policies/content-length-guardrail + ai-gateway/next/llm-proxy/guardrails/url.md: https://wso2.com/api-platform/policy-hub/policies/url-guardrail + ai-gateway/next/llm-proxy/guardrails/pii-masking-regex.md: https://wso2.com/api-platform/policy-hub/policies/pii-masking-regex + ai-gateway/next/llm-proxy/guardrails/semantic-prompt-guard.md: https://wso2.com/api-platform/policy-hub/policies/semantic-prompt-guard + ai-gateway/next/llm-proxy/guardrails/azure-content-safety.md: https://wso2.com/api-platform/policy-hub/policies/azure-content-safety-content-moderation + ai-gateway/next/llm-proxy/guardrails/aws-bedrock-guardrail.md: https://wso2.com/api-platform/policy-hub/policies/aws-bedrock-guardrail + ai-gateway/next/llm-proxy/prompt-management/prompt-decorator.md: https://wso2.com/api-platform/policy-hub/policies/prompt-decorator + ai-gateway/next/llm-proxy/prompt-management/prompt-template.md: https://wso2.com/api-platform/policy-hub/policies/prompt-template + # AI Gateway revamp phase 2: per-policy reference moved to Policy Hub + ai-gateway/next/control-cost-and-traffic/model-round-robin.md: https://wso2.com/api-platform/policy-hub/policies/model-round-robin + ai-gateway/next/control-cost-and-traffic/model-weighted-round-robin.md: https://wso2.com/api-platform/policy-hub/policies/model-weighted-round-robin + ai-gateway/next/control-cost-and-traffic/semantic-caching.md: https://wso2.com/api-platform/policy-hub/policies/semantic-cache + # Phase 1 paths, repointed past the deleted pages to the same Policy Hub targets + ai-gateway/next/llm-proxy/load-balancing/model-round-robin.md: https://wso2.com/api-platform/policy-hub/policies/model-round-robin + ai-gateway/next/llm-proxy/load-balancing/model-weighted-round-robin.md: https://wso2.com/api-platform/policy-hub/policies/model-weighted-round-robin + ai-gateway/next/llm-proxy/semantic-caching.md: https://wso2.com/api-platform/policy-hub/policies/semantic-cache + ai-gateway/next/resiliency/timeouts.md: ai-gateway/next/control-cost-and-traffic/timeouts.md + ai-gateway/next/observability/logging.md: ai-gateway/next/monitor-traffic/logging.md + ai-gateway/next/observability/tracing.md: ai-gateway/next/monitor-traffic/tracing.md + ai-gateway/next/analytics/moesif-analytics.md: ai-gateway/next/monitor-traffic/moesif-analytics.md + ai-gateway/next/analytics/analytics-header-filter.md: ai-gateway/next/monitor-traffic/analytics-header-filter.md + ai-gateway/next/gateway-controller-management-api/overview.md: ai-gateway/next/reference/management-api/overview.md + ai-gateway/next/gateway-controller-management-api/llm-provider-template-management.md: ai-gateway/next/reference/management-api/llm-provider-template-management.md + ai-gateway/next/gateway-controller-management-api/llm-provider-management.md: ai-gateway/next/reference/management-api/llm-provider-management.md + ai-gateway/next/gateway-controller-management-api/llm-proxy-management.md: ai-gateway/next/reference/management-api/llm-proxy-management.md + ai-gateway/next/gateway-controller-management-api/mcp-proxy-management.md: ai-gateway/next/reference/management-api/mcp-proxy-management.md + ai-gateway/next/gateway-controller-management-api/certificate-management.md: ai-gateway/next/reference/management-api/certificate-management.md + ai-gateway/next/gateway-controller-management-api/secrets-management.md: ai-gateway/next/reference/management-api/secrets-management.md + ai-gateway/next/gateway-controller-management-api/schemas.md: ai-gateway/next/reference/management-api/schemas.md # AI Workspace self-hosted docs moved from next/ai-workspace to the # versioned ai-workspace/ layout; 1.0.0 is the latest release. next/ai-workspace/index.md: ai-workspace/1.0.0/overview.md