AI Gateways like litellm proxy support virtual API tokens to track usage of LLM tokens per client.
We need to do at least two things:
- Find a way to abstract and configure these virtual api tokens for different gateway implementations
- Wire the tokens between agent instances and gateways
This task requires further refinement, before implementation can start.
AI Gateways like litellm proxy support virtual API tokens to track usage of LLM tokens per client.
We need to do at least two things:
This task requires further refinement, before implementation can start.