Add GPT-5.5 and GPT-5.4 support via LiteLLM proxy#4
Open
rarmist wants to merge 31 commits into
Open
Conversation
Raw Bedrock model IDs, Opus 4.7/4.8, /swe + /summarize skills, uv-managed deps
Pulls the official wording from the Amazon Bedrock inference docs (https://docs.aws.amazon.com/bedrock/latest/userguide/inference.html) and applies it consistently across every place the architecture is described: - The bedrock-mantle endpoint is named explicitly (no more vague 'OpenAI-compatible Bedrock endpoint') - Both architecture diagrams (root + bedrock READMEs) now show bedrock-mantle and the actual API paths under each Bedrock node (/anthropic/v1/messages and /v1/chat/completions) - 'Why a proxy?' and 'Why this endpoint?' paragraphs added to the ROOT README (previously only in the bedrock sub-readme), with links to the AWS docs for each of the three Mantle APIs - 'How It Works' steps in bedrock/README.md re-worded to match - Updated header comments in bedrock/scripts/setup-proxy.sh and bedrock/config/litellm-config.yaml so internal docs match the user-facing wording - Footnote about 'OpenAI-compatible Bedrock endpoint available in us-east-1' renamed to 'bedrock-mantle endpoint' - Cleaned up the stale phrasing that put 'OpenAI-compatible' on the models rather than on the endpoint
- Move API path labels OUT of the Bedrock boxes and onto the arrows feeding into them. Boxes are simpler now: just 'Amazon Bedrock' (left) and 'Amazon Bedrock (mantle endpoint)' (right), with the model groups listed below each. - Use /v1/messages (the standard Anthropic Messages API path) on the left arrow rather than /anthropic/v1/messages — even though Bedrock's mantle endpoint actually serves it at /anthropic/v1/messages, the generic path is more recognizable to readers familiar with the Anthropic API. - Drop the redundant 'OpenAI Chat Completions' format label between the proxy and Bedrock — the path /v1/chat/completions on the arrow already conveys the same thing. - Make root README and bedrock README diagrams identical so they don't drift again.
Both arrows leaving Claude Code carry the same Anthropic Messages format
('POST /v1/messages'), so labeling them once at the source is cleaner
than labeling each arrow individually:
Before — duplicated label appeared on both lanes (right-side near the
proxy AND left-side near the direct-to-Bedrock arrow). Made
the diagram visually noisy.
After — single 'POST /v1/messages' next to the Claude Code box.
Only the right lane needs the '/v1/chat/completions' label
below the proxy, since that lane is where the format changes.
Both READMEs (root + bedrock) updated to keep the diagrams identical.
Replace the four hand-drawn ASCII diagrams (Bedrock arch + self-hosted arch, each appearing twice across the three READMEs) with proper Mermaid flowcharts. GitHub renders Mermaid natively in markdown so the diagrams now display as real boxes-and-arrows on the repo page, not monospace ASCII art. Each diagram uses styled classes: - Claude Code box: dark grey (agent) - LiteLLM Proxy: purple (translation layer) - Amazon Bedrock and EC2 boxes: AWS orange Edge labels carry the wire format / network detail (Anthropic Messages, /v1/chat/completions, SSH tunnel:11434). Source-of-truth is now version-controlled markdown text instead of fragile ASCII alignment. The repo-structure file tree at root README:281 stays as text fence -- it is a directory listing, not an architecture diagram.
The first Mermaid pass had two visible problems on GitHub: 1. Box text was clipped (e.g. "Claude Code CL" instead of "Claude Code CLI", "POST /v1/messag" instead of "POST /v1/messages"). This was caused by HTML tags (<b>, <sub>) inside node labels — Mermaid measures box width based on the unrendered label and undersizes when sub/strong tags compress text. 2. The saturated AWS-orange and bright-purple fills were loud for technical docs. Changes: - Drop all HTML tags from node labels (<b>, <sub>); use plain text + <br/> for line breaks. Mermaid sizes the boxes correctly now. - Use lighter, less saturated palette: light grey for the agent, lavender for the proxy, soft cream for the Bedrock/EC2 nodes (with the AWS-orange as the border accent rather than the fill). - Add a horizontal rule (───) inside the Bedrock nodes to visually separate the service name from the model list. - Switch -->|label|node syntax to -- "label" --> node so labels with slashes and arrows render cleanly. Diagrams pushed to origin only for preview; aws-samples mirror waits until the rendering is verified.
The previous Mermaid auto-layout placed the left lane (Anthropic models / direct to Bedrock) to the right of the LiteLLM proxy lane because the right lane has more depth and the layout engine balanced by pulling the shorter lane outward. Wrap each lane in its own subgraph: Left: "Anthropic models (direct)" contains BedrockA Right: "Third-party models (via proxy)" contains Proxy + BedrockM Mermaid keeps subgraph contents grouped, so the left lane stays on the left and the right lane stays on the right. The subgraph headers also double as readable lane labels, making the two routes self-explanatory. Pushed to origin for preview only — aws-samples mirror still pending visual approval.
…acement" This reverts commit 700fd02.
Add a transparent ghost node SpacerL beneath BedrockA and an invisible edge BedrockA ~~~ SpacerL. Mermaid sees the left lane as 2 levels deep (matching the right lane: Proxy -> BedrockM), so the layout engine should now place BedrockA directly under the Anthropic-Messages arrow on the left instead of pushing it sideways to balance. Pushed to origin only for visual verification — if it makes layout worse or breaks rendering (Mermaid is finicky about ~~~ + :::ghost combos), revert again.
…n FAISS removal Three new model/problem cells under benchmarks/swe-benchmark-data/mcp-gateway-registry/: - remove-efs-from-terraform-aws-ecs/qwen-qwen3-coder-next/ - remove-efs-from-terraform-aws-ecs/minimax-m2-5/ - remove-faiss/minimax-m2-5/ Each cell contains the four /swe artifacts: github-issue.md, lld.md, review.md, testing.md. Tag pinned at 1.24.4 (matches existing Qwen remove-faiss reference). Kimi K2-Thinking and Claude Opus 4.8 still pending; will land in follow-up commits.
Two new model/problem cells under benchmarks/swe-benchmark-data/mcp-gateway-registry/: - remove-faiss/kimi-k2-thinking/ - remove-efs-from-terraform-aws-ecs/kimi-k2-thinking/ Each cell contains the four /swe artifacts: github-issue.md, lld.md, review.md, testing.md. Slug normalized to vendor-less kebab-case to match the existing minimax-m2-5 and qwen-qwen3-coder-next style. Tag pinned at 1.24.4. Claude Opus 4.8 still pending; will land in a follow-up commit.
Two new model/problem cells under benchmarks/swe-benchmark-data/mcp-gateway-registry/: - remove-faiss/claude-opus-4-8/ - remove-efs-from-terraform-aws-ecs/claude-opus-4-8/ Each cell contains the four /swe artifacts: github-issue.md, lld.md, review.md, testing.md. Slug normalized to vendor-less kebab-case to match the existing kimi-k2-thinking, minimax-m2-5, and qwen-qwen3-coder-next folders. Tag pinned at 1.24.4. Completes the 4 model x 2 problem benchmark matrix for mcp-gateway-registry.
… scoring rubric - New tasks: ssrf-hardening (#1282), migrate-ecs-env-vars (#1134), replace-keycloak-db-password (#1303) - Add Mistral Devstral 2 123B artifacts for remove-faiss and remove-efs - Update benchmark README with 5-task table, difficulty ratings, issue links, and 25%-per-artifact scoring rubric
All 5 benchmark models completed the ssrf-hardening-outbound-url-validation task against mcp-gateway-registry @ 1.24.4: - Claude Opus 4.8 - Kimi K2 Thinking - Mistral Devstral 2 123B - MiniMax M2.5 - Qwen Coder Next Each produced the 4 standard /swe artifacts (github-issue, lld, review, testing). Model folder names normalized to match the convention used in remove-faiss/ and remove-efs-from-terraform-aws-ecs/.
… #1134, #1303) Completes the 5-task × 5-model benchmark matrix against mcp-gateway-registry @ 1.24.4. All 5 models produced the 4 standard /swe artifacts (github-issue, lld, review, testing) for both new tasks: - Claude Opus 4.8 - Kimi K2.5 (substituted for K2-Thinking on these tasks due to backend reliability issues; tasks 1-3 still use K2-Thinking) - Mistral Devstral 2 123B - MiniMax M2.5 - Qwen Coder Next Also updates the /swe skill with a "Benchmark Isolation" constraint block: no reading sibling model artifacts, no claude-peers messaging during runs, only the cloned repo as code source. And tightens the task aws-samples#4 README description to match the ECS-native (secrets block) phrasing that was used in the actual prompts.
… gitignore - bedrock/pyproject.toml: replaced unrelated NYC-analytics project with the actual proxy + benchmark dependencies (litellm[proxy], aws-bedrock-token-generator, datasets). Regenerated bedrock/uv.lock. Without this, a fresh `uv sync` did not install litellm at all. - bedrock/scripts/setup-proxy.sh: clarified that --refresh writes a new token file but a running proxy keeps the old token until restart, in the usage comment, the --help output, and the runtime message. Removed the misleading "no restart" wording. - bedrock/README.md: added a security warning to the HumanEval reproduce section noting that model-generated Python is executed on the host without sandboxing. Suggests running in a disposable container/VM or wrapping the subprocess in firejail/bwrap/docker. - .gitignore: added **/benchmarks/swe-benchmark-data/*/repo/ so a nested copy of the cloned target repo (e.g. accidentally created under bedrock/) does not get committed and scanned by CI. - bedrock/config/litellm-config.yaml: refined the header comment.
- 25 per-cell judge-gpt.json files with criterion breakdown (completeness, correctness, specificity, risk-awareness) plus judge notes - JUDGE_RESULTS.md with the 5x5 matrix, per-model leaderboard, per-task difficulty ranking, and synthesis Judge: independent ChatGPT session (cross-lineage). Prompt template kept in a local working doc; only the input artifacts, per-cell scores, and aggregate results are checked in here so the chain is reproducible. Headline: Opus 4.8 wins every row (89.95 avg). Kimi family combined is a clear aws-samples#2 (82.15 across 3 K2-Thinking + 2 K2.5 cells, substituted due to K2-Thinking backend hangs mid-benchmark). The mid/budget tier clusters at 74-80 with task-by-task variance. Qwen Coder Next has the highest mid-tier average (79.80) but its lead is driven by an SSRF outlier; Devstral wins remove-efs and MiniMax wins keycloak-iam. SSRF was the hardest task by score (76.3 avg, 23.8-point spread), not the README-labelled "High" tasks.
Dependabot flagged cryptography 46.0.7 (transitive via boto3/msal/azure) as vulnerable to GHSA-537c-gmf6-5ccf: the bundled OpenSSL in pyca/ cryptography wheels prior to 48.0.1 has a security issue (advisory published 2026-06-15). Added explicit cryptography>=48.0.1 to bedrock/pyproject.toml so the resolver picks 48.0.1 even when upstream deps would accept 46.x. Regenerated uv.lock. Verified `uv sync --frozen` produces a working install with LiteLLM 1.89.4 and cryptography 48.0.1. Real exploitability in this repo is low (cryptography is only used indirectly for AWS API TLS), but keeping a known-vulnerable transitive in a public aws-samples lock file is bad optics.
- Root README "How it runs" now describes both deployment paths in proportion: Bedrock uses LiteLLM (Anthropic Messages -> OpenAI Chat Completions translation), self-hosted EC2 uses Ollama directly via SSH tunnel (Ollama accepts Anthropic Messages natively, no proxy). Architecture diagram label corrected from "OpenAI-compatible" to "Anthropic Messages compatible" for the Ollama box. - Root README "Worked example" section updated from the stale "two problems, two Qwen artifact sets" framing to the actual 5 task x 5 model = 25 artifact bundle matrix. Adds difficulty + issue-source columns, lists all 5 benchmarked models, links the GPT-judged scores at JUDGE_RESULTS.md, and surfaces the headline result. - Root README repo-structure tree now shows all 5 task folders + JUDGE_RESULTS.md + judge-gpt.json per cell. - Benchmark README task 5 description corrected: the repo uses an Aurora MySQL cluster for Keycloak, not PostgreSQL. Updated the task text to reference the actual engine and the rds:GenerateDBAuthToken IAM mechanism. (Most model artifacts caught this on their own from reading the Terraform; the README prose was the only thing wrong.) - Benchmark README scoring section tightened from "weighted equally (25% each)" to the actual 4-criterion x 25-point rubric the judge used (Completeness, Correctness, Specificity, Risk Awareness), with a pointer to JUDGE_RESULTS.md and per-cell judge-gpt.json. - self-hosted README architecture sentence corrected: Claude Code uses ANTHROPIC_BASE_URL, Ollama accepts Anthropic Messages natively, the SSH tunnel is just network plumbing — not a format-translation layer.
…nd URLs - Root README Overview now names the /swe skill explicitly (was: "a per-task Software Engineering benchmark") and surfaces the 5x5 matrix + GPT-judged result up front. - Root README Evaluation 1 section gains a "Scoring rubric (LLM-as-judge)" subsection with the full 4-criterion x 25-point table (Completeness, Correctness, Specificity, Risk Awareness) and calibration notes. A reader scanning top-to-bottom now sees how scores were produced without clicking through to benchmarks/swe-benchmark-data/README.md. - bedrock/README.md corrects the provider count from 12 to 11 (Anthropic + 10 third-party) and adds "from 11 providers" to the headline sentence. - bedrock/scripts/setup-proxy.sh simplifies the post-start usage example to point at the bundled settings file + --model flag rather than inline env vars. - self-hosted/README.md replaces stale shekharprateek/claude-code-on-amazon-ec2 URLs with the canonical aws-samples/sample-claude-code-multi-model paths (curl, git clone, See Also), and points LICENSE link to ../LICENSE. - self-hosted/scripts/ec2-setup.sh clone-path comment now matches the current repo name (sample-claude-code-multi-model) instead of the old claude-code-on-amazon-ec2 name.
Replaces the prose-only headline ("Opus wins every row...") with the
actual data a reader needs:
- 5x5 matrix of GPT-judged scores, all 25 cells visible
- Per-model leaderboard (Opus 89.95 / Kimi 82.15 / Qwen 79.80 /
Devstral 75.95 / MiniMax 74.70)
- "What the data says" — 6 findings covering Opus margin, Kimi as aws-samples#2,
mid/budget cluster, SSRF as the genuine hardest task, Qwen's
coder-specialist sweet spot, and the 20x cost / 15-point quality
spread
A reader scanning the root README now sees the rubric (added previously)
followed immediately by the results. Per-cell judge notes and the full
synthesis still live in JUDGE_RESULTS.md for anyone who wants the
detail.
The "What you get end to end" bullet for the GPT-judged matrix was just a hyperlink to JUDGE_RESULTS.md. Replaced with the actual top-5 model ranking inline (Opus 89.95 / Kimi 82.15 / Qwen 79.80 / Devstral 75.95 / MiniMax 74.70), plus an anchor link down to the Worked Example section that has the full 5x5 matrix and findings. A reader scanning the README front-matter now sees the headline result without having to leave the page.
- README, JUDGE_RESULTS.md, and benchmark README now render scores with % suffixes (89.95% vs 89.95) so the unit is unambiguous. Added a "scores are 0–100%" header note above the score matrix in both consolidated docs. - Tasks 1 (`remove-faiss`) and 2 (`remove-efs-from-terraform-aws-ecs`) now reference the upstream issues that match them: - #1285 + #452 (FAISS / file-backend / scopes removal) - #1286 (EFS removal from terraform/aws-ecs) Both were filed by upstream maintainers in the same week we built our task descriptions — independent triangulation that the scope was real. All 5 tasks are now linked to upstream GitHub issues; no more "Internal" sourcing.
Split self-hosted/ into ollama/ (existing single-GPU convenience path) and a new vllm/ throughput path that serves open-weight coding models tensor-parallel across all GPUs — the serving layer the hosting-strategy cost model depends on. Verified end to end on a g6e.12xlarge (4x L40S, Ubuntu 24.04 DLAMI): vLLM 0.24.0 serving Qwen3-Coder-30B-A3B across 4 GPUs, real chat completion, ~32x max concurrency at 32K context. - scripts/vllm-install.sh: driver check, apt deps, uv venv, vLLM, nvtop + gpustat, GPU verify. Handles the two DLAMI-specific fixes (missing python3.12-dev headers; FlashInfer nvcc/CUDA_HOME crash). - scripts/vllm-serve.sh: tensor-parallel serve, env-var params, --foreground/--stop, native sampler + CUDA_HOME fixes, logs tee'd to a gitignored logs/ dir. - scripts/vllm-verify.sh: real chat-completion smoke test. - pyproject.toml + clients/hello_inference.py: uv-managed client test program (openai/httpx/tiktoken) for exercising the endpoint. - README.md: reference-node specs, full dependency stack, serving-config explanation (TP, continuous batching, PagedAttention, MoE, BF16), and inference examples. - .claude/skills/vllm-setup: /vllm-setup skill driving the whole flow. - CLAUDE.md: 76-char markdown wrap convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enable tool calling in vllm-serve.sh by default and wire up opencode as an agentic client against the self-hosted model. Verified end to end on the g6e.12xlarge: opencode's build agent driving Qwen3-Coder-30B via vLLM. - vllm-serve.sh: add TOOL_PARSER (default qwen3_coder) → --enable-auto-tool-choice --tool-call-parser. Agentic clients send tool_choice=auto, which vLLM rejects without this. TOOL_PARSER=none for a plain completion server; hermes for non-coder Qwen3 chat models. - opencode-setup.sh: idempotent installer — checks if opencode is already installed and skips if so; else installs and writes an OpenAI-compatible provider config pointing at the local vLLM endpoint. --launch / --check. - config/opencode.json: reference provider config. - --help/-h on all four scripts (install, serve, verify, opencode-setup), each listing env vars, defaults, and options. - README: full serve-parameter table mapping every env var to its vLLM flag, an explicit "spell out every parameter" reproducible command, a "which kernels are we using" table (FlashAttention 2 / Triton MoE / PagedAttention / native sampler / PYNCCL / CUDA graphs), and an opencode section. - CLAUDE.md: switch convention to NOT hard-wrapping markdown prose; unwrap the vLLM README and skill accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- pyproject.toml: matplotlib as an optional `dev` extra, for plotting throughput/cost results from the client-side probes. uv.lock regenerated to match. - .gitignore: ignore tmp/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dling - Per-model .md files in self-hosted/vllm/models/ with serve commands, parsers, and tuning notes - YaRN rope scaling (ROPE_SCALING env var) in vllm-serve.sh for extending context past native window - MAX_NUM_SEQS env var to cap concurrent sequences (required for hybrid Mamba models) - HF_HOME auto-provisioning to NVMe scratch (/opt/dlami/nvme) to avoid root disk overflow - .hf_token file auto-resolution (repo root, vllm dir, ~) with loud warning if missing - --enable-prefix-caching always enabled - Improved stop: now also kills VLLM::Worker and VLLM::EngineCore processes - Update README with model table, per-model links, rope scaling section
Document that changes must go through a feature branch + PR review rather than committing or merging directly to main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add git workflow rules to AGENTS.md
…ADMEs updated to 5x6 matrix - 30 artifact bundles (5 tasks x 6 models) now on disk - Qwen 3.6 35B averaged 69.70% across 5 tasks (self-hosted on g6e.12xlarge, 4x L40S) - Updated root README, benchmark README, and JUDGE_RESULTS.md with 6th model column - Added self-review step to /swe skill - Added vllm-setup skill updates and claude-local.sh script
GPT-5.5/GPT-5.4 on Bedrock's bedrock-mantle endpoint are Responses-API-only — they reject Chat Completions (used by every other model in this repo) with a 400. Routes them through /openai/v1/responses instead by setting model_info.mode: responses in litellm-config.yaml; no other model needs this since they all speak Chat Completions natively. Also updates model counts and tables in both READMEs and adds a CHANGELOG entry, since this repo tracks exact model/provider counts in several places (40 third-party, 47 total).
shekharprateek
added a commit
that referenced
this pull request
Jul 16, 2026
… #1134, #1303) Completes the 5-task × 5-model benchmark matrix against mcp-gateway-registry @ 1.24.4. All 5 models produced the 4 standard /swe artifacts (github-issue, lld, review, testing) for both new tasks: - Claude Opus 4.8 - Kimi K2.5 (substituted for K2-Thinking on these tasks due to backend reliability issues; tasks 1-3 still use K2-Thinking) - Mistral Devstral 2 123B - MiniMax M2.5 - Qwen Coder Next Also updates the /swe skill with a "Benchmark Isolation" constraint block: no reading sibling model artifacts, no claude-peers messaging during runs, only the cloned repo as code source. And tightens the task #4 README description to match the ECS-native (secrets block) phrasing that was used in the actual prompts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bedrock-mantleendpoint are Responses-API-only — they reject Chat Completions (400) the way every other model in this repo speaks it./openai/v1/responsesinstead by settingmodel_info.mode: responsesinconfig/litellm-config.yaml, using the sameMANTLE_API_KEY/region as the rest of the third-party models (no separate token or region needed as of the June 2026 us-east-1 GA).gpt-5.5/gpt-5.4entries to thescripts/claude-model.shpicker.Test plan
gpt-5.5andgpt-5.4respond correctly through the proxy via the Anthropic Messages API (curl localhost:4000/v1/messages)scripts/claude-model.sh --model gpt-5.5claude-model.sh,model_name:entries inlitellm-config.yaml, per-provider table sums in README) rather than by hand