Skip to content

Add model discovery integration tests without managed config - #662

Open
andy-xu-db wants to merge 1 commit into
andy/model-discovery-integration-testsfrom
andy/model-discovery-integration-unmanaged
Open

andy-xu-db wants to merge 1 commit into
andy/model-discovery-integration-testsfrom
andy/model-discovery-integration-unmanaged

Conversation

@andy-xu-db

@andy-xu-db andy-xu-db commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add Tests-table Cases 13–24, where managed config does not exist, stacked on Add model discovery integration tests with managed config #655.
  • Cover 12 named tests and 20 configured/fresh executions across Claude Code and Codex.
  • Add only the remaining unmanaged discovery inputs, fixtures, and Claude-owned cache inspection support.
  • Keep hard observable assertions for provider- and location-scoped discovery; there are no skips, xfails, retries, or fallback expectations.

Cases

Cases Scope Executions
13–16 saved location and discovery-disabled native fallback 4
17–24 provider/location discovery and disabled fallback, configured + fresh 16

Each test function includes its Tests-table case number (test_case_13_... through test_case_24_...). Together with #655, the stack has 44 numbered discovery executions.

Managed workspace coverage inherited from #655

This branch is rebased on #655 head 76d42794a3061efe256f32dc3f5a6b5d7a644168. It therefore inherits the one-live-fetch-per-agent managed-config fixtures and their dedicated ca-central-1 MPS targets:

  • Claude Code: main.default.ci_e2e_anthropic_mps
  • Codex: main.default.ci_e2e_openai_mps

Both managed-config lanes pass on CI run 35284318193.

Live result

Run 35284318193 exercised Claude Code 2.1.268 and Codex 0.154.0 against the integration workspace.

  • Managed Cases 1–12 pass for both agents.
  • Unmanaged discovery-disabled Cases 15, 16, and 21–24 pass.
  • Codex Cases 14, 18, and 20 currently fail their exact scoped-catalog assertions: Codex exposes its native catalog instead of the single saved/provider/location model.
  • Claude Cases 13, 17, and 19 currently fail their exact scoped-picker assertions: Claude Code exposes its native picker and, where expected, does not create its gateway cache.

These failures are intentionally reported rather than weakened or skipped; they demonstrate the current client/catalog gaps the new tests cover. A separate Claude agent-launch shard also fails because system.ai.claude-opus-5 rejects Claude Code's advisor_20260301 tool type.

Stack

Verification

  • Head: 6fb6e7f9fb4ef6b25d15e771dee8d24eb806e6a6.
  • Unit-test rerun 105415208862 passed; the original first-prompt socket failure did not recur.
  • tests/test_integration_contract.py: 5 passed.
  • This PR adds 20 numbered executions; cumulative discovery collection is 44; complete integration collection is 99.
  • Ruff, formatting, and diff checks passed.
  • No local Databricks profile was selected.

@andy-xu-db andy-xu-db changed the title Add unmanaged model discovery integration tests Add model discovery integration tests without managed config Sep 16, 2026
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from 5204d14 to 9751ad8 Compare September 16, 2026 15:02
@andy-xu-db
andy-xu-db changed the base branch from main to codex/model-discovery-test-stack September 16, 2026 15:02
@andy-xu-db
andy-xu-db force-pushed the codex/model-discovery-test-stack branch from d85847c to 2c0b3df Compare September 16, 2026 18:32
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from 9751ad8 to 88b4d6f Compare September 16, 2026 18:32
@andy-xu-db
andy-xu-db changed the base branch from codex/model-discovery-test-stack to andy/model-discovery-integration-tests September 16, 2026 20:06
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch 3 times, most recently from e907761 to e4a7d5c Compare September 17, 2026 20:03
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-tests branch from 8cc5eb8 to ecd7c40 Compare September 17, 2026 21:37
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from e4a7d5c to 5404b69 Compare September 17, 2026 21:37
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from 5404b69 to debbbfb Compare September 17, 2026 22:47
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from debbbfb to 6fb6e7f Compare September 17, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants