ci: run the Python SDK test suites on pull requests - #1418
Open
inahus99 wants to merge 2 commits into
Open
Conversation
The Python packages ship real test suites that nothing executes. `ci.yml` only covers TypeScript (Bun, turbo check-types, Biome), and the publish-*-python workflows publish to PyPI without running any tests, so these packages are released on the strength of local runs alone. Adds a path-filtered workflow covering the three packages whose suites currently pass: - agent-framework-python installs the package and runs pytest (54 tests) - cartesia-sdk-python and pipecat-sdk-python stub their heavy runtime dependencies at import time, so they run under unittest against the sources with nothing installed Python 3.10 is the lowest version these packages declare, so a change that only works on a newer interpreter fails here. `fail-fast` is disabled so one package failing still reports the others. openai-sdk-python is deliberately left out: its suite cannot be collected against current supermemory, which is issue supermemoryai#1235. It should be added here once that is fixed.
The package was left out because its suite could not be collected against the current supermemory release, which was issue supermemoryai#1235. That was fixed by supermemoryai#1236, which caps supermemory to <3.5, so the suite now runs green on main: 24 passed, 11 skipped, with the skips gated on SUPERMEMORY_API_KEY. python-dotenv is a dev-group dependency imported at module scope by both test modules, so it is installed alongside pytest rather than relying on the package dependencies alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1417.
What and why
The Python packages ship real test suites that nothing executes.
ci.ymlcovers only TypeScript (Bun,turbo run check-types, Biome), and the fourpublish-*-python.ymlworkflows publish to PyPI without running any tests. These packages are released on the strength of local runs alone.The cost of that is already on the record:
openai-sdk-pythonspent a month unimportable onmain— its suite could not be collected against thesupermemoryrelease of the day (ImportError: cannot import name 'MemoryAddResponse'). That is #1235, reported by an outside user on Jul 11 and fixed on Aug 12 by #1236. Any CI run of the tests that already existed would have caught it the day it broke.The workflow
Path-filtered, so it only fires on PRs touching these packages or the workflow itself. Matrix over all four packages:
agent-framework-pythonpip install -e . pytest pytest-asyncio, thenpytestopenai-sdk-pythonpip install -e . pytest pytest-asyncio python-dotenv, thenpytestcartesia-sdk-pythonPYTHONPATH=src python -m unittest, no installpipecat-sdk-pythonPYTHONPATH=src python -m unittest, no installcartesia-sdk-pythonandpipecat-sdk-pythonstub their heavy runtime dependencies (cartesia-line,pipecat-ai,loguru) at import time, so they need nothing installed and finish in well under a second. The two pytest suites pay an install cost.python-dotenvis called out explicitly becauseopenai-sdk-pythonimports it at module scope in both test modules, but declares it only in its dev dependency group, sopip install -e .alone leaves the suite uncollectable.Two deliberate choices:
requires-python = ">=3.10"for three of them,>=3.8.1foropenai-sdk-python, andagent-framework-coreagrees), so a change that only works on a newer interpreter fails here rather than at a user's install.fail-fast: false, so one package failing still reports the others instead of masking them.Verification
All four commands run green against current
main:agent-framework-pythonopenai-sdk-pythoncartesia-sdk-pythonpipecat-sdk-pythonThe 11 skips in
openai-sdk-pythonare gated onSUPERMEMORY_API_KEYand skip in CI too — that suite's real coverage in this workflow is the 24.This workflow is in its own path filter, so it will run against itself on this PR — but as a fork PR from a first-time contributor it needs a maintainer to approve workflow runs before that can happen. Until someone does, the results above are local runs, and the checks on this PR are only Graphite and Socket.
One note on scope: this only wires up the suites that exist.
cartesia-sdk-pythonandpipecat-sdk-pythonhave a single test each, so a green check on those means very little coverage today. Getting them running is the prerequisite for that being worth improving.