Release 2.0.0-rc3 published no PyPI wheel: stale poetry.lock broke the publish job, but the release still looks successful

[ling-senpeng13]

Summary

Release 2.0.0-rc3 reports success but publishes nothing installable. The GitHub release exists and carries its e2e bundle, but conductor-python 2.0.0rc3 was never uploaded to PyPI, so anything pinning that version fails at install time.

$ pip install "conductor-python[agents]==2.0.0-rc3"
ERROR: Could not find a version that satisfies the requirement conductor-python==2.0.0-rc3
       (from versions: …, 2.0.0rc1, 2.0.0rc2)
ERROR: No matching distribution found for conductor-python==2.0.0-rc3

PyPI currently has only 2.0.0rc1, 2.0.0rc2.

Root cause

jsonschema was added to pyproject.toml without regenerating poetry.lock.

file	2.0.0-rc2	2.0.0-rc3
pyproject.toml	blob 8e949e27	blob 3c2d2232 — +jsonschema = ">=4.23" under [tool.poetry.group.dev.dependencies]
poetry.lock	blob 6607f404	blob 6607f404 — unchanged

Poetry's stored content-hash (f3c1edfc…) was computed from rc2's pyproject.toml, so the release build fails its consistency check:

#15 [python_test_base 4/6] RUN poetry config virtualenvs.create false && poetry install --only main --no-root …
#15 1.586 pyproject.toml changed significantly since poetry.lock was last generated. Run `poetry lock` to fix the lock file.
#15 ERROR: process "/bin/sh -c poetry config virtualenvs.create false && poetry install …" did not complete successfully: exit code: 1

Failing run: Continuous Delivery → publish-release-at-pypi → step 3 Publish — https://github.com/conductor-oss/python-sdk/actions/runs/30473030818

Note this is a bookkeeping mismatch, not an unresolvable dependency. jsonschema 4.26.0 is already in the lock (satisfying >=4.23), just recorded as an optional extras-gated transitive dep rather than a dev-group one. So poetry lock should touch only content-hash plus that package's groups/optional/markers — no version churn.

Why it escaped

1. The signal fired pre-merge and was merged red. On 19f20bcc (the commit adding jsonschema), CI was already failing:

failure   build-and-push          ← runs the same `poetry install` that later broke the release
failure   integration-test (core)
success   unit-test

2. There is no explicit lock-freshness gate. No workflow in the repo runs poetry check --lock, poetry lock --check, or --locked, so staleness only surfaces indirectly as a Docker build failure whose message is buried in build output.

3. The release process cannot detect the contradiction. Both release jobs trigger on release: published with no dependency between them:

release: published ─┬─→ Continuous Delivery       (PyPI publish)    ✗ failed
                    └─→ Release Agent E2E Bundle  (attach tarball)  ✓ succeeded

So the bundle was attached to a release whose package does not exist, and the release page still looks complete. This is the worst part of the bug: the failure is silent to consumers, who see a normal release and a normal asset.

Impact

Anything pinning 2.0.0-rc3 cannot install. Concretely, the released e2e bundle conductor-ai-e2e-python-2.0.0-rc3.tar.gz pins conductor-python[agents]==2.0.0-rc3 in its requirements.txt, so the bundle is unusable by every downstream consumer — including the orkes-conductor SDK-e2e lane (orkes-io/orkes-conductor#3863), where it fails the gating python job.

Immediate fix

poetry lock            # or `poetry lock --no-update` on poetry 1.x
git commit -am "chore: regenerate poetry.lock for the jsonschema dev dependency"

Then cut 2.0.0-rc4. Re-running rc3's Continuous Delivery does not help — it rebuilds the 2.0.0-rc3 tag, which still has the stale lock committed in it.

Reviewer check: confirm the lock diff is only content-hash + jsonschema metadata. If unrelated versions move, poetry is resolving fresh rather than respecting existing pins.

Proposed hardening (the release process should make this impossible)

1. Validate before publishing. Run poetry check --lock and poetry build as the first release step, so a stale lock blocks the release instead of failing it halfway with an opaque message.
2. Don't attach artifacts until the package is live. Gate the e2e-bundle upload on conductor-python==<version> appearing on PyPI, and fail rather than upload if it never does. A bundle pinning a nonexistent package is worse than no bundle, because downstream cannot tell the difference. (This pattern is implemented for csharp-sdk in Export the agent e2e suite as a release bundle for downstream repos csharp-sdk#166 — the upload polls the NuGet flat container and refuses to attach otherwise. It ports directly to the PyPI JSON API. javascript-sdk has the same gap.)
3. A failed publish must not leave a green-looking release. Cut as prerelease/draft and promote only after the publish succeeds, so "release exists" implies "package is installable."
4. Add poetry check --lock to PR CI and make build-and-push a required check, so the next stale lock is caught on the PR that causes it.

[ling-senpeng13]

Correction to the "Why it escaped" section above — the gap is worse than I wrote.

I said build-and-push was red on 19f20bcc and merged anyway, implying the signal was available at PR time. It wasn't. build-and-push lives in harness-image.yml, not the PR CI workflow, and it triggers only on:

on:
  push:
    branches: [main]
  release:
  workflow_dispatch:

There is no pull_request trigger. So its red runs on 19f20bcc / b581a5e3 were post-merge, on main — the PR itself never had that check. Nothing available at review time could have caught the stale lock.

That makes proposal 4 the important one, not a nice-to-have: nothing in PR CI exercises poetry install, so a stale lock cannot be detected before merge today. poetry check --lock is one step, needs no server or network beyond the install, and reproduces this exact failure — verified locally with poetry 2.4.1: exit 1 before #457, exit 0 after.

Suggested minimal addition to the PR CI workflow:

      - name: Verify poetry.lock is in sync with pyproject.toml
        run: poetry check --lock

Also worth noting the post-merge signal was itself ignored: build-and-push went red on main at 19f20bcc on 07-27 and stayed red through the 07-29 release, which is the window where this could still have been caught cheaply.