Skip to content

Add lock-driven skill key rotation - #6662

Open
samuv wants to merge 7 commits into
mainfrom
samuv/skills-key-reanchor
Open

samuv wants to merge 7 commits into
mainfrom
samuv/skills-key-reanchor

Conversation

@samuv

@samuv samuv commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Projects that pin a cosign public key cannot follow a publisher key rotation through the lock-driven workflow. This adds an explicit re-anchor path while preserving fail-closed behavior and the existing trust policy when no replacement key is supplied.
  • Add --public-key support to skill upgrade and sync adoption, require --allow-signer-change for replacement anchors, and try the recorded key before a proposed replacement.
  • Make project lock updates transactional and compare-and-swap protected, including bounded rollback for trust-only updates.
  • Use strict Sigstore bundle retrieval for explicit replacement-key re-anchors and update the CLI, API, SDK, architecture documentation, and regression coverage.

Part of #6640

Type of change

  • Bug fix
  • New feature
  • Refactoring (no behavior change)
  • Dependency update
  • Documentation
  • Other (describe):

Test plan

  • Unit tests (task test)
  • E2E tests (task test-e2e)
  • Linting (task lint-fix)
  • Manual testing (describe below)

task gen, task docs, and task build pass. Changed-package unit tests pass. The full task test run reaches five unrelated pkg/api failures because no container runtime is available. The focused key-trust E2E suite compiles and reaches setup, where it stops for the same missing runtime.

Changes

Area Change
Skill CLI and API Accept and validate replacement public keys for upgrade and sync adoption.
Skill service Re-anchor only after successful verification, preserve the old key when it still verifies, and update trust state transactionally.
Verification Add strict OCI bundle retrieval for explicit replacement-key decisions.
Generated reference Regenerate CLI, OpenAPI, and Go SDK outputs.
Tests and architecture Cover rotation, adoption, rollback, concurrency, and documented trust semantics.

Does this introduce a user-facing change?

Yes. thv skill upgrade --allow-signer-change --public-key <PUBLIC_KEY_PATH> can re-anchor a key-pinned OCI skill, and thv skill sync --adopt --public-key <PUBLIC_KEY_PATH> can adopt an installed key-signed skill.

Implementation plan

Approved implementation plan
  1. Extend skill CLI and API options with an explicitly supplied replacement public key.
  2. Keep lock-first policy selection, try the recorded key first, and consider the replacement only after a conclusive mismatch.
  3. Require strict OCI bundle snapshots only for explicit replacement-key re-anchors while retaining legacy verification for ordinary upgrades.
  4. Protect lock and installed-state changes with project transactions, compare-and-swap writes, and bounded rollback.
  5. Verify sync adoption offline against the supplied key and record the verified anchor.
  6. Add unit and E2E coverage, then regenerate CLI and API documentation.

Special notes for reviewers

This is the first PR in a two-PR stack. The plugin parity change follows in #6663. Strict bundle retrieval depends on stacklok/toolhive-core#297. The approved PR-size exception keeps the complete skills workflow and its generated outputs together while splitting plugin parity into the second PR.

@samuv
samuv added this pull request to stack #6664 September 15, 2026 13:17
@samuv samuv changed the title samuv/skills key reanchor Add lock-driven skill key rotation Sep 15, 2026
@samuv samuv mentioned this pull request Sep 15, 2026
10 tasks
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 15, 2026
@codecov

codecov Bot commented Sep 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 79.48244% with 111 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.20%. Comparing base (59ba54d) to head (95adeb3).

Files with missing lines Patch % Lines
pkg/skills/skillsvc/upgrade.go 74.36% 71 Missing ⚠️
pkg/skills/verifier/oci.go 14.28% 18 Missing ⚠️
pkg/skills/skillsvc/sync.go 85.54% 12 Missing ⚠️
pkg/skills/skillsvc/service.go 78.26% 5 Missing ⚠️
pkg/skills/lockfile/lockfile.go 87.50% 1 Missing ⚠️
pkg/skills/skillsvc/install.go 97.77% 1 Missing ⚠️
pkg/skills/skillsvc/install_extraction.go 85.71% 1 Missing ⚠️
pkg/skills/skillsvc/install_git.go 88.88% 1 Missing ⚠️
pkg/skills/skillsvc/project_tx.go 91.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6662      +/-   ##
==========================================
+ Coverage   79.14%   79.20%   +0.05%     
==========================================
  Files         789      790       +1     
  Lines       79021    79411     +390     
==========================================
+ Hits        62545    62894     +349     
- Misses      16471    16512      +41     
  Partials        5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 15, 2026

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 560a05b, with separate passes over trust decisions/adoption, project transactions and rollback, CLI/API/SDK integration, standards, and reuse.

Spec: the local re-anchor flow follows #6640: old-key-first, lock-first policy selection, explicit replacement authorization, shared strict evidence, and offline adoption against the installed digest. No separate spec mismatch found.
Standards: no separate merge-blocking convention findings.
Domain: one cancellation/rollback correctness issue in the newly added same-digest metadata refresh path needs fixing (inline). A candidate finding about transitive entries becoming explicit was compared with the base and discarded as pre-existing.

Dependency: strict evidence safety remains blocked by the two findings posted on stacklok/toolhive-core#297; please consume the corrected core revision before merging this integration.

Nonblocking API documentation: please describe public_key as base64-encoded DER SPKI in the sync/upgrade request fields, including adopt/check and allow_signer_change constraints, so generated SDK/OpenAPI consumers do not have to infer the wire format from internal options.

All 46 current CI checks passed. No local tests or linters were run.

Comment on lines +60 to +61
updated := buildInstalledSkill(opts, scope, clientTypes, existing.Clients)
if err := s.store.Update(ctx, updated); err != nil {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve managed state and make metadata-refresh compensation cancellation-safe

This new same-digest/reference-change path builds a fresh record without copying existing.Managed, so the first store update writes Managed=false. If cancellation lands after that update and the lock CAS but before recordLockState writes Managed=true, the latter update fails. rollbackInstall then tries to restore the previous row using the same cancelled context; the SQLite store cannot start that transaction, while the context-free lock rollback can still restore the old pin. The failed upgrade leaves an unmanaged installed row paired with a live lock entry. A subsequent uninstall skips lock removal (existing.Managed is false), so sync can reinstall the supposedly removed skill.

Before this PR, equal digests returned UpToDate without this metadata mutation. Please preserve the existing managed state in both this branch and applyGitInstallExisting, and run rollback compensation with a bounded cancellation-detached context, as the trust-only rollback already does. A cancellation regression should assert the failed refresh restores both row and lock, then verify uninstall removes the lock entry.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2a73313 and updated again in 987021e for the released core dependency. The same-digest and Git refresh paths now preserve Managed, and rollback uses a bounded five-second context detached from request cancellation. The regression cancels after the metadata write and lock CAS, verifies both the prior database row and lock entry are restored, then uninstalls and confirms the managed lock entry is removed. The full pkg/skills/skillsvc suite passes with -race against toolhive-core v0.0.49.

@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 16, 2026

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the update at d655077. The exact tree delta from 560a05b is only go.mod/go.sum, moving to core e5c9df0. I re-reviewed and approved stacklok/toolhive-core#297, so the upstream completeness concerns in my previous review are cleared.

The local metadata-refresh cancellation/rollback finding remains unchanged: the implementation blobs are identical to the reviewed head. My request-changes disposition therefore remains for that P2 only; the public_key wire-format documentation suggestion is nonblocking. No local tests or linters were run.

@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 16, 2026
@samuv
samuv requested review from blkt and jerm-dro as code owners September 16, 2026 16:17
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 16, 2026
@samuv
samuv force-pushed the samuv/skills-key-reanchor branch from 987021e to db62269 Compare September 16, 2026 16:38
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 16, 2026
Signed-off-by: Samuele Verzi <samu@stacklok.com>
Signed-off-by: Samuele Verzi <samu@stacklok.com>
Signed-off-by: Samuele Verzi <samu@stacklok.com>
Signed-off-by: Samuele Verzi <samu@stacklok.com>
Signed-off-by: Samuele Verzi <samu@stacklok.com>
Signed-off-by: Samuele Verzi <samu@stacklok.com>
Signed-off-by: Samuele Verzi <samu@stacklok.com>
@samuv
samuv force-pushed the samuv/skills-key-reanchor branch from db62269 to 95adeb3 Compare September 17, 2026 16:14
@samuv samuv self-assigned this Sep 17, 2026
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL Extra large PR: 1000+ lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants