Skip to content

Optimize sync for repositories with specific tag lists - #2475

Open
parmstro wants to merge 1 commit into
pulp:2.26from
parmstro:upstream-pr-bypass-optimization
Open

Optimize sync for repositories with specific tag lists#2475
parmstro wants to merge 1 commit into
pulp:2.26from
parmstro:upstream-pr-bypass-optimization

Conversation

@parmstro

Copy link
Copy Markdown

Add bypass optimization to skip expensive /tags/list pagination when syncing container repositories with specific (non-wildcard) tag lists.

When include_tags contains only specific references without wildcards, and exclude_tags is empty or contains only harmless patterns that won't match the includes (e.g., *-source), the sync bypasses /tags/list enumeration entirely and fetches manifests directly.

Cosign companion tags (signatures, attestations, SBOMs) are discovered via concurrent HEAD request probing instead of full tag enumeration, maintaining security artifact discovery while avoiding expensive pagination through deep tag lists.

Performance impact: Reduces sync time from 3-8 minutes to ~20 seconds for repositories with 50,000+ tags when syncing specific digest references.

Changes:

  • Add auto_discover_cosign field to ContainerRemote model (default True)
  • Implement _can_bypass_taglist() detection in ContainerFirstStage
  • Add _discover_cosign_companions_without_taglist() for HEAD probing
  • Add _tag_exists() helper for tag existence validation
  • Pass mirror parameter through synchronize() to ContainerFirstStage
  • Add database migration for auto_discover_cosign field
  • Add comprehensive unit test coverage

closes #2474
Assisted by: Claude Sonnet 4.5 (model ID: claude-sonnet-4-5@20250929)

@parmstro
parmstro force-pushed the upstream-pr-bypass-optimization branch 4 times, most recently from 5837e3f to 5889d3e Compare August 21, 2026 19:00
Add bypass optimization to skip expensive /tags/list pagination when
syncing container repositories with specific (non-wildcard) tag lists.

When include_tags contains only specific references without wildcards,
and exclude_tags is empty or contains only harmless patterns that won't
match the includes (e.g., *-source), the sync bypasses /tags/list
enumeration entirely and fetches manifests directly.

Cosign companion tags (signatures, attestations, SBOMs) are discovered
via concurrent HEAD request probing instead of full tag enumeration,
maintaining security artifact discovery while avoiding expensive
pagination through deep tag lists.

Performance impact: Reduces sync time from 3-8 minutes to ~20 seconds
for repositories with 50,000+ tags when syncing specific digest
references.

Changes:
- Add auto_discover_cosign field to ContainerRemote model (default True)
- Implement _can_bypass_taglist() detection in ContainerFirstStage
- Add _discover_cosign_companions_without_taglist() for HEAD probing
- Add _tag_exists() helper for tag existence validation
- Pass mirror parameter through synchronize() to ContainerFirstStage
- Add database migration for auto_discover_cosign field
- Add comprehensive unit test coverage

closes pulp#2474
@parmstro
parmstro force-pushed the upstream-pr-bypass-optimization branch from 5889d3e to 92978f0 Compare August 21, 2026 19:04
@gerrod3

gerrod3 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution, but we won't accept this as is.

We do not allow adding migrations in older branches. It prevents upgrading to a newer version.

From what I read it seems the slow part is the tag pagination fetching, did you try increasing the number of results for each tag request with ?n=. https://distribution.github.io/distribution/spec/api/#listing-image-tags I think if we provide a value, say 1000, then we could drastically reduce the time doing fetches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants