Skip to content

Support source archives in rosdistro_additional_recipes.yaml - #132

Open
diegoferigo-rai wants to merge 1 commit into
RoboStack:masterfrom
diegoferigo-rai:diegoferigo/archive-sources
Open

Support source archives in rosdistro_additional_recipes.yaml#132
diegoferigo-rai wants to merge 1 commit into
RoboStack:masterfrom
diegoferigo-rai:diegoferigo/archive-sources

Conversation

@diegoferigo-rai

Copy link
Copy Markdown
Contributor

Today every entry of rosdistro_additional_recipes.yaml has to be a git repository. The manifest is fetched from the raw file endpoint of github.com or gitlab.com, and anything else is refused at this line with Cannot handle unknown repository hoster. On the generation side, generate_source() always emits a git/rev source.

I have a vendor ROS driver that is not published in any rosdistro and does not live in a git repository at all: it is shipped as a plain .zip on a server that requires authentication. There is no way to layer it on a distro at the moment, even though rattler-build handles a url/sha256 source natively.

This PR accepts an entry whose url points at a source archive and carries a sha256 instead of a rev or a tag:

my_vendor_driver:
  url: https://artifacts.example.com/my_vendor_driver-0.5.0.zip
  sha256: 0f3d9c1b6a4e5d2f8b7c0a9e1d3f5b7c9a1e3d5f7b9c1a3e5d7f9b1c3a5e7d9f
  version: 0.5.0
  additional_folder: src/my_vendor_driver

The package.xml is read directly out of the downloaded archive, and the generated source block becomes url/sha256. A missing sha256 is a hard error, since a URL source without a checksum would silently be fetched unverified.

The subtle part is additional_folder. When rattler-build unpacks a url archive it strips a single common top-level directory, and the generated build script resolves additional_folder relative to that stripped root. So the manifest lookup inside the archive strips the same single common root, otherwise one config key would mean two different things on the two sides. This is what _strip_common_root() and _resolve_member() do, and both zip and tar are handled, including tars whose members are prefixed with ./.

Such an archive usually sits behind authentication, so the two download helpers now go through requests, which is already a dependency and already used in vinca/utils.py. That way netrc, proxy and CA settings are resolved from the environment, and the Authorization header is dropped when a redirect crosses to another host, which is what an artifact server does when it hands out a presigned storage URL. I deliberately did not add any credential handling to vinca itself: the existing GITHUB_TOKEN and GITLAB_TOKEN support from #128 is untouched, and everything else is left to requests.

.tar.zst is intentionally left out of the accepted suffixes, because tarfile only reads zstd from Python 3.14 on while requires-python is still >=3.9.

The git path is unchanged: for a non-archive entry, get_released_repo() and the generated source block produce exactly what they produced before.

Tested

  • pytest on the repo: 113 passed, including 24 new tests covering the archive helpers, the root stripping for zip and tar, the missing member and the missing checksum cases, and the redirect behaviour that is relied upon.
  • ruff check and ruff format --check are clean.
  • End to end against a real private server: generated the recipe for the driver above and built it with rattler-build for both linux-64 and linux-aarch64. I also checked that the download fails without credentials, so the delegation to requests is what makes it work.

Previously an entry of rosdistro_additional_recipes.yaml was always
assumed to be a git repository: its package.xml was read from the raw
file endpoint of github.com or gitlab.com, and generate_source() emitted
a git/rev source. Any other URL was rejected with "Cannot handle unknown
repository hoster". A ROS package distributed as a plain source archive,
for instance a vendor driver hosted on an Artifactory server, could
therefore not be layered on a distro at all.

Accept an entry whose url points at a source archive (.zip, .tar.gz and
the other common suffixes) and carries a sha256 checksum instead of a
rev or a tag. Its package.xml is read out of the downloaded archive,
after dropping a single common top-level directory so that
additional_folder means the same thing for this lookup and for the
generated build script, and the source block becomes url/sha256. A
missing checksum is a hard error, since a URL source without one would
silently be fetched unverified.

Archives can live behind authentication, so the downloads now go through
requests, which is already a dependency. It resolves the ambient
credentials, proxy and CA settings on its own, and it drops the
Authorization header when a redirect crosses to another host, which is
what an Artifactory server does when it hands out a presigned storage
URL. Vinca therefore keeps no credential handling of its own beyond the
existing GITHUB_TOKEN and GITLAB_TOKEN support.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@diegoferigo-rai
diegoferigo-rai marked this pull request as ready for review August 6, 2026 16:37
@diegoferigo-rai diegoferigo-rai changed the title Support source archives in rosdistro_additional_recipes.yaml Support source archives in rosdistro_additional_recipes.yaml Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants