Skip to content

Migrate all references to the newer avocado.yaml structure (ENG-2182) - #23

Open
nicksinas wants to merge 9 commits into
mainfrom
nsinas-eng-2182
Open

Migrate all references to the newer avocado.yaml structure (ENG-2182)#23
nicksinas wants to merge 9 commits into
mainfrom
nsinas-eng-2182

Conversation

@nicksinas

Copy link
Copy Markdown
Collaborator

What

Migrates all 30 references to the current avocado.yaml structure (ENG-2182).

Per-file changes:

  • Remove the deprecated cli_requirement key
  • Point the SDK image at docker.io/avocadolinux/sdk:{{ config.distro.release }} (drop the -channel suffix; use the config.* namespace)
  • Move the root password out of the config/config-dev confext into a top-level permissions profile
  • Add top-level rootfs and initramfs keys referencing that profile
  • Drop the now-empty config/config-dev extension and its runtime references

Overlays, packages, services, on_merge/on_unmerge, docker_images, SDK args, and all real extensions are untouched. iphone-travel-router keeps its non-empty avocado password (Cockpit/PAM). icam-540 additionally retargets to jetson-orin-nx (its underlying SoM, default_target_board: icam-540) so it resolves in the feed.

Also included:

  • .gitignore: repo-wide rules so per-build state is never committed (avocado.lock, .avocado-desktop.json, .avocado/, .avocado-state) — previously only 1 of 30 references ignored these.
  • scripts/build-check-references.sh: a reference-by-reference build harness (cleanunlockinstallbuild→teardown-to-clean-git-state), with a live markdown report and PASS/FAIL summary.

Build verification

Every reference was run through install + build via the new harness. Each migrated file also validates against the avocado schema (ok:true).

24 / 30 build clean. The remaining failures were investigated and are not caused by this migration — they fail either before the migrated sections are consumed (feed fetch) or inside code compilation (host toolchain), and 24 references using the identical migration pattern build green:

Reference Cause Category
astra-1680-deepx avocado-ext-dev missing from feed for grinn-astra-1680-sbc feed gap
imx8mp-npu-nnstreamer avocado-ext-dev missing from feed for ucm-imx8m-plus feed gap
imx8mp-npu-pose avocado-ext-dev missing from feed for ucm-imx8m-plus feed gap
linux-custom-kernel HOSTCC = target cross-compiler → host tool can't run when cross-building x86 kernel on an arm64 host (builds on x86_64 hosts) cross-arch env
rubicon upstream avocado-ext-cli package build exits non-zero after compiling + installing the CLI upstream package

c-gpio initially failed on a transient clock-skew flake and passes on a clean rebuild. zephyr-imx8mp-evk was still building at PR time (Zephyr pulls dozens of module repos).

Follow-ups for maintainers (not blocking this PR): publish avocado-ext-dev for grinn-astra-1680-sbc and ucm-imx8m-plus to 2024/edge; investigate the avocado-ext-cli package build (CLI 1.0.0-rc.1).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Migrates reference avocado.yaml files to the newer schema structure (removing deprecated keys, updating SDK image templating, and moving root-password configuration into top-level permissions profiles), and adds repo-wide tooling/ignores to support repeatable reference build verification without committing build state.

Changes:

  • Update reference avocado.yaml files to use permissions + rootfs/initramfs and remove deprecated cli_requirement and config/config-dev confext usage.
  • Standardize SDK image to docker.io/avocadolinux/sdk:{{ config.distro.release }}.
  • Add a reference-by-reference build verification harness and expand .gitignore to exclude Avocado-generated state.

Reviewed changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
zephyr-imx8mp-evk/avocado.yaml Removes deprecated keys and introduces top-level permissions + rootfs/initramfs; updates SDK image template.
webkit-ui/avocado.yaml Moves root password config into permissions and updates SDK image template; removes config-dev runtime reference.
shell-heartbeat/avocado.yaml Drops config confext usage, adds permissions + rootfs/initramfs, updates SDK image template.
scripts/build-check-references.sh New harness to clean/unlock/install/build each reference and produce a live markdown report.
rust-vitals/avocado.yaml Removes config-dev confext and adds permissions + rootfs/initramfs; updates SDK image template.
rubicon/avocado.yaml Removes config-dev confext and adds permissions + rootfs/initramfs; updates SDK image template.
ros2-ufactory-lite6/avocado.yaml Removes deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
react-dashboard/avocado.yaml Removes config-dev runtime refs across targets; adds permissions + rootfs/initramfs; updates SDK image template.
qemu-quickstart/avocado.yaml Drops config runtime/confext; adds permissions + rootfs/initramfs; updates SDK image template.
python-yolo/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
python-whisper/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
python-multiversion-uv/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
python-mqtt/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
python-flask/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
pi-metrics-exporter/avocado.yaml Removes config-dev confext; adds permissions + rootfs/initramfs; updates SDK image template.
nvidia-gstreamer-yolo/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
nvidia-deepstream/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
nodejs-dashboard/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
linux-custom-kernel/avocado.yaml Removes config-dev runtime/confext; adds permissions + rootfs/initramfs; updates SDK image template.
java-hello/avocado.yaml Removes config-dev confext; adds permissions + rootfs/initramfs; updates SDK image template.
iphone-travel-router/avocado.yaml Moves root password config into permissions while preserving non-empty password; updates SDK image template.
imx8mp-npu-pose/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
imx8mp-npu-nnstreamer/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
icam-540/avocado.yaml Updates default target/board mapping and migrates to permissions + rootfs/initramfs; updates SDK image template.
elixir-phoenix/avocado.yaml Removes config-dev runtime/confext and a now-empty confext; adds permissions + rootfs/initramfs; updates SDK image template.
docker-save/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
docker-registry/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
dev/avocado.yaml Removes config-dev confext; adds permissions + rootfs/initramfs; updates SDK image template.
cpp-tui-dashboard/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
c-gpio/avocado.yaml Drops deprecated keys/config confext; adds permissions + rootfs/initramfs; updates SDK image template.
astra-1680-deepx/avocado.yaml Migrates to permissions + rootfs/initramfs; updates SDK image template; also removes a local distro repo override.
.gitignore Adds repo-wide ignores for Avocado lock/state files.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread scripts/build-check-references.sh
Comment thread astra-1680-deepx/avocado.yaml

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 37 out of 38 changed files in this pull request and generated 1 comment.

Comment thread scripts/build-check-references.sh Outdated

@jetm jetm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the avocado.yaml schema migration. It's clean and consistent across all references: no dangling config/config-dev extension references, the SDK image tag change (dropping the channel) is applied uniformly, and every reference got the new permissions: block. One minor cross-file consistency nit inline on icam-540.

Comment thread icam-540/avocado.yaml
@@ -1,6 +1,8 @@
default_target: icam-540
default_target: jetson-orin-nx

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

icam-540/README.md frontmatter target is now stale. This renames the target to jetson-orin-nx (with icam-540 moved to default_target_board), but icam-540/README.md:4 frontmatter still lists `targets:

  • icam-540. Every other reference README uses the target slug (matching its default_target), so a catalog keyed on target slug files this one under icam-540, which is no longer a valid target. Suggest updating the README frontmatter to jetson-orin-nx`.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a new change on the backend. This particular target is now used as documented this reference. A docs update will accompany this. This is apart of the bigger change towards target and target_board. This is already documented in docs in the supported hardware table, but a more detailed page will be added to docs regarding this differentiation.

Comment thread c-gpio/avocado.yaml
Comment thread dev/avocado.yaml
Comment thread icam-540/avocado.yaml
Comment thread scripts/build-check-references.sh Outdated
Update all 30 references to the current avocado.yaml layout:
- Remove the deprecated cli_requirement key
- Point the SDK image at docker.io/avocadolinux/sdk:{{ config.distro.release }}
  (drop the -channel suffix; use the config.* namespace)
- Move the root password out of the config/config-dev confext into a
  top-level permissions profile
- Add top-level rootfs and initramfs keys referencing that profile
- Drop the now-empty config/config-dev extension and its runtime references

icam-540 additionally retargets to jetson-orin-nx (its underlying SoM,
with default_target_board: icam-540), which resolves in the package feed.
Add top-level .gitignore rules so per-build generated files are never
committed from any reference:
- avocado.lock
- .avocado-desktop.json
- .avocado/
- .avocado-state

Previously only 1 of 30 references ignored these locally.
Reference-by-reference build check: for each reference runs
clean/unlock/install/build, then tears down to a clean git state and
strips build artifacts. Writes a live markdown report after each step
(scripts/build-report.md), continues past failures, and prints a
PASS/FAIL summary.
- elixir-phoenix: restore the avocado-ext-connect-config extension that
  `avocado connect clean` stripped from avocado.yaml during a verification
  run (unintended side effect, not part of the migration). Verified the
  reference still builds with it in place.
- build-check-references.sh: snapshot avocado.yaml per reference and restore
  it after teardown so `avocado connect clean` can no longer leak edits into
  tracked config; fix the REPORT_FILE default doc to match the code
  (scripts/build-report.md).
The reference shipped without the spec-required docs. Add a frontmatter
README (language / targets / topics) and a full getting_started guide
(prerequisites → install → build → deploy → verify → customize), sourced
from the reference's avocado.yaml and files/ scripts.
Bring the three remaining references into compliance with the docs spec
(spec_readme.md / spec_getting_started.md):
- imx8mp-npu-nnstreamer, imx8mp-npu-pose: prepend README frontmatter
  (language / targets / topics); README bodies unchanged.
- zephyr-imx8mp-evk: prepend README frontmatter and add the missing
  getting_started.md.

All references now pass the spec (frontmatter README + getting_started.md).
Teardown previously ran `git clean -fdx`, which deletes ALL untracked files
under a reference (including a developer's local work-in-progress). Switch to
`-X` (ignored-only): build state is fully gitignored so it's still removed,
while untracked non-ignored files are left in place and flagged by the
residual-changes check. Addresses PR review feedback.
- getting_started.md (dev, rubicon, webkit-ui, qemu-quickstart): drop the
  `- config` / `- config-dev` line from the "add/remove extensions" examples.
  The migration deleted those extensions, so the copy-paste blocks pointed at
  a non-existent extension.
- build-check-references.sh: the terminal summary piped the loop to `tee`,
  running it in a subshell so the pass/fail counters were lost (always printed
  PASS: 0 FAIL: 0). Redirect to the summary file and `cat` it, keeping the
  counters in the parent shell. The markdown report totals were already correct.

@jetm jetm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed cold at 7cf4106. The restructure is mechanically clean, and I want to say that first because most of what follows is edge cases: all 30 avocado.yaml files parse, all 30 carry rootfs: + initramfs: + a matching permissions.dev profile, every runtime extension name resolves, there are no config/config-dev/cli_requirement leftovers in YAML, no duplicate top-level keys, and the SDK image line is byte-identical across all 30 and matches the CLI's own scaffold (avocado-cli/configs/default.yaml:93). default_target_board is correct for CLI 0.37.0+ and keeps avocado-bsp-{{ avocado.target.board }} resolving to avocado-bsp-icam-540. The summary-counter fix in 7cf4106 is genuinely correct - a redirected for ... done > file does not subshell in bash, which I checked rather than assumed.

Five findings are inline. Separately, I independently landed on the two issues already raised in open threads here, and traced up some extra evidence worth having on each:

On the cli_requirement removal thread. Agreed, and it is the most serious thing in the PR. Two details that sharpen it. cli_requirement is not deprecated - it is enforced on every config load (avocado-cli/src/utils/config.rs:1415, called at :1732 and :3441), and the CLI's own avocado init template still emits cli_requirement: ">=0.41.0" (configs/default.yaml:1, asserted by src/commands/init.rs:1149). So if the PR body describes the key as deprecated, that premise is wrong. And the breakage is actually two distinct windows, not one: on CLI 0.26-0.35 the new blocks are silently dropped (no deny_unknown_fields anywhere in config.rs at any version), the build reports success, and the image ships root:*: in /etc/shadow so nobody can log in - but on 0.36-0.40.2 it is a hard error instead, because permissions is missing from the rootfs singleton whitelist at that version (git show 0.40.2:src/utils/config.rs:148-159), so rootfs: {permissions: dev} takes the named-map branch and dies with invalid type: string ... expected struct ImageConfig. Cryptic, but at least loud. Restoring cli_requirement: ">=0.41.0" across all 30 covers both.

On the icam-540 retarget thread. Agreed on the README/spec mismatch. Two corrections to the rationale, though, if it is going in the commit message: the icam-540 package feed does still resolve (2024/edge/target/icam-540/repodata/repomd.xml returns 200, revision 2026-05-01), it is just about two months staler than jetson-orin-nx at 2026-07-08 - so "stale" is accurate and "does not resolve" would not be. And I could not confirm avocado-bsp-icam-540 is installable against the jetson-orin-nx feed; no avocado-bsp-* packages showed up in any feed I could enumerate, so that one wants an actual avocado install against the reference rather than a reasoned argument.

Three more findings that I could not anchor inline, two of them because the files are not in this diff:

  • docker-registry/getting_started.md:78 and nvidia-deepstream/getting_started.md:99 still say "the default config extension sets an empty root password for development". 7cf4106 swept the YAML-snippet form of this out of the dev, qemu-quickstart, rubicon, and webkit-ui docs but missed the prose form in these two. A reader greps avocado.yaml for config, finds nothing, and cannot work out where the empty root password comes from. astra-1680-deepx/getting_started.md:93 has the wording you want.
  • docker-save/getting_started.md:221 and docker-registry/getting_started.md:367 both still run docker.io/avocadolinux/sdk:2024-edge while their own avocado.yaml now resolves to docker.io/avocadolinux/sdk:2024. The config change is right (it matches configs/default.yaml:93, and docs/.../0.41.0.md:94 documents dropping the channel suffix from the generated tag; distro.channel stays load-bearing because it feeds the dnf $releasever). Both tags currently resolve to the same digest, so nothing breaks - the docs are just stale.
  • Worth one sentence somewhere: moving the root password out of a confext and into the rootfs (avocado-cli/src/commands/rootfs/image.rs:309-322 edits $ROOTFS_WORK/etc in place pre-mkfs) means changing that password is no longer a ~KB confext delta shipped via avocado deploy - it is now part of the full .aos OS bundle. Still OTA-updatable, just a much bigger payload. No reference's getting-started doc mentions this, and the "Customize -> Permissions" bullets imply the old cheap path.

Traced and deliberately dropped, so nobody re-audits them: the iphone-travel-router password: "avocado" divergence (deliberate, documented in-file for Cockpit PAM); the removed distro.repo.url: http://localhost:8080 in astra-1680-deepx (that is a fix); the {{ avocado.distro.* }} to {{ config.distro.release }} switch (both namespaces are valid, and the config.* form fails loudly on a missing key, which is better); the dead-but-defined extensions deepx-app, htop, avocado-ext-connect (all pre-existing); and the redundancy between the new root .gitignore and the 28 per-reference ones that already carry .avocado/ and .avocado-state.

Comment thread .gitignore

# Avocado build state — generated per build, never commit these.
# Matches in every reference directory (patterns without a slash apply recursively).
avocado.lock

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitignoring avocado.lock contradicts the CLI's tracked-lockfile design. avocado-cli/src/commands/init.rs:851-853 says explicitly: "avocado.lock is intentionally NOT ignored - it's a tracked, committed pin file (like Cargo.lock)", and init.rs:889 writes the literal comment # Avocado scratch + state (avocado.lock is tracked) into every generated .gitignore. docs/src/docs-guides/lockfiles-and-build-stamps.md:17 sells those pins as the reason "two team members ... get byte-identical sysroots". scripts/build-check-references.sh:146 reinforces the ignore with rm -f avocado.lock.

Effect: no reference ever commits its pins. Two users running avocado init --reference python-yolo a month apart resolve different package versions off the edge feed and get different images, and a reference that built last week can silently stop building when an upstream package moves.

If this is a deliberate policy for a references repo (arguably reasonable - you may want references to track the feed), the comment above should say so. As written, "generated per build, never commit these" asserts the opposite of the CLI's documented contract, so it reads as a mistake rather than a decision.

Comment thread .gitignore
avocado.lock
.avocado-desktop.json
.avocado/
.avocado-state No newline at end of file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This .avocado-state rule is a no-op for the two files that actually violate it. docker-registry/.avocado-state and docker-save/.avocado-state are both tracked, and gitignore never affects tracked files. They currently hold another developer's local state: a source_path of /Users/nick/work/repos/avocado/references/docker-registry plus a docker volume UUID.

Root cause of the original leak: docker-registry/.gitignore:2 and docker-save/.gitignore:2 use .avocado-state/ (trailing slash, directory-only) while the other 28 references use the correct .avocado-state. And scripts/build-check-references.sh:151-155 explicitly restores both files with a checkout after teardown, cementing them.

So anyone who clones and builds docker-registry hands the CLI a state file naming a docker volume and a /Users/nick/... path that does not exist locally. Fixing this needs an index removal (--cached) on both files plus the trailing-slash fix in those two per-reference ignores; this root rule alone changes nothing for them.

---
language: C++
targets:
- grinn-astra-1680-sbc

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three new README frontmatter blocks use target ids the repo spec rejects. This one (grinn-astra-1680-sbc), plus imx8mp-npu-nnstreamer/README.md:4 and imx8mp-npu-pose/README.md:4 (both ucm-imx8m-plus). None of the three appears in spec_readme.md's "Valid target identifiers" table, and that spec's Validation item 3 requires targets to be "*" or a list of valid identifiers.

So the docs build either rejects these three references at parse time or silently drops them from the index. The fix is one of two things, and I can't tell which you intend: add the three boards to the spec table (they're real hardware - they're the supported_targets in these same files), or the frontmatter is wrong. The fourth new block, zephyr-imx8mp-evk/README.md, uses imx8mp-evk and is fine.

out="$(avocado config show 2>/dev/null)"
dt="$(printf '%s\n' "$out" | sed -n 's/^[[:space:]]*default_target:[[:space:]]*//p' | tr -d '"' | head -1)"
if [ -n "$dt" ] && [ "$dt" != "*" ]; then printf '%s' "$dt"; return; fi
st="$(printf '%s\n' "$out" | sed -n 's/.*supported_targets:[[:space:]]*\[\([^]]*\)\].*/\1/p' \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolve_target's supported_targets fallback never matches, so it silently builds qemuarm64. This sed only matches a YAML flow sequence (supported_targets: [a, b]), but every reference declares a block list, so st comes back empty and line 126 falls through to the hardcoded qemuarm64.

Concretely this fires for ros2-ufactory-lite6, whose default_target is "*" and whose supported targets are imx8mp-evk, imx91-frdm, imx93-evk, imx93-frdm, raspberrypi5 - none of them qemuarm64. The sweep then either reports FAIL for a target the reference never claimed, or PASS on one that proves nothing. Silent either way: the report's Target column shows qemuarm64 with no indication it was a guess.

#
# Environment:
# LOG_DIR directory for per-reference logs (default: a fresh mktemp dir)
# REPORT_FILE markdown report path (default: $REPO_ROOT/scripts/build-report.md,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two smaller issues in this script's reporting. First, this header calls $REPO_ROOT/scripts/build-report.md "the tracked master report", but no such file is tracked at head and it isn't gitignored either. A default run therefore leaves an untracked file that the -fdX clean at line 150 won't remove and the per-reference residual check (line 209, scoped -- "$ref") won't flag, so the user is told the tree is clean when it isn't.

Second, line 209's grep -vE '^ M .*/avocado.yaml$' only matches unstaged modifications. Porcelain emits M (staged) when the edits are added to the index, so a maintainer who stages the config edits before running the sweep gets "residual git changes" on all 30 references and quickly learns to ignore the warning.

jetson-trt landed on main after the ENG-2182 migration and still carried
`cli_requirement: ">=0.41.0"`. Remove it so no reference pins a CLI floor
(consistent with the other 30). The reference was otherwise already in the
new structure (top-level permissions/rootfs/initramfs, config.* SDK image).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants