Skip to content

system/nxpkg: Add network sync and harden package lifecycle. - #3642

Open
aviralgarg05 wants to merge 1 commit into
apache:masterfrom
aviralgarg05:gsoc/nxpkg-sync-lifecycle-hardening-pr5
Open

system/nxpkg: Add network sync and harden package lifecycle.#3642
aviralgarg05 wants to merge 1 commit into
apache:masterfrom
aviralgarg05:gsoc/nxpkg-sync-lifecycle-hardening-pr5

Conversation

@aviralgarg05

@aviralgarg05 aviralgarg05 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Note: Please adhere to Contributing Guidelines.

Summary

Complete the network and package-lifecycle work started by #3474.

This adds:

  • catalog synchronization and artifact downloads through
    netutils/webclient;
  • SHA-256 verification of downloaded artifacts;
  • sync, available, update, remove, and rollback commands;
  • failure cleanup and stale-lock reclamation;
  • serialized catalog and installed-database updates, with live-owner records
    that do not depend on filesystem timestamp accuracy;
  • validation of repository-provided paths, sizes, installed metadata, and
    version fields;
  • an optional package icon and installed-manifest loading for graphical
    frontends; and
  • support for custom boards where CONFIG_ARCH_BOARD is not defined.

The synchronized catalog records its source in the same atomically replaced
file. This keeps relative artifact resolution consistent with the catalog
under concurrent synchronization. The previous repo.url sidecar is still
read for migration from development images.

User documentation is in apache/nuttx#18875. #3643 is the graphical frontend
that depends on the APIs added here.

Impact

  • New feature: YES. nxpkg gains repository synchronization and the
    install/update/remove/rollback lifecycle described above.
  • User adaptation: YES when persistence is required. The default root is
    /var/lib/nxpkg; boards must mount persistent storage there or override
    CONFIG_SYSTEM_NXPKG_ROOT.
  • Build: YES only for configurations enabling the new network operations;
    they use the selected netutils/webclient HTTP/TLS facilities.
  • Hardware: NO architecture or driver change.
  • Documentation: YES; provided by Documentation: Update nxpkg and add nxstore guide. nuttx#18875.
  • Security: YES. SHA-256 detects artifact corruption or substitution relative
    to the downloaded catalog, but is not package signing. Plain HTTP does not
    authenticate the repository.
  • Compatibility: Existing configurations with CONFIG_SYSTEM_NXPKG disabled
    are unaffected. The legacy repo.url sidecar remains readable for
    migration.
  • Other: system/nxstore: Add LVGL touchscreen app-store frontend. #3643 is stacked on this PR and must be rebased when this branch is
    rewritten.

Testing

Build host:

  • macOS 26.5, arm64
  • xtensa-esp-elf-gcc 14.2.0
    (esp-14.2.0_20251107)

Target:

  • Xtensa / ESP32-S3
  • Waveshare ESP32-S3-Touch-LCD-7 custom configuration

Verification:

  • nxstyle on all changed C sources
  • git diff --check
  • all system/nxpkg sources cross-compiled for ESP32-S3
  • all system/nxpkg sources compiled in the simulator configuration
  • the complete ESP32-S3 firmware linked and was flashed successfully
  • the board loaded and parsed all eight manifests from the cached SD catalog
  • an offline SD-card repository completed sync, install, update, reset
    persistence, rollback, remove, and relative-artifact execution
  • two concurrent 968,304-byte installs preserved the live package lock and
    committed exactly one install
  • a live synchronization lock was retained and the contender returned
    -EBUSY
  • dead-task and previous-boot lock records were reclaimed
  • a bad SHA-256 install left no database entry, version directory, lock, or
    transaction file

Testing logs before change:

N/A: the synchronization and complete lifecycle commands are new in this PR;
there is no equivalent operation on the PR base to reproduce.

Testing logs after change:

These are the nxpkg diagnostic lines observed on the target, grouped by
check. They are not one continuous console session: the checks were run
separately.

Catalog load and parse:

nxpkg: info: loading index from /mnt/sdcard/nxpkg/index.jsn
nxpkg: info: index read complete (3595 bytes)
nxpkg: info: cJSON_Parse returned success
nxpkg: info: parsed manifest nxdoom 1

Install, update, rollback and remove:

nxpkg: info: synced package index from /mnt/sdcard/nxpkg/audit/v1.jsn
nxpkg: info: installed auditpkg version 1
nxpkg: info: synced package index from /mnt/sdcard/nxpkg/audit/v2.jsn
nxpkg: info: installed auditpkg version 2
auditpkg current=2 previous=1 type=elf arch=xtensa compat=esp32s3-touch-lcd-7 versions=1,2
nxpkg: info: rolled back auditpkg to version 1
nxpkg: info: removed auditpkg

Lock contention and reclaim:

nxpkg: error: unable to acquire package lock for 'locktest': -16
nxpkg: error: unable to acquire sync lock: -16
nxpkg: error: reclaiming lock from exited task 999
nxpkg: error: reclaiming lock from an earlier boot

Checksum rejection, which left behind no database entry, version directory,
lock or transaction file:

nxpkg: error: sha256 mismatch: -84

The lifecycle transcript above was captured against a repository served from
the SD card, which exercises the same pkg_sync()/pkg_install() paths that
the HTTP source feeds. The HTTP transport itself has been exercised on this
board through the companion store frontend, which synchronizes its catalog
over Wi-Fi at boot; I can attach a dedicated nxpkg sync http://... transcript
on request.

PR verification Self-Check

  • This PR introduces only one functional change.
  • I have updated all required description fields above.
  • I have reviewed and signed every rewritten commit.
  • I have attached complete before/after real-hardware runtime logs.
  • This PR adheres to the current contribution and coding guidelines.
  • My PR is still work in progress.
  • My PR is ready for review and can be safely merged.

Comment thread system/nxpkg/Kconfig Outdated
Comment thread system/nxpkg/README.txt Outdated
Comment thread system/nxpkg/pkg_install.c
Comment thread system/nxpkg/pkg_repo.c Outdated
Comment thread system/nxpkg/pkg_repo.c Outdated
Comment thread system/nxpkg/pkg_store.c Outdated
Comment thread system/nxpkg/pkg_store.c Outdated
@aviralgarg05
aviralgarg05 force-pushed the gsoc/nxpkg-sync-lifecycle-hardening-pr5 branch from c4f639c to f363a8f Compare July 24, 2026 10:01
@aviralgarg05 aviralgarg05 changed the title system/nxpkg: network sync, install hardening, and CLI completion system/nxpkg: Add network sync and harden package lifecycle. Jul 24, 2026
Complete the package manager beyond the initial local-only slice: fetch a
catalog and artifacts over HTTP, verify every payload against the SHA-256
recorded in its manifest, and wire update, remove, rollback and available
into the CLI alongside install.

Make the on-disk state safe to interrupt. Installs stage into temporary
paths and reclaim state left behind by a previous interrupted attempt.
The installed-packages database is committed before the current/previous
pointer files are refreshed and before any payload directory is removed,
so a crash can leave reclaimable orphaned files but never leaves the
database pointing at a payload that is already gone.

Serialize concurrent operations with three lock kinds: a per-package
install lock, the installed-database lock and a synchronization lock that
covers the whole read-fetch-write sequence, so a catalog and the source
URL it was fetched from are always committed as a pair. Each lock records
a per-boot token and its owner PID, and a contender honors a lock whose
owner task is still alive rather than trusting file modification times;
targets whose mounted filesystem clock does not track CLOCK_REALTIME would
otherwise mistake a just-created lock for a decades-old stale one. Locks
are renewed as data actually arrives, so a slow download is not reclaimed
while it is still progressing.

Treat catalog and database contents as untrusted: reject path-traversal
sequences in manifest and installed-entry name/version fields before they
reach the filesystem, require that the current and previous versions
appear in the recorded version list, apply explicit size limits, and
report a zero-byte write as an error instead of looping on it.

Compare versions by numeric prefix without overflowing, falling back to a
lexical comparison of the remainder so that versions sharing a numeric
prefix are not treated as identical.

Add an optional manifest icon field and pkg_metadata_load_manifest_path()
for a GUI frontend that needs the manifest recorded for one specific
installed version, which can differ from the catalog's current entry after
a rollback. Default the storage root to /var/lib/nxpkg, overridable via
CONFIG_SYSTEM_NXPKG_ROOT.

Fix a build break on custom boards: pkg_runtime_compat() referenced
CONFIG_ARCH_BOARD, which has no default under ARCH_BOARD_CUSTOM and is
therefore undefined rather than empty; fall back to
CONFIG_ARCH_BOARD_CUSTOM_NAME. Route diagnostics through syslog, since
stdio is not visible to a supervisor with no attached console.

Assisted-by: OpenAI Codex:gpt-5.6-sol
Signed-off-by: aviralgarg05 <gargaviral99@gmail.com>
@aviralgarg05
aviralgarg05 force-pushed the gsoc/nxpkg-sync-lifecycle-hardening-pr5 branch from f363a8f to 45a9292 Compare August 6, 2026 16:35
@aviralgarg05

Copy link
Copy Markdown
Contributor Author

Rebased onto current master (2aebae740) and squashed to a single commit.

The squash also removes an artefact of the previous history: the first commit added system/nxpkg/README.txt and a later commit removed it again after review, which made tools/checkpatch.sh fail on the intermediate commit while trying to resolve a path that no longer exists. All checks pass on the squashed commit.

The diff is limited to system/nxpkg/. Keeping this as a draft until I attach the full HTTP lifecycle transcript (sync, install, update, rollback, remove, and checksum rejection) from the ESP32-S3 target.

@aviralgarg05
aviralgarg05 marked this pull request as ready for review August 7, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants