Skip to content

fix(updater): harden upgrade restart and rollback#569

Closed
xiami762 wants to merge 3 commits into
devfrom
fix/daemon-restart-readiness
Closed

fix(updater): harden upgrade restart and rollback#569
xiami762 wants to merge 3 commits into
devfrom
fix/daemon-restart-readiness

Conversation

@xiami762

@xiami762 xiami762 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR hardens the self-update restart path for remotely bound and custom-host deployments. It keeps the existing upgrade architecture while fixing dependency-sync timeouts, service-config handoff, daemon shutdown/readiness races, source-only rollback behavior, and failure-state retention.

Key Changes

  • Updater dependency synchronization:
    • run uv sync --no-python-downloads without --frozen;
    • use a 300-second timeout on Linux, macOS, and Windows;
    • retry without the configured mirror after the first mirror timeout;
    • record TimeoutExpired.stdout and TimeoutExpired.stderr in structured logs.
  • Source backup and replacement:
    • keep backups source-only and continue excluding all dist directories;
    • preserve an existing webui/dist when the downloaded source archive does not contain one;
    • retain existing runtime/user directory preservation rules.
  • Restart handoff configuration:
    • serialize the complete ServiceConfig into the backend environment and handoff arguments;
    • support legacy handoff invocations that do not contain --service-config-json by rebuilding the config from the existing host/port and restart CLI arguments;
    • persist legacy backend host/port, migration hints, browser behavior, and frontend-build behavior in upgrade state.
  • Daemon lifecycle and readiness:
    • wait for both the supervisor control endpoint and the recorded daemon PID to stop;
    • require a real backend TCP listener before reporting startup success;
    • fail startup when the TCP listener never becomes ready;
    • prevent an old Unix supervisor from unlinking a replacement control socket by checking socket identity;
    • bypass environment proxy settings for local upgrade-page probes.
  • Rollback and recovery:
    • preserve rollback_failed state and its last error when recovery fails;
    • clean temporary upgrade-page/PID artifacts without deleting the retained failure state;
    • clear upgrade state only after a successful recovery or explicit cleanup.
  • Tests:
    • add regression coverage for mirror timeout fallback/output logging, legacy handoff compatibility, full config persistence, daemon PID shutdown, TCP readiness, Unix socket ownership, source-only backup behavior, dist preservation during source replacement, and rollback-state retention.

Impact Scope

  • User-visible behavior: Upgrade/restart output now distinguishes a spawned daemon process from a fully ready service. A startup whose backend TCP listener never becomes available reports an error instead of success.
  • Compatibility / migration: Backward-compatible with handoff commands generated by versions that predate the JSON config snapshot. No manual migration is required.
  • Configuration / environment: Adds the internal _FLOCKS_SERVICE_CONFIG environment snapshot. It is generated automatically; there are no new required user settings or secrets.
  • Dependencies: No new or upgraded third-party dependencies.
  • Performance / resources: Dependency sync may wait up to 300 seconds per attempt. A mirror timeout can trigger a fallback attempt against the default index. Startup performs an additional local TCP readiness probe.
  • Security / permissions: No authentication, authorization, secret-handling, or external permission changes. The serialized service config contains endpoint and lifecycle settings only.
  • Public APIs: No public API changes.
  • Frontend: No UI code changes. This PR preserves webui/dist during source replacement but does not change frontend-build atomicity.

Business Logic to Review

  • flocks.updater.updater._sync_project_dependencies
    • The first mirror timeout switches to a command without --default-index.
    • Every TimeoutExpired path logs normalized stdout/stderr.
  • flocks.updater.restart_handoff._prepare_upgrade_handover
    • New releases use the JSON snapshot.
    • Upgrades initiated by older running versions fall back to the host/port arguments already present in the legacy handoff command and do not query the supervisor for config.
  • flocks.updater.updater._prepare_upgrade_handover
    • Upgrade state must contain the complete service config so rollback/resume cannot reset custom server endpoints.
  • flocks.cli.service_manager._stop_all_unlocked and flocks.updater.restart_handoff._stop_supervisor_before_restart
    • Shutdown is complete only when both the control API and known daemon PID are gone.
    • If the daemon PID cannot be read, existing control-request error handling remains the safety boundary.
  • flocks.cli.service_manager._wait_for_supervisor_ready
    • A healthy/static supervisor payload is insufficient without a reachable local TCP listener.
  • flocks.updater.updater.resolve_upgrade_runtime_state
    • Temporary page artifacts are removed while rollback_failed diagnostics remain available across a normal start attempt.
  • flocks.cli.service_supervisor.SupervisorDaemon
    • Unix socket cleanup is ownership-safe; Windows continues using TCP control and does not enter the inode path.

Why This Approach

The change intentionally applies focused fixes to the existing updater/handoff/supervisor contracts instead of rewriting the upgrade state machine. It uses data already available in legacy handoff arguments for backward compatibility, keeps platform-specific lifecycle behavior isolated, and adds regression tests at each failure seam. Frontend build atomicity is explicitly outside this PR's scope.

Test Plan

  • .venv/bin/ruff check on all changed Python files
  • git diff --check
  • 225 selected updater, restart-handoff, and service-manager tests passed
  • Real Unix socket and process integration paths included in the selected suite
  • Windows-specific command, timeout, runtime-path, and managed-Python branches covered by platform-mocked tests
  • End-to-end upgrade/restart validation on a physical Windows host

One pre-existing stale test is excluded from the selected suite because it monkeypatches a helper that no longer exists: test_run_reports_pending_install_receipt_after_pro_bundle_tasks.

Compatibility, Migration & Rollback

  • No breaking API or configuration changes.
  • Existing custom host/IP and legacy server host/port settings are carried through the restart handoff and persisted in upgrade state.
  • Older handoff invocations without the new JSON option remain supported.
  • Backups remain source-only. Existing webui/dist is preserved when the incoming source archive omits it.
  • On upgrade-task failure, the updater restores the source backup and attempts to resume the previous service configuration.
  • If recovery still fails, rollback_failed and last_error are retained while temporary upgrade-page artifacts are cleaned.

@xiami762 xiami762 closed this Jul 20, 2026
@xiami762

Copy link
Copy Markdown
Contributor Author

reimplement in #571

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant