Arcane Projects CLI (arcane_projects.py)

A small, dependency-free Python tool for inspecting and cleaning up Arcane Docker-management projects (Compose "stacks") over its REST API.

Why this exists: it started as a way to diagnose and fix a runaway bug where ~800 dead Dozzle-N projects kept reappearing in Arcane, then grew into a general-purpose housekeeping tool. The full root-cause story and a recovery playbook are in INCIDENT.md.

TL;DR

python3 arcane_menu.py                      # interactive menu — easiest way to use it
python3 arcane_projects.py list             # or use the CLI directly: list projects
python3 arcane_projects.py remove           # preview a cleanup (dry-run)
python3 arcane_projects.py remove --execute # actually remove stopped/archived

Two ways to use it

1. Interactive menu (arcane_menu.py) — run python3 arcane_menu.py and pick numbered options. No flags to remember; destructive actions preview first and require you to type yes. It just calls arcane_projects.py under the hood, so behaviour is identical. Easiest for occasional use.

   On launch it prompts you to paste your API key (hidden input) unless one is already provided via arcane_config.json or ARCANE_API_KEY. A prompted key is held in memory for the session only — never written to disk — and is gone when you quit. The menu's session is authoritative: it injects its own url/key/env into each command and strips inherited ARCANE_* vars, so a stale shell export can't shadow it. Change settings any time with "Set / change API key & connection", or snap back to the file with "Reload from config file (ignore env vars)". The header shows a masked status like key=set (••••5678). Quit with q (or 0 / Ctrl-C).

2. Direct CLI (arcane_projects.py) — full control via subcommands and flags, documented below. Better for scripting and precise filtering.

---

Contents

• Requirements
• Configuration
• Safety model
• Command reference
  • Inspection
  • Removal
  • GitOps
  • Project lifecycle
  • Reports
  • Housekeeping
  • Diagnostics
• How it works
• Housekeeping notes & gotchas
• Incident write-up & recovery playbook → INCIDENT.md

---

Requirements

• Python 3.8+ (standard library only — no pip install needed).
• Network access to the Arcane host.
• An Arcane API key (Settings → API in the Arcane UI).

The script talks only to Arcane's REST API. It never touches Docker or the host filesystem directly — so anything it does can also be undone or inspected from the Arcane UI.

---

Configuration

Three settings, resolved in this order (first match wins): command-line flag → environment variable → config file → built-in default.

Setting	Flag	Env var	Config-file key
Arcane base URL	--url	ARCANE_URL	url
API key	--key	ARCANE_API_KEY	key
Environment id	--env	ARCANE_ENV_ID	env

Config file (arcane_config.json)

The easiest place to keep your settings is arcane_config.json, which lives next to the scripts and is plain JSON:

{
  "url": "http://192.168.10.101:3552",
  "env": "0",
  "key": ""
}

• Edit it by hand, or use the menu's "Set / change API key & connection" option, which can write it for you (and asks whether to include the key).
• Run python3 arcane_projects.py config to see what's currently resolved and from where.
• When the menu saves it, the file is chmod 600 and a .gitignore entry is added so a saved key can't be committed. The shipped .gitignore already excludes arcane_config.json, arcane_openapi.json, and *.db.bak.
• Leaving "key": "" is fine — the menu will prompt for the key per session, or you can use the ARCANE_API_KEY env var. Storing a real key in the file is convenient but plaintext; the menu's session-prompt is the more secure option.

Env vars vs config: an exported ARCANE_API_KEY (or ARCANE_URL/ARCANE_ENV_ID) takes precedence over the config file — a common gotcha if you exported a now-deleted key earlier in the same terminal (you'll get 401 invalid API key). The menu warns at startup when the env var differs from the file, and its "Reload from config file (ignore env vars)" option switches to the file's values without you having to unset anything in the shell. From a plain CLI, just unset ARCANE_API_KEY.

Security: prefer the menu's per-session key prompt or the ARCANE_API_KEY env var over storing a key in arcane_config.json, and never commit a real key to git (the .gitignore guards against this). Delete temp keys from Arcane (Settings → API) when done.

Find your environment id with python3 arcane_projects.py envs if it isn't 0.

Color output: both tools print color-coded, aligned output (green = running, amber = stopped, blue = archived; ✓/✗ markers on actions). Color turns off automatically when output is piped/redirected or when NO_COLOR is set, and you can force it off with --no-color.

---

Safety model

• Nothing destructive runs without --execute. Every removal/pause/resume/delete command defaults to a dry run that prints exactly what it would do.
• Removals are validated. After destroying a project the script re-fetches it and reports ✓ REMOVED (HTTP 404) or ✗ NOT removed so you know it actually worked.
• Soft failures are surfaced. Some endpoints return HTTP 200 with success:false (e.g. prune); those print the API's ✗ ... errors rather than a misleading success.
• destroy keeps your data by default. It removes containers/networks but leaves named volumes and on-disk compose files unless you pass --remove-volumes / --remove-files. (See gotchas — this matters.)

---

Command reference

Global flags (--url, --key, --env) work with every command.

Inspection commands

Command	What it does	Why you'd run it
envs	Lists Docker environments and their ids.	Find your --env id if it isn't 0.
list [--archived]	Lists all projects (paginated) with status, sorted, plus a totals line. --archived lists the archived set instead.	Your day-to-day "what's in Arcane and what state is it in" view.
get --id <id>	Fetches one project and prints its raw JSON.	Confirm a single project's status/fields.
raw	Dumps the raw JSON of the first projects page.	Verify the API response shape if something looks off.
config	Shows the resolved url/env/key and where each came from (flag/env/file/default) + the config file path. No key required.	"Why is it connecting to X / using which key?"

python3 arcane_projects.py list
python3 arcane_projects.py list --archived
python3 arcane_projects.py get --id e77d233e-0730-4f7b-b577-f0836c3ffaf0

Removal commands

Command	What it does	Why you'd run it
remove-one --id <id> [--archived] [--remove-volumes] [--remove-files] [--execute]	Removes one project, then validates it's gone.	Safely test removal on a single project before a bulk run.
remove [--status …] [--name-contains …] [--name-regex …] [--archived] [--remove-volumes] [--remove-files] [--execute] [--delay N]	Bulk-removes every project matching the filters, validating each, with a running [n/total] progress log and a final tally.	Mass-cleanup of dead/stopped/archived projects.

Filters for remove:

• --status — comma-separated statuses to target (default stopped,archived).
• --name-contains — only projects whose name contains this substring.
• --name-regex — only projects whose name matches this regex (e.g. '^Dozzle-\d+$').
• --archived — operate on the archived set (?archived=true) instead of the normal list.
• --remove-volumes / --remove-files — also delete named volumes / on-disk compose files.
• --delay — seconds between removals (default 0.5).

# Preview removing all stopped/archived (no changes):
python3 arcane_projects.py remove

# Remove only the Dozzle-N test stacks:
python3 arcane_projects.py remove --name-regex '^Dozzle-\d+$' --execute

# Test one archived project first, then sweep the archived set:
python3 arcane_projects.py remove-one --archived --id <ID> --execute
python3 arcane_projects.py remove --archived --execute

GitOps commands

GitOps syncs are the usual reason projects get auto-created/redeployed, so most "mystery project" problems are diagnosed here.

Command	What it does	Why you'd run it
gitops	Lists all GitOps syncs: auto-sync on/off, interval, last status, name, bound project, id. Flags any sync with a last-sync error.	See what's syncing, how often, and whether anything is failing.
gitops-inspect --id <id>	Dumps a sync's full config, status, and the repo file tree it pulls.	Diagnose why a sync misbehaves (bad composePath, unbound project, etc.).
gitops-pause [--id <id>] [--all] [--execute]	Sets autoSync=false (reversible). Default targets all auto-sync-on syncs.	Stop the bleed before cleaning up auto-created projects.
gitops-resume [--id <id>] [--all] [--interval N] [--execute]	Sets autoSync=true, optionally a new interval (minutes). Default targets all paused syncs.	Turn syncs back on after cleanup; optionally slow them down.
gitops-delete --id <id> [--execute]	Deletes a sync by id.	Remove a rogue/duplicate sync.
gitops-sync --id <id> [--execute]	Triggers one sync now (pulls repo + applies). Guarded.	Test a sync on demand instead of waiting for its interval.
env-sync [--execute]	Runs every enabled sync in the environment now. Guarded.	Force a full re-sync.

python3 arcane_projects.py gitops
python3 arcane_projects.py gitops-pause --all --execute        # stop auto-sync
python3 arcane_projects.py gitops-resume --all --interval 60 --execute  # back on, hourly

Project lifecycle commands

Manage a stack directly (by --id or --name). down, restart, redeploy, and archive are impactful, so the menu confirms them; the CLI runs them immediately (like docker compose ...).

Command	What it does
up --id|--name <p>	Deploy / start the project.
down --id|--name <p>	Stop it (keeps containers).
restart --id|--name <p>	Restart its containers.
redeploy --id|--name <p>	Recreate it (pull + up).
pull --id|--name <p>	Pull latest images.
archive / unarchive --id|--name <p>	Move it in/out of the archived set.

python3 arcane_projects.py restart --name CodeServer
python3 arcane_projects.py pull --name NPM

Reports

Command	What it does	Why you'd run it
updates	Image-update summary (how many images have newer versions).	Quick "what needs updating" check.
counts	Resource counts across projects/containers/images/networks/volumes.	At-a-glance inventory.
info	Key Docker daemon facts (version, OS, CPU/mem, container/image counts).	Confirm what the daemon sees.

Housekeeping

Command	What it does	Why you'd run it
prune [--images] [--networks] [--build-cache] [--containers] [--volumes] [--mode all|dangling|olderThan] [--until 24h] [--execute]	Prunes the selected unused resources. Dry-run unless --execute. Each resource is sent as {mode, until?}; --mode applies to all selected resources and defaults to all (remove all unused). Images also accept dangling (untagged only) and olderThan (with --until, e.g. 24h); until isn't valid for volumes. The menu always uses all. --volumes can cause data loss and is flagged. Soft failures (success:false) print the API's per-resource errors.	Reclaim disk space.

python3 arcane_projects.py prune --images --networks --build-cache          # preview (mode=all)
python3 arcane_projects.py prune --images --mode dangling --execute         # dangling images only
python3 arcane_projects.py prune --images --networks --build-cache --execute # do it

Because --mode applies to every selected resource, use image-specific modes (dangling / olderThan) with --images on its own.

Diagnostic commands

Command	What it does	Why you'd run it
routes [--grep <substr>]	Finds Arcane's OpenAPI spec and prints the real API routes (optionally filtered).	Discover the correct endpoint for your Arcane version (routes differ between releases).
events [--grep <substr>] [--limit N]	Reads the environment event log; prints a sample raw event then recent events with timestamp, type, and actor.	Find out what or who created a resource (a user, the System scheduler, a GitOps sync, an API token). This is the single most useful debugging command.
jobs	Lists Arcane's scheduled jobs and the job-schedule cron config.	Check whether a scheduled job (auto-heal, auto-update, gitops sync interval, etc.) is involved.
probe-archived	Probes several query-param/route variants to find how your Arcane exposes archived projects.	One-time discovery (we found it's ?archived=true).

python3 arcane_projects.py events --grep dozzle   # who created the Dozzle projects?
python3 arcane_projects.py routes --grep gitops
python3 arcane_projects.py jobs

---

How it works

• Auth: every request sends the X-Api-Key header.
• Projects = Compose stacks. Listed under GET /api/environments/{env}/projects, which is paginated — the script loops with limit/start offsets to pull all projects (the API caps a single page at ~100, which is why a naive list only ever showed the first 100).
• Removal uses DELETE /api/environments/{env}/projects/{id}/destroy. Archived projects are fetched with ?archived=true and, if a destroy is rejected, the script unarchives then retries.
• Status: a project is running (all services up), stopped (none up), or archived. Archived projects come back as a separate set, not merged into the normal list.
• Resilience: responses are unwrapped from Arcane's {success, data} envelope, and field lookups try several key names so the tool tolerates minor version differences.

---

Housekeeping notes & gotchas

For the full root-cause story behind these and a step-by-step recovery playbook, see INCIDENT.md.

• Disk is the source of truth. Arcane re-imports projects from …/persistent-volumes/arcane/projects on boot. To remove a project permanently, either use --remove-files on destroy, or delete its folder on disk. A DB-only delete will reappear after a restart.
• Use ls -la when checking that folder — staging artifacts are hidden dotfiles (.gitops-sync-stage-*).
• Permissions: some on-disk project files are written by the container as root, so a host rm may hit "Permission denied." Use sudo, or let Arcane delete them via remove-one --remove-files (Arcane runs as root inside its container).
• *.gitops-backup-* projects are full snapshots Arcane takes before applying a sync change — they can contain real service data (volumes, certs), not just compose files. They're only created on an actual change, not every cycle. Safe to prune, but know what's in them first.
• API key: keep it in arcane_config.json (git-ignored), the menu's session prompt, or ARCANE_API_KEY — not hard-coded in the script. Delete temp keys from Arcane when finished. Remember env vars override the config file (see the Configuration note).
• Routes differ by Arcane version — if a command 404s, run routes --grep <thing> to find the correct path for your build.