diff --git a/plans/2026-08-06_review-findings-host-http.md b/plans/2026-08-06_review-findings-host-http.md deleted file mode 100644 index 79f6b28c..00000000 --- a/plans/2026-08-06_review-findings-host-http.md +++ /dev/null @@ -1,51 +0,0 @@ -# RustScript Host/HTTP Review Findings - -**Review baseline:** `origin/master` (`ddfe64025456dc87dce4c7c7f79311a0b1c47a5a`) → `0f28ee9930890daffae5aa3c0bb70daf9c123997`, plus the current working-tree changes in `pd-host-function`. - -**Review mode:** read-only review by `gpt-5.6-sol` with high reasoning. No tests or files were run/changed by the reviewer. The main agent owns the follow-up implementation. - -## Blocking findings and directions - -1. **SSRF DNS check is detached from the actual connection** - - `src/builtins/runtime/http.rs:359-464` (`validate_url`, `execute_request`). - - `ToSocketAddrs` validates one resolution, while Reqwest resolves again during `send()`. DNS rebinding and IPv4-mapped IPv6 can bypass the policy. - - Use one validated resolver result for the connection; repeat resolve/validate/pin for every redirect; normalize mapped IPv6 and special-use ranges. - -2. **HTTP cancellation leaves blocking request threads alive** - - `src/builtins/runtime/http.rs:102-154,449-504`. - - Cancellation removes the receiver but does not stop `send()`/`read_to_end()` work. - - Use cancellable async tasks or explicit abort handles, enforce per-VM/process concurrency limits, and confirm task termination before dropping records. - -3. **Host capability binding is fail-open** - - `src/vm/host.rs:139-158,1795-1835`; `build.rs:997-1052`. - - Default registry construction and lazy fallback prevent an embedding from declaring an explicit host capability set. - - Add an explicit empty registry/capability profile, a no-default-fallback mode, and import preflight before side effects. - -4. **Blocking DNS runs before the host call becomes pending** - - `src/builtins/runtime/http.rs:193-216`. - - Keep synchronous validation structural; move DNS, IP policy checks, and connect into the cancellable task. - -5. **Empty port allowlist is interpreted as unrestricted** - - `HttpConfig::default`, `validate_url`. - - Fail closed or add an explicit unrestricted-port option. - -6. **Timeout and response representation need explicit contracts** - - Use one absolute deadline across redirects/DNS/connect/body. - - Preserve duplicate and byte-valued response headers instead of silently dropping them. - -7. **`pd_host_function` edge integration is coupled to downstream private layout** - - `pd-host-function/src/lib.rs`, `pd-host-function/src/edge.rs`. - - The macro currently emits `crate::abi_impl`, `::vm`, and `::linkme` paths. Introduce a stable support facade or separate edge extension boundary. - -8. **Macro diagnostics and public API compatibility** - - Infer execution type from `ItemFn.sig.asyncness` and return signatures rather than attribute shape. - - Validate generic arity and reject unsupported native async forms with direct `syn::Error` diagnostics. - - Assess the public `CallableDef` field addition and provide a compatibility constructor/API. - -## Implementation order - -1. Define the host macro contract: async from signature, scope only registration metadata. -2. Fix HTTP DNS/cancellation/timeout/port policy. -3. Add explicit host capability binding and remove implicit fallback where requested. -4. Stabilize the proc-macro support boundary and compile-fail coverage. -5. Run focused tests, then workspace gates. diff --git a/plans/2026-08-09_architecture-plan-index.md b/plans/2026-08-09_architecture-plan-index.md new file mode 100644 index 00000000..8c2f798d --- /dev/null +++ b/plans/2026-08-09_architecture-plan-index.md @@ -0,0 +1,88 @@ +# RustScript Architecture Plan Index + +**Goal:** Classify the current VM/compiler/runtime findings into independent implementation plans with explicit dependency order. + +**Architecture:** Correctness fixes are separated from structural refactors. Each plan owns one architectural surface and defines its own boundary and target criteria. Agent-owned behavior remains in `rustscript-agent` plans. + +**Tech Stack:** RustScript compiler, VM, host runtime, interpreter/JIT/AOT/no-std, agent integration contracts. + +--- + +## Classification + +| Category | Problem surface | Independent plan | Depends on | +| --- | --- | --- | --- | +| Wire/ABI | Count-derived builtin indices change existing call IDs | `2026-08-09_static-builtin-id.md` | none | +| Compiler correctness | UTF-8 rewrite, parent-path normalization, nested source diagnostics, public entry-point parity | `2026-08-09_nested-module-correctness.md` | none | +| Compiler architecture | Text rewrite, synthetic preludes, flat global symbols, basename identity, source ownership | `2026-08-09_semantic-module-system.md` | nested correctness | +| VM ownership | Monolithic VM mixes engine/program/instance/run/host state | `2026-08-09_vm-runtime-decomposition.md` | static IDs | +| Host lifecycle | Generic resource/operation code unused; IO/HTTP/SQLite duplicate lifecycle/cancellation | `2026-08-09_unified-host-lifecycle.md` | VM decomposition | +| Execution contract | Return/event ambiguity, buffered-only events, string errors, fragmented terminal state | `2026-08-09_run-outcome-event-error-contract.md` | RunContext; host lifecycle for final cancellation integration | +| Authorization | Builtin fast-path bypass, mutable identity/cache complexity, Edge macro leakage | `2026-08-09_capability-profile-host-binding.md` | static IDs | +| Async host transport | Core macro contains Edge scope knowledge; HTTP owns a synchronous scheduler; IO lacks feature-selected blocking/async bindings | `2026-08-09_http-transport-security-executor.md` | capability profile and host lifecycle | +| Structured concurrency | One waiting slot, no generic multi-operation/child-program supervisor | `2026-08-09_structured-task-supervisor.md` | VM decomposition, host lifecycle, capability profile | +| Backend architecture | Repeated semantics across interpreter/JIT/AOT/native/no-std | `2026-08-09_backend-semantic-convergence.md` | static IDs, VM decomposition | + +## Agent-owned plans + +| Category | Plan | +| --- | --- | +| Canonical product/framework roadmap | `rustscript-agent/plans/2026-07-30_rustscript-agent-gateway-api.md` | +| Run admission, structured input, result/events, timeout/cancellation, live delivery | `rustscript-agent/plans/2026-08-09_agent-run-lifecycle-events.md` | +| Transactional RSS storage, retention, restart, replay, idempotency | `rustscript-agent/plans/2026-08-09_agent-durable-state.md` | + +## Implementation route + +### Wave 0: Immediate correctness and identity + +Can run independently: + +1. Static builtin IDs. +2. Nested module correctness. + +Exit gate: static IDs no longer depend on catalog length; nested UTF-8/path/diagnostic regressions have executable coverage. + +### Wave 1: Foundational ownership and authorization + +Can run in parallel after relevant Wave 0 gates: + +1. VM runtime decomposition after static IDs. +2. Capability profile/host binding after static IDs. +3. Semantic module system after nested correctness. + +Exit gate: module semantics have explicit identities; VM has explicit ownership layers; every privileged call uses one authorization path. + +### Wave 2: Unified execution lifecycle + +1. Unified host resource/operation/cancellation lifecycle after VM ownership exists. +2. RunOutcome/event/error implementation on RunContext, integrating lifecycle cancellation as it becomes available. + +Exit gate: production host subsystems use one lifecycle, and every run has one structured terminal outcome with live bounded events. + +### Wave 3: Specialized consumers + +Can run in parallel after their dependencies: + +1. Generic host-driven async ABI, IO dual implementation, and async-only HTTP transport security. +2. Structured task supervisor. +3. Backend semantic convergence. +4. Agent run lifecycle and durable state integration. + +## Scope boundary + +- No plan adds compatibility decoding for pre-static-ID VMBC. +- No plan adds agent/provider/platform policy to `rustscript`. +- No agent plan defines VM internal implementation. +- Correctness plans do not wait for structural refactors. +- Structural plans remove superseded transitional paths after migration; they do not retain dual long-term architectures. +- New generic host functions require their own implementation plans; this index covers the architecture findings already identified. +- Async host futures are driven by the embedding host. VM, HTTP, and IO do not own a private executor or synchronous polling scheduler. +- Core host macros contain no pd-edge scopes, context types, registry generation, or downstream module paths. + +## Target criteria + +- Every finding from the architecture/current-worktree review maps to one owning plan. +- Cross-plan dependencies are explicit and acyclic. +- Each plan has implementation route, scope boundary, and target criteria. +- Core and agent ownership do not overlap. +- Obsolete review-finding and mixed historical plans are removed after their live requirements are represented here. diff --git a/plans/2026-08-09_backend-semantic-convergence.md b/plans/2026-08-09_backend-semantic-convergence.md new file mode 100644 index 00000000..9b280532 --- /dev/null +++ b/plans/2026-08-09_backend-semantic-convergence.md @@ -0,0 +1,136 @@ +# Backend Semantic Convergence Plan + +**Goal:** Reduce semantic duplication across interpreter, Trace JIT, AOT, native bridge, and no-std execution by introducing one canonical instruction/operation contract and differential verification. + +**Architecture:** Bytecode semantics, builtin signatures, ownership rules, traps, and deoptimization outcomes are defined once. Each backend lowers or interprets the same contract. Generated coverage tables and differential fixtures detect missing or divergent implementations. + +**Tech Stack:** Rust 2024, interpreter, Trace JIT, Cranelift AOT, native bridge, `pd-vm-nostd`, property/differential tests. + +--- + +## Independence and dependency + +- Independent of agent framework and module loading. +- Static builtin IDs should land first. +- VM decomposition should define Engine/Program ownership before large backend file moves. +- This plan does not block immediate correctness plans. + +## Scope boundary + +### In scope + +- One canonical semantic description for opcodes and builtins. +- Generated backend coverage checks. +- Shared ownership/trap/helper contracts. +- Differential interpreter/JIT/AOT/no-std tests. +- Incremental removal of duplicated lowering logic. + +### Out of scope + +- New optimization targets or benchmark promises. +- New bytecode opcodes solely to simplify one backend. +- A complete JIT rewrite in one milestone. +- Agent, HTTP, SQLite, or gateway behavior. + +## Implementation route + +### Milestone 1: Build a backend coverage inventory + +**Files:** +- Create backend coverage tests/tools under `tests/` or `src/backend/` +- Read interpreter, JIT recorder/lowerer, AOT IR/lowerer, no-std dispatch + +Generate a matrix for every opcode/builtin: + +```text +semantic definition +interpreter +trace recorder +JIT lowering +AOT lowering +no-std +fallback/deopt rule +``` + +Fail CI when a newly added operation lacks an explicit backend disposition. + +### Milestone 2: Define canonical operation semantics + +**Files:** +- Create: `src/semantics/` or equivalent +- Modify opcode/builtin metadata generation + +Represent: + +- operand/result types and stack effect; +- ownership/borrow/clone/drop behavior; +- trap/error conditions; +- side-effect and suspension classification; +- interpreter helper and native helper ABI; +- deopt/fallback permission. + +Keep explicit Rust implementation hooks where declarative metadata is insufficient. + +### Milestone 3: Generate shared dispatch metadata + +1. Generate interpreter validation/stack-effect tables. +2. Generate JIT/AOT eligibility and helper IDs. +3. Generate no-std support/fallback declarations. +4. Key builtins by static ID. +5. Reject mismatched arity/type/ownership metadata at build time. + +### Milestone 4: Consolidate native helper contracts + +**Files:** +- Modify native bridge/helper modules +- Modify JIT/AOT lowerers + +1. Define one helper ABI for tagged/scalar/heap operands. +2. Centralize owned temporary and Arc/raw-pointer rules. +3. Centralize trap/status routing. +4. Remove backend-specific reinterpretation of the same helper payload. + +### Milestone 5: Add differential execution harness + +For generated and curated programs, compare: + +- return value and structured error; +- side-effect/event sequence; +- ownership/drop counters where observable; +- fuel/deadline behavior; +- interpreter, JIT, AOT, and no-std supported subsets. + +Include arrays/maps/bytes, calls/closures, branches/loops, host-call boundaries, traps, and deopt cases. + +### Milestone 6: Migrate one semantic family at a time + +Recommended order: + +1. scalar arithmetic/comparison; +2. stack/local/frame operations; +3. collection access/mutation; +4. calls/closures; +5. builtin/native helper calls; +6. suspension/deopt/terminal outcomes. + +Each family removes superseded duplicate tables after differential parity passes. + +### Milestone 7: Verification + +```bash +cargo fmt --all -- --check +cargo test --locked --workspace --all-features +cargo test --locked -p pd-vm-nostd +cargo clippy --locked --workspace --all-targets --all-features -- -D warnings +git diff --check +``` + +## Target criteria + +- Every opcode and builtin has one canonical semantic entry. +- Every backend declares implement/fallback/unsupported explicitly. +- New operations cannot compile without a complete backend disposition. +- Interpreter/JIT/AOT/no-std differential fixtures agree on the supported subset. +- Native ownership and trap ABI is shared by JIT and AOT. +- Backend-specific large files lose duplicated semantic policy over incremental milestones. +- Performance changes are measured separately from semantic convergence. diff --git a/plans/2026-08-09_capability-profile-host-binding.md b/plans/2026-08-09_capability-profile-host-binding.md new file mode 100644 index 00000000..578bd203 --- /dev/null +++ b/plans/2026-08-09_capability-profile-host-binding.md @@ -0,0 +1,137 @@ +# Capability Profile and Host Binding Contract Plan + +**Goal:** Authorize every privileged builtin and host import through one immutable, auditable capability profile keyed by static callable identity. + +**Architecture:** Compilation records required callable IDs; binding validates them against a `CapabilityProfile` with a stable fingerprint. Builtin fast paths, registry functions, overrides, cached plans, and cloned registries all use the same authorization decision before dispatch. + +**Tech Stack:** Rust 2024, generated builtin catalog, host function registry, proc macro, binding-plan tests. + +**Status:** Completed + +--- + +## Independence and dependency + +- Depends on static builtin IDs for durable callable identity. +- Can be implemented alongside VM decomposition; the VM-owned part is callable authorization only. +- HTTP/IO/SQLite configuration remains state owned by each host implementation and is not embedded in the profile. + +## Scope boundary + +### In scope + +- One authorization path for privileged builtins and registry host imports. +- Immutable capability profiles and stable fingerprints. +- A clean boundary between callable authorization and host-owned path/network/database/process limits. +- Binding-plan cache correctness under clone/mutation/generation changes. +- Explicit ordinary versus Edge proc-macro contracts. + +### Out of scope + +- Adding agent/provider-specific capabilities. +- Implementing filesystem/process/task functions. +- Backward-compatible acceptance of implicit broad profiles. +- HTTP DNS/SSRF implementation details. +- pd-edge release workflow changes. + +## Target model + +```text +CallableId = static builtin ID or explicit host import identity +CapabilityProfile + allowed callables + delegation limits + fingerprint + +HostFunctionState (type-erased by the VM, typed by each host module) + HTTP configuration and in-flight permits + IO path/process/size policy + SQLite root and resource limits + +BindingPlan + program requirements + resolved call targets + capability fingerprint + registry generation/identity +``` + +## Implementation route + +### Milestone 1: Add authorization regression tests + +Required failing cases: + +- empty profile rejects privileged builtin fast paths; +- empty profile rejects registry hosts and overrides; +- language-pure builtins remain available under the documented baseline profile; +- cached/uncached plans make identical decisions; +- sibling registry mutation invalidates or rejects stale plans; +- clone identity and structural independence follow one documented rule; +- parameter policy cannot be widened by script input. + +### Milestone 2: Define immutable profiles + +**Files:** +- Modify: `src/vm/host.rs` +- Create: `src/vm/capability.rs` +- Modify: `src/lib.rs` + +1. Separate pure language builtins from privileged host capabilities. +2. Key builtin permissions by explicit static ID. +3. Key external hosts by a stable import identity, not a mutable vector slot alone. +4. Keep HTTP/SQLite/IO policy types out of the VM capability layer. +5. Compute a deterministic fingerprint from callable authorization only. + +### Milestone 3: Put authorization before every dispatch path + +**Files:** +- Modify: `src/vm/host.rs` +- Modify builtin dispatch and fast-path entry points + +1. Resolve callable identity. +2. Authorize before builtin fast path, override, registry call, or native continuation. +3. Ensure a plan cannot grant a capability absent from the current profile. +4. Reject undeclared imports during preflight before side effects. + +### Milestone 4: Simplify plan cache identity + +1. Bind plans to program identity, registry identity/generation, and capability fingerprint. +2. Remove Arc-token combinations that encode overlapping identity concepts. +3. Define clone behavior explicitly and test source/sibling mutation. +4. Make stale-plan errors deterministic. + +### Milestone 5: Separate proc-macro contracts + +**Files:** +- Modify: `pd-host-function/src/lib.rs` +- Modify: `pd-host-function/src/edge.rs` +- Add independent consumer compile fixtures + +1. Require an explicit Edge marker/scope for Edge-only expansion. +2. Ordinary async functions must receive an ordinary supported expansion or a direct compile-time diagnostic. +3. Apply one scoped HTTP routing rule to VM-aware and args-only forms. +4. Hide Edge implementation paths from ordinary downstream consumers. +5. Test from a separate crate that has no pd-edge internals. + +### Milestone 6: Verification + +```bash +cargo fmt --all -- --check +cargo test --locked -p pd-host-function +cargo test --locked --test runtime_host_tests +cargo test --locked --test http_host_tests --features http-client +cargo test --locked --workspace --all-features +cargo clippy --locked --workspace --all-targets --all-features -- -D warnings +git diff --check +``` + +## Target criteria + +- No privileged builtin bypasses the capability decision. +- Empty means deny for every privileged path. +- Capability identity does not depend on catalog order or registry slot alone. +- Binding-plan cache keys contain one stable capability fingerprint. +- Clones and stale plans have tested deterministic semantics. +- Ordinary proc-macro consumers never receive Edge-private symbols. +- Parameter policies remain host-owned native upper bounds that script data cannot widen. +- `HostRuntime`, `CapabilityProfile`, and inherent `Vm` APIs do not name HTTP/IO/SQLite policy types. diff --git a/plans/2026-08-09_http-transport-security-executor.md b/plans/2026-08-09_http-transport-security-executor.md new file mode 100644 index 00000000..75ec902e --- /dev/null +++ b/plans/2026-08-09_http-transport-security-executor.md @@ -0,0 +1,182 @@ +# Async Host Transport Security Plan + +**Goal:** Make HTTP a host-driven async host function, provide feature-selected blocking/async IO implementations, and preserve transport security without any HTTP-owned scheduler. + +**Architecture:** `#[pd_host_function] async fn` produces a generic async host factory. The VM allocates an operation ID and hands the resulting `'static` future to the embedding's `HostAsyncBridge`; the host owns submission, waking, polling, cancellation, and reactor/executor integration. HTTP contains only policy, request construction, transport, redirect, deadline, and response decoding logic. Edge scopes and `SharedProxyVmContext` expansion remain owned by `pd-edge`. + +**Tech Stack:** Rust 2024, proc macros, `Future`, `HostAsyncBridge`, Reqwest/Tokio under the `async` feature, local HTTP fixtures. + +--- + +## Independence and dependency + +- Depends on the capability profile and unified host lifecycle. +- Adds a generic async host ABI before migrating IO or HTTP. +- Requires a coordinated `pd-edge` adapter migration because the current core proc-macro contains Edge-specific expansion. +- Independent of agent provider protocols and source-language `async`/`await` syntax. + +## Hard ownership rules + +1. The VM owns script suspension state and operation-ID allocation. +2. The embedding host owns future storage, waking, polling, cancellation, and executor/reactor integration. +3. A subsystem async host function owns only its future body and typed policy/context snapshot. +4. HTTP must not create a thread, Tokio runtime, oneshot completion scheduler, private pending map, private poller, or independent operation-ID namespace. +5. Core `pd-host-function` must not contain Edge scope enums, `SharedProxyVmContext`, `crate::abi_impl` paths, or pd-edge registry generation. +6. With the `async` feature disabled, IO keeps its blocking implementation and HTTP is absent from callable metadata and runtime registration. +7. With the `async` feature enabled, IO binds its async implementation and HTTP is available only through the async host ABI. +8. Async operation driving code lives in dedicated folders: core contracts/lifecycle under `src/vm/async_host/`, and the pd-edge driver under `pd-edge/src/async_host/`. It must not accumulate in `host.rs`, `abi_impl/mod.rs`, HTTP, or IO modules. + +## Scope boundary + +### In scope + +- Generic async `#[pd_host_function(name = "...")]` expansion using owned arguments. +- Host-driven future submission/poll/cancel ABI. +- Migration of pd-edge scope expansion to a pd-edge-owned proc-macro adapter. +- Feature-selected blocking/async IO implementations. +- Async-only generic HTTP host. +- Complete IPv4/IPv6 special-use classification. +- Async DNS, total/connect/first-byte/idle deadlines, destination pinning, redirect revalidation, and body limits. +- Cancellation/reset/drop tests across host-driver and transport phases. + +### Out of scope + +- A VM-owned Tokio runtime or process executor. +- HTTP-specific scheduling infrastructure. +- Provider JSON, retries, model selection, SSE semantic parsing, or agent loops. +- Ambient proxy support by default. +- Script-controlled policy relaxation. +- Source-language futures or `await` syntax. + +## Implementation route + +### Milestone 1: Freeze generic async host semantics with RED tests + +**Files:** +- Modify: `pd-host-function/src/lib.rs` +- Modify: `tests/host_binding_generation_tests.rs` +- Modify: VM host lifecycle tests + +Cover: + +- ordinary owned-argument async signatures are accepted; +- borrowed typed parameters and raw borrowed args are rejected for async hosts; +- generated wrappers submit exactly one `'static` future to the installed host driver; +- missing driver fails before a pending operation becomes visible; +- completion returns exactly one terminal result; +- cancellation/reset/drop cancel the driver operation exactly once; +- rejected submission retires the operation ID and leaves no waiting state; +- interpreter/JIT/AOT all suspend through the same `CallOutcome::Pending` boundary. + +### Milestone 2: Add the host-driven async ABI + +**Files:** +- Create: `src/vm/async_host/mod.rs` +- Create supporting files under `src/vm/async_host/` for lifecycle/bridge concerns when needed +- Modify: `src/vm/host.rs` only to remove superseded inline async lifecycle code +- Modify: `src/vm/host_runtime.rs` +- Modify: `src/vm/mod.rs` +- Modify: `build.rs` +- Modify: `pd-host-function/src/lib.rs` + +1. Define the boxed `'static` host future output contract. +2. Extend or replace `HostAsyncBridge` with explicit submit, poll, and cancel operations. +3. Allocate IDs through the shared operation registry before submission and retire them on every fallible path. +4. Generate an async host adapter that takes owned script arguments, constructs a future, and submits it through the VM boundary. +5. Classify async generated hosts as suspension-capable and exclude them from non-yielding native fast paths. +6. Keep cached registry binding and direct VM binding equivalent. + +### Milestone 3: Return Edge scope ownership to pd-edge + +**Core files:** +- Remove: `pd-host-function/src/edge.rs` +- Modify: `pd-host-function/src/lib.rs` + +**pd-edge files:** +- Create a pd-edge-owned proc-macro adapter crate or equivalent owned macro module. +- Create: `src/async_host/mod.rs` and focused driver/operation files beneath that folder. +- Migrate `scope = runtime/http/http_extension/transport`, bind parameters, registry generation, and `SharedProxyVmContext` preparation. +- Adapt `VmAsyncOpBridge` to the generic host submit/poll/cancel contract. +- Remove future storage, operation allocation, reactor entry, and bridge polling logic from `src/abi_impl/mod.rs`. + +The core proc-macro must retain only name-based generic sync/async host expansion. + +### Milestone 4: Provide blocking and async IO implementations + +**Files:** +- Modify: IO runtime modules, build generation, and IO tests + +1. Keep blocking IO available without the `async` feature. +2. Under `async`, bind the same script-facing IO API to async host functions with owned parameters. +3. Make build-time callable discovery honor the active feature so only one implementation enters metadata/registration. +4. Route async IO futures through the embedding host driver. +5. Preserve canonical path, process permission, byte, line, and handle policies in both variants. + +### Milestone 5: Migrate HTTP to an async host function + +**Files:** +- Modify: `Cargo.toml` +- Modify: `src/builtins/runtime/http.rs` +- Modify: `src/builtins/runtime/mod.rs` +- Modify: `tests/vm/http_host_tests.rs` + +1. Gate HTTP dependencies, callable metadata, configuration, and tests on `async`. +2. Convert `http::client::request` to a true async host function with an owned request and immutable policy/context snapshot. +3. Delete `schedule_request`, `HttpCompletion`, `HttpRequestResource`, oneshot completion, per-request threads/runtimes, and HTTP-specific pending polling. +4. Resolve DNS inside the submitted future and count it against the total deadline. +5. Validate and pin every resolved address before connection while preserving TLS hostname/SNI verification. +6. Repeat policy, credential stripping, pinning, and deadline checks for every redirect. +7. Enforce request/header/body, connect, first-byte, idle, total, and response-byte limits. +8. Disable ambient environment proxies unless the embedding supplies explicit policy. + +### Milestone 6: Cancellation and lifecycle convergence + +1. Propagate run cancellation/deadline/reset/drop to `HostAsyncBridge::cancel_op_with_reason`. +2. Ensure driver completion after a terminal run cannot re-enter the VM. +3. Verify cancellation during DNS, connect, response headers, and body streaming. +4. Confirm no HTTP or IO async subsystem resource remains after completion/cancellation. +5. Remove runtime owner-poller routing that became obsolete after host-driver migration. + +### Milestone 7: Verification + +Core: + +```bash +cargo fmt --all -- --check +cargo test --locked -p pd-host-function +cargo test --locked --test host_binding_generation_tests --all-features +cargo test --locked --test io_builtin_edge_tests --all-features +cargo test --locked --test http_host_tests --features async +cargo test --locked --workspace --all-features +cargo test --locked --workspace --no-default-features --tests --no-run +cargo clippy --locked --workspace --all-targets --all-features -- -D warnings +RUSTDOCFLAGS='-D warnings' cargo doc --locked --workspace --all-features --no-deps +git diff --check +``` + +pd-edge: + +```bash +cargo fmt --all -- --check +cargo test --locked --workspace --all-features +cargo clippy --locked --workspace --all-targets --all-features -- -D warnings +git diff --check +``` + +All temporary and target directories must be under `/mnt/TEMP/rustscript/` and removed after final review. + +## Target criteria + +- A generic ordinary async host function compiles and suspends through the host driver. +- The host driver, not VM/HTTP/IO, stores and drives submitted futures. +- Async driving code is isolated in the dedicated core and pd-edge async-host folders. +- Core proc-macro code has no pd-edge scope or context knowledge. +- IO has blocking and async implementations selected by `async`. +- HTTP is unavailable without `async` and uses no private scheduler. +- DNS time counts against the total deadline. +- Every connected address is validated and pinned before connection. +- Special-use IPv4/IPv6 ranges are denied by default. +- Redirects repeat destination and credential validation. +- Cancellation during every transport phase reaches the driver and finishes within the documented bound. +- Reset/drop leave no future, operation, permit, stream, or response body live. +- HTTP host code contains no provider or agent policy. diff --git a/plans/2026-08-09_nested-module-correctness.md b/plans/2026-08-09_nested-module-correctness.md new file mode 100644 index 00000000..a19f2f46 --- /dev/null +++ b/plans/2026-08-09_nested-module-correctness.md @@ -0,0 +1,127 @@ +# Nested Module Correctness Implementation Plan + +**Goal:** Correct the current nested RSS module loader without expanding it into a new module architecture. + +**Architecture:** Keep the existing source-loader, import rewrite, and public `SourcePathError` shape, while repairing UTF-8 preservation, lexical path normalization, source ownership in diagnostics, and parity across public compile entry points. This is the short corrective plan; semantic module replacement is covered separately. + +**Tech Stack:** Rust 2024, RustScript compiler/source loader, `SourceMap`, CLI diagnostics, Cargo integration tests. + +--- + +## Independence and dependency + +- Independent of VM runtime, builtin IDs, agent gateway, HTTP, and persistence. +- Must complete before the semantic module-system plan begins. +- The agent repository only supplies a composition fixture; compiler behavior remains owned by `rustscript`. + +## Scope boundary + +### In scope + +- UTF-8-safe call-site rewrite. +- Correct preservation of unmatched relative `..` components. +- Canonical identity for equivalent disk module paths. +- Nested parse and strict-type diagnostics rendered from the real module source. +- Equivalent behavior for file, source-with-options, and source-at-path public entry points. +- Regression coverage for import/export/cycle behavior already changed in the worktree. + +### Out of scope + +- New public error enum variants. +- A compatibility wrapper for the removed `SourceAt` experiment. +- Semantic symbol resolution or a new IR. +- Agent storage migration or gateway changes. +- VM-visible opcodes or host capabilities. + +## Implementation route + +### Milestone 1: Add failing UTF-8 rewrite tests + +**Files:** +- Modify: `tests/compiler/module_import_tests.rs` +- Modify: unit tests in `src/compiler/source_loader/rewrite.rs` + +Add nested modules that trigger namespace and named-import rewriting while containing: + +- non-ASCII string literals; +- line and block comments; +- non-ASCII source outside rewritten spans where syntax permits it. + +Assert byte-for-byte preservation of untouched source and runtime preservation of values such as `"猫"`. + +### Milestone 2: Make scanners copy source slices + +**Files:** +- Modify: `src/compiler/source_loader/rewrite.rs` + +1. Stop appending UTF-8 bytes with `byte as char`. +2. Advance by valid UTF-8 scalar boundaries or copy untouched ranges as source slices. +3. Keep token recognition ASCII-specific where the grammar requires ASCII identifiers and separators. +4. Preserve comments, strings, escapes, and line counts exactly. + +### Milestone 3: Correct path normalization and identity + +**Files:** +- Modify: `src/compiler/source_loader/imports.rs` +- Modify: `src/compiler/source_loader/graph.rs` +- Test: `tests/compiler/module_import_tests.rs` + +1. Pop `ParentDir` only when the previous normalized component is `Normal`. +2. Never cancel an unmatched `ParentDir` with a later `ParentDir`. +3. Preserve root semantics for absolute paths. +4. Use canonical disk identity for files that exist; use a normalized explicit virtual identity for source overrides. +5. Key `seen`, `visiting`, exports, and overrides with the same module identity. +6. Test consecutive `super::`, absolute above-root input rejection/normalization policy, path aliases, cycle aliases, and duplicate import aliases. + +### Milestone 4: Carry nested source context to diagnostics + +**Files:** +- Modify: `src/compiler/source_loader.rs` +- Modify: `src/compiler/source_loader/graph.rs` +- Modify: `src/compiler/pipeline.rs` +- Modify: `src/cli.rs` +- Test: `tests/compiler/module_import_tests.rs` +- Add or modify CLI diagnostic integration tests + +1. Keep `SourcePathError` public enum shape unchanged. +2. Carry internal `{ path, source text, SourceId/span }` context through compilation. +3. Render nested parse and strict-type errors against the nested source, not the root source map. +4. Apply the same path/source enrichment to: + - `compile_source_file`; + - `compile_source_with_flavor_and_options`; + - `compile_source_at_path_with_flavor_and_options`. +5. Test path, line, code frame, underline, and source override content, not message text alone. + +### Milestone 5: Preserve import/export behavior + +Add regression cases for: + +- nested namespace aliases; +- nested named imports; +- public-only exports; +- no transitive re-export; +- same-directory `self::` and parent-directory `super::`; +- missing modules and normalized cycles; +- root and nested host namespace imports. + +### Milestone 6: Verification + +```bash +cargo fmt --all -- --check +cargo test --locked --test compiler_tests module_import +cargo test --locked --test compiler_tests +cargo test --locked --workspace --all-features +git diff --check +``` + +Run the CLI diagnostic fixture and assert that the rendered path and highlighted line both belong to the nested source. + +## Target criteria + +- Rewriting never changes untouched UTF-8 bytes. +- Consecutive unmatched parent components retain their lexical meaning. +- Equivalent disk paths resolve to one module identity. +- Nested diagnostics display the actual nested source line and underline. +- All public module-capable compile entry points identify the failing module. +- Existing public error enum shape remains unchanged. +- Import, export, and cycle tests pass without agent-specific compiler behavior. diff --git a/plans/2026-08-09_run-outcome-event-error-contract.md b/plans/2026-08-09_run-outcome-event-error-contract.md new file mode 100644 index 00000000..c0222507 --- /dev/null +++ b/plans/2026-08-09_run-outcome-event-error-contract.md @@ -0,0 +1,140 @@ +# Run Outcome, Event Stream, and Runtime Error Contract Plan + +**Goal:** Define one structured execution result that keeps return values, events, usage, cancellation, and errors separate and machine-readable. + +**Architecture:** A run produces a terminal `RunOutcome`; events flow during execution through a bounded sink/channel and never replace the function return value. Runtime and host failures retain structured codes and context through the VM embedding boundary. + +**Tech Stack:** Rust 2024, VM embedding API, runtime context/events, host errors, agent runner integration tests. + +--- + +## Independence and dependency + +- Contract design can start independently. +- Implementation depends on RunContext ownership from the VM decomposition plan. +- Operation cancellation details depend on the unified host-lifecycle plan. +- The agent run-lifecycle plan consumes this API. + +## Scope boundary + +### In scope + +- `RunOutcome`, terminal reason, usage, return value, and structured error. +- Bounded event emission during execution. +- Event receipt/sequence semantics at the VM boundary. +- Structured runtime/host error propagation. +- Removal of stack-top/event-last inference in embedding code. + +### Out of scope + +- Agent event names, provider protocols, SSE framing, or Telegram rendering. +- Durable event persistence. +- Source-language concurrency syntax. +- Compatibility wrappers for ambiguous prior return behavior. + +## Target contracts + +```text +RunOutcome + return_value: optional Value + termination: completed | cancelled | failed | budget_exhausted + error: optional RuntimeError + usage: RunUsage + last_event_sequence + +RuntimeEvent + sequence + value + payload_bytes + +RuntimeError + code + message + subsystem + operation/resource context + retryability where meaningful + source error where meaningful +``` + +## Implementation route + +### Milestone 1: Add contract tests + +Add tests proving: + +- a script may emit events and return a different value; +- zero events does not alter the return value; +- event order is monotonic; +- sink rejection/backpressure has a documented terminal behavior; +- cancellation reason survives the public VM API; +- host/runtime codes survive without string equality checks; +- usage is finalized for success, error, cancellation, and budget exhaustion. + +### Milestone 2: Define terminal and usage types + +**Files:** +- Modify: `src/lib.rs` +- Create: `src/vm/outcome.rs` +- Modify runtime error modules + +1. Define `RunOutcome`, `RunTermination`, and `RunUsage`. +2. Make halt/failure/cancellation paths produce exactly one terminal outcome. +3. Stop requiring embedders to inspect stack top, yield reason, and side channels to infer completion. + +### Milestone 3: Make events live and bounded + +**Files:** +- Modify: `src/builtins/runtime/context.rs` +- Modify: `src/builtins/runtime/event.rs` +- Modify: `src/builtins/runtime/context_host.rs` +- Modify: RunContext + +1. Define a bounded event sink contract. +2. Emit each accepted event during execution. +3. Allocate sequence numbers once at the run boundary. +4. Define overflow policy explicitly: block/yield, return a typed limit error, or drop only where configured with a receipt. Silent loss is prohibited. +5. Keep event values independent from function return storage. + +### Milestone 4: Preserve structured errors + +**Files:** +- Modify: runtime error types +- Modify: `src/vm/host.rs` +- Modify: public VM error surface + +1. Carry `RuntimeErrorCode` through host completion and `RunOutcome`. +2. Include structured cancellation/deadline/resource/operation context. +3. Remove embedding logic that compares error strings such as `"cancelled"`. +4. Define rendering separately from machine-readable fields. + +### Milestone 5: Migrate embedders and remove ambiguous APIs + +**Files:** +- Modify examples and tests in `rustscript` +- Coordinate later changes in `rustscript-agent/src/lib.rs` + +1. Consume `RunOutcome.return_value` directly. +2. Subscribe to events through the sink/channel. +3. Remove event-last and stack-last fallback behavior. +4. Remove superseded internal return APIs after migration; no dual long-term contract. + +### Milestone 6: Verification + +```bash +cargo fmt --all -- --check +cargo test --locked --test runtime_context_tests +cargo test --locked --test runtime_host_tests +cargo test --locked --workspace --all-features +cargo clippy --locked --workspace --all-targets --all-features -- -D warnings +git diff --check +``` + +## Target criteria + +- Emitting an event never changes the function return value. +- Events are observable before run completion through a bounded contract. +- Every run produces one structured terminal outcome. +- Cancellation, deadline, resource, and host errors retain machine-readable codes. +- Embedders do not infer results from stack/event ordering. +- String equality is absent from cancellation/error control flow. +- Usage and event sequence metadata are finalized for every terminal path. diff --git a/plans/2026-08-09_semantic-module-system.md b/plans/2026-08-09_semantic-module-system.md new file mode 100644 index 00000000..121bd5f6 --- /dev/null +++ b/plans/2026-08-09_semantic-module-system.md @@ -0,0 +1,145 @@ +# Semantic Module System Implementation Plan + +**Goal:** Replace textual import rewriting and synthetic declarations with a semantic module graph and symbol resolution model. + +**Architecture:** Parse imports as syntax, assign every source a `ModuleId` and `SourceId`, resolve declarations to `SymbolId`, and link by resolved identity. Module namespaces, visibility, private helpers, and diagnostics become first-class compiler data instead of rewritten text and parallel metadata arrays. + +**Tech Stack:** Rust 2024, RustScript parser, frontend IR, compiler pipeline, linker, `SourceMap`. + +--- + +## Independence and dependency + +- Depends on completion of `2026-08-09_nested-module-correctness.md` so the current branch has a verified baseline. +- Independent of VM execution, host capabilities, HTTP, agent gateway, and backend optimization. +- May proceed in compiler-only milestones with bytecode output held behaviorally constant. + +## Scope boundary + +### In scope + +- `ModuleId`, `SourceId`, `SymbolId`, import edges, export tables, and visibility. +- Semantic namespace and named-import resolution. +- Private function identity across modules. +- Source-owned diagnostics after graph merge. +- Removal of synthetic function preludes and call-site text rewriting for file modules. + +### Out of scope + +- Package manager, remote modules, registry resolution, or dependency downloads. +- Dynamic module loading at VM runtime. +- New bytecode opcodes solely for module names. +- Host namespace redesign. +- Agent-specific storage or provider modules. + +## Implementation route + +### Milestone 1: Define compiler-owned identities + +**Files:** +- Create: `src/compiler/modules.rs` +- Modify: `src/compiler/source_loader.rs` +- Modify: `src/compiler/pipeline.rs` +- Test: compiler module tests + +Define: + +```text +ModuleId +SourceId +SymbolId +ModuleGraph +ModuleNode { source, imports, declarations, exports } +ResolvedImport +``` + +IDs are deterministic within one compilation and never derived only from a file stem. + +### Milestone 2: Parse import syntax into AST/IR + +**Files:** +- Modify RustScript parser/frontend import nodes +- Modify source-loader import discovery +- Test parser and module fixtures + +1. Stop using line-prefix stripping as the authoritative import parser. +2. Preserve import spans and clauses in the parsed unit. +3. Resolve `self::`, `super::`, namespace aliases, and named imports from structured nodes. +4. Keep host namespace imports on their existing dedicated resolution path. + +### Milestone 3: Build declarations and export tables + +**Files:** +- Modify frontend IR declaration metadata +- Modify `src/compiler/source_loader/graph.rs` +- Modify linker symbol collection + +1. Assign each declaration a symbol owned by its module. +2. Mark public exports explicitly. +3. Keep imported symbols separate from local declarations. +4. Prevent implicit transitive re-export. +5. Permit different modules to have private or public functions with the same source name. + +### Milestone 4: Resolve calls by symbol identity + +**Files:** +- Modify expression/call IR +- Modify `src/compiler/linker.rs` +- Modify lowering consumers + +1. Resolve local, named-import, and namespace calls to `SymbolId` before merge. +2. Replace string-based global function matching with symbol lookup. +3. Use deterministic internal mangling only at the final flat bytecode boundary if required. +4. Remove basename-only scope prefixes. + +### Milestone 5: Preserve source ownership through merge + +**Files:** +- Modify `src/compiler/pipeline.rs` +- Modify diagnostic/source-map structures +- Test rendered diagnostics + +1. Every span retains its source identity. +2. Merging units cannot reinterpret one module's offset in another source. +3. Parse, typing, duplicate symbol, visibility, and unresolved import errors render from the owning source. +4. Remove parallel `stmt_sources`, `function_sources`, and ad hoc prelude line remapping where replaced by source-owned IR. + +### Milestone 6: Remove textual compatibility machinery + +**Files:** +- Remove obsolete paths in `src/compiler/source_loader/rewrite.rs` +- Remove synthetic prelude generation and related line maps +- Update tests and compiler docs + +Do not retain a second module pipeline after semantic resolution reaches parity. + +### Milestone 7: Verification + +Required cases: + +- two directories containing modules with the same stem; +- two namespaces exporting the same function name; +- same-named private helpers in multiple modules; +- visibility errors and no transitive re-export; +- cycles through path aliases; +- in-memory overrides mixed with disk modules; +- deterministic output independent of import discovery order; +- source-correct diagnostics for every module. + +```bash +cargo fmt --all -- --check +cargo test --locked --test compiler_tests +cargo test --locked --workspace --all-features +cargo clippy --locked --workspace --all-targets --all-features -- -D warnings +git diff --check +``` + +## Target criteria + +- File-module calls are resolved by `SymbolId`, not rewritten source text. +- Module identity never depends only on a basename. +- Same-named declarations in independent modules coexist. +- Public/private and re-export rules are represented in compiler data. +- Synthetic imported-function preludes are removed from the module path. +- Every diagnostic span retains its owning source after linking. +- No agent-specific compiler rule is introduced. diff --git a/plans/2026-08-09_static-builtin-id.md b/plans/2026-08-09_static-builtin-id.md new file mode 100644 index 00000000..6296ecd7 --- /dev/null +++ b/plans/2026-08-09_static-builtin-id.md @@ -0,0 +1,107 @@ +# Static Builtin ID Implementation Plan + +**Goal:** Replace count-derived builtin call indices with explicit static IDs that remain fixed after assignment. + +**Architecture:** Every VM-visible builtin receives an explicit `u16` ID in one authoritative catalog. `build.rs`, the interpreter, compiler, wire encoder/decoder, and `pd-vm-nostd` consume generated tables from that catalog. This migration may break existing VMBC once; the implementation will bump the bytecode ABI and reject the previous format instead of carrying an old-ID decoder. + +**Tech Stack:** Rust 2024, `build.rs` code generation, VMBC wire format, `pd-vm`, `pd-vm-nostd`. + +--- + +## Independence and dependency + +- Independent of agent framework, module loading, HTTP behavior, and JIT refactoring. +- Must land before more builtins are added. +- Later capability plans may key permissions by the static builtin ID. + +## Scope boundary + +### In scope + +- Explicit IDs for ordinary, internal, and special-call builtins. +- One authoritative catalog and generated forward/reverse lookup. +- A one-time VMBC ABI version bump. +- Compile-time duplicate/range validation. +- Shared std/no-std ID generation. + +### Out of scope + +- Compatibility decoding for prior VMBC versions. +- Aliases from old IDs to new IDs. +- New builtin behavior or host capabilities. +- Changes to source-language names. + +## Implementation route + +### Milestone 1: Freeze the ID contract with failing tests + +**Files:** +- Modify: `tests/wire/wire_tests.rs` +- Modify: `src/bytecode.rs` +- Add fixture/catalog tests under `tests/wire/` + +1. Add assertions for explicit IDs of representative ordinary, internal, and special builtins. +2. Add a uniqueness test over the complete catalog. +3. Add range tests proving static IDs do not overlap opcodes or reserved sentinels. +4. Add a test that appending a synthetic catalog entry does not change existing IDs. + +**RED command:** + +```bash +cargo test --locked --test wire_tests builtin +``` + +### Milestone 2: Introduce the authoritative catalog + +**Files:** +- Modify: `build.rs` +- Modify: `src/builtins/mod.rs` or create `src/builtins/catalog.rs` +- Modify generated builtin metadata consumers + +1. Define each entry as `{ id, source_name, Rust variant, class, feature gate }`. +2. Remove `BUILTIN_CALL_BASE` arithmetic from ID assignment. +3. Generate `BuiltinFunction::call_index`, reverse lookup, dispatch tables, and catalog iteration from explicit IDs. +4. Fail the build on duplicate IDs, duplicate names, out-of-range IDs, or a missing explicit ID. +5. Reserve documented ID blocks for ordinary, internal, and future extension entries without deriving IDs from catalog length. + +### Milestone 3: Share IDs with no-std + +**Files:** +- Modify: `pd-vm-nostd/src/vm.rs` +- Modify: `pd-vm-nostd/build.rs` or generate a shared checked-in artifact +- Modify: no-std wire tests + +1. Remove the duplicated `BUILTIN_BASE` constant. +2. Generate or import the same explicit ID table without requiring std-only dependencies. +3. Verify std compiler output executes under `pd-vm-nostd` with identical builtin dispatch. + +### Milestone 4: Declare the format break + +**Files:** +- Modify: `src/bytecode.rs` +- Modify: VMBC format tests and documentation + +1. Increment `BYTECODE_ABI_VERSION` once. +2. Reject the previous version with a deterministic unsupported-version error. +3. Do not add migration, dual decoding, or legacy aliases. +4. Regenerate only current-version fixtures. + +### Milestone 5: Full verification + +```bash +cargo fmt --all -- --check +cargo test --locked --test wire_tests +cargo test --locked -p pd-vm-nostd +cargo test --locked --workspace --all-features +cargo clippy --locked --workspace --all-targets --all-features -- -D warnings +git diff --check +``` + +## Target criteria + +- Every VM-visible builtin has one explicit static ID. +- Adding or reordering catalog entries leaves all prior explicit IDs unchanged. +- Duplicate or missing IDs fail during generation. +- std and no-std use the same IDs without manually mirrored base arithmetic. +- VMBC declares the one-time incompatible format change and rejects the old version. +- No compatibility decoder or old-ID alias remains. diff --git a/plans/2026-08-09_structured-task-supervisor.md b/plans/2026-08-09_structured-task-supervisor.md new file mode 100644 index 00000000..40ae0db9 --- /dev/null +++ b/plans/2026-08-09_structured-task-supervisor.md @@ -0,0 +1,134 @@ +# Structured Task Supervisor Implementation Plan + +**Goal:** Add implementation-independent structured concurrency for multiple pending operations and isolated child program runs without exposing Rust futures, threads, or executor handles to scripts. + +**Architecture:** A run-scoped `TaskSupervisor` owns child operation/program tasks, concurrency permits, cancellation tree, result ordering, and cleanup. Tasks are descriptors validated against delegated capability profiles. Parent completion cannot leave active descendants. + +**Tech Stack:** Rust 2024, RunContext, HostRuntime operation registry, isolated VM instances, bounded executor. + +--- + +## Independence and dependency + +- Depends on VM decomposition and unified host lifecycle. +- Consumes static capability identity/profile delegation. +- Independent of agent tool/subagent policy; agent RSS may wrap it later. + +## Scope boundary + +### In scope + +- Multiple active operation/task records per run. +- Bounded `all`, `pool`, `race`, fail-fast, and isolated program fanout semantics. +- Parent/child cancellation and resource budgets. +- Ordered result collection and event association. +- Removal of the one-waiting-slot architectural limitation for structured tasks. + +### Out of scope + +- Source-language `async`, `await`, arbitrary futures, or shared-memory threads. +- Agent-specific tool, provider, or subagent descriptors. +- Mutable resource sharing between child VMs. +- Distributed execution or durable background jobs. + +## Target contracts + +```text +TaskSupervisor + spawn(descriptor, delegated_profile) + all(task_ids) + pool(descriptors, max_concurrency, fail_fast) + race(task_ids) + cancel(task_id/reason) + cancel_all(reason) + +TaskDescriptor + host operation + isolated program + input + +TaskResult + index + terminal status + return value or structured error + usage +``` + +## Implementation route + +### Milestone 1: Freeze structured semantics with tests + +Cover: + +- ordered all/pool results despite completion order; +- race returns first success and cancels remaining tasks; +- fail-fast cancellation; +- collect-all partial failures; +- parent cancellation reaches every descendant; +- child cancellation does not affect siblings by default; +- depth/fanout/active/time/fuel/operation limits; +- no child result/event after parent terminal state; +- isolated stacks/resources/capability profiles. + +### Milestone 2: Add TaskSupervisor to RunContext/HostRuntime + +**Files:** +- Create: `src/builtins/runtime/task.rs` +- Modify: RunContext and HostRuntime component files +- Modify: operation registry integration + +1. Store task state outside Instance's single wait marker. +2. Register every task/child operation in the shared operation registry. +3. Allocate permits before spawn. +4. Create child cancellation tokens under the run token. +5. Associate result/event/usage with task and parent run identity. + +### Milestone 3: Support isolated child programs + +1. Spawn a fresh Instance and RunContext from immutable Program/Engine references. +2. Delegate only a subset of the parent capability profile. +3. Prohibit mutable resource-handle transfer. +4. Bound child input/output/event bytes. +5. Finalize child outcome before collection. + +### Milestone 4: Add generic task host surface + +Expose implementation-independent operations such as: + +```text +task::all +task::pool +task::race +task::run_program +task::run_program_many +``` + +Descriptors contain only generic host-operation or program references. They cannot contain agent tool/provider names. + +### Milestone 5: Integrate scheduler wake/resume + +1. Permit multiple active task operations while Instance waits on one structured join/select result. +2. Wake the instance when the requested aggregate condition is met. +3. Keep remaining task state under supervisor ownership. +4. Cancel and clean all descendants before parent terminal completion. + +### Milestone 6: Verification + +```bash +cargo fmt --all -- --check +cargo test --locked --test runtime_task_tests +cargo test --locked --workspace --all-features +cargo clippy --locked --workspace --all-targets --all-features -- -D warnings +git diff --check +``` + +Include stress tests for permit exhaustion, cancellation races, deterministic ordering, nested depth, and cleanup counters. + +## Target criteria + +- One run can own multiple active tasks under explicit bounds. +- Structured joins/races have deterministic documented semantics. +- Parent terminal state implies zero active descendants. +- Child VMs share immutable program/engine data only. +- Capability delegation can only narrow access. +- Mutable resource handles never cross child boundaries. +- Agent/provider concepts do not appear in the core task descriptors. diff --git a/plans/2026-08-09_unified-host-lifecycle.md b/plans/2026-08-09_unified-host-lifecycle.md new file mode 100644 index 00000000..5a28066e --- /dev/null +++ b/plans/2026-08-09_unified-host-lifecycle.md @@ -0,0 +1,160 @@ +# Unified Host Resource, Operation, and Cancellation Plan + +**Goal:** Make every privileged runtime subsystem use one resource arena, operation registry, cancellation model, and cleanup lifecycle. + +**Architecture:** `HostRuntime` owns typed opaque resources, operation identity, and cancellation state. Blocking IO and SQLite resources use the shared arena/registry. Async host futures are submitted to and driven by the embedding `HostAsyncBridge`; the VM retains only script suspension and lifecycle identity. Cancellation carries a structured reason and propagates from run to bridge operation and resource cleanup. + +**Tech Stack:** Rust 2024, existing runtime hosts, `HostOpId`, HTTP client, SQLite, VM reset/drop tests. + +**Status:** Completed + +**Async correction:** The original HTTP/IO migration used subsystem pollers and, for HTTP, a per-request thread/runtime. Those transitional paths are superseded by `2026-08-09_http-transport-security-executor.md`. Completion of this plan does not authorize VM-, HTTP-, or IO-owned async executors. + +**Directory correction:** Generic async host contracts and VM lifecycle glue belong under `src/vm/async_host/`; each embedding's concrete driver belongs in its own dedicated async-host folder. `host.rs` and subsystem modules must remain binding/business-logic surfaces rather than async driver containers. + +--- + +## Independence and dependency + +- Depends on the HostRuntime/RunContext ownership contract from `2026-08-09_vm-runtime-decomposition.md`. +- Capability authorization can be implemented in parallel if it targets the same HostRuntime boundary. +- Agent lifecycle consumes this contract but is not implemented here. + +## Scope boundary + +### In scope + +- One opaque resource handle format and typed resource validation. +- One operation-ID/cancellation registry, with async future dispatch delegated to the embedding host bridge. +- Cancellation tree/reasons, deadlines, cleanup, and terminal state. +- Migration of blocking IO and SQLite state plus lifecycle identity for host-driven async operations. +- Removal of unused generic substrate and subsystem-specific duplicate registries. + +### Out of scope + +- Agent subagent semantics or provider retries. +- New filesystem/process/task host APIs. +- Source-language futures, `async`, or `await` syntax. +- Sharing mutable resources across VMs. + +## Target contracts + +```text +ResourceArena + insert(type, value, cleanup) + get(handle, expected_type) + close(handle, reason) + close_all(reason) + +OperationRegistry + start(owner, cancellation, cleanup) + complete(id, result) + cancel(id, reason) + cancel_all(reason) + +HostAsyncBridge + submit(id, future) + poll(id, waker) + cancel(id, reason) + +CancellationToken + parent + reason + deadline + child tokens +``` + +Handles must encode enough table/generation/type identity to reject stale, forged, cross-type, and cross-VM use. + +## Implementation route + +### Milestone 1: Freeze lifecycle semantics with tests + +Add tests for: + +- stale handle after close; +- handle reuse with generation change; +- wrong resource type; +- cross-VM handle rejection; +- operation completion/cancel race; +- reset/drop cleanup exactly once; +- parent cancellation propagation; +- timeout, user stop, resource close, and VM reset reasons. + +### Milestone 2: Replace the unused generic substrate + +**Files:** +- Modify: `src/builtins/runtime/resource.rs` +- Modify: `src/builtins/runtime/cancellation.rs` +- Modify: `src/vm/host_runtime.rs` + +1. Store opaque host resources, not only language `Value` objects. +2. Define a resource type identifier and cleanup contract. +3. Make operation owner/poll/cancel routing data-driven. +4. Remove APIs that remain unused after the contract is fixed. + +### Milestone 3: Migrate SQLite + +**Files:** +- Modify: `src/builtins/runtime/sqlite.rs` +- Modify: SQLite tests + +1. Replace SQLite-local handle counters and connection maps with ResourceArena handles. +2. Replace SQLite-local pending-op maps/signals with OperationRegistry. +3. Register `InterruptHandle` cancellation cleanup. +4. Ensure reset/drop waits only for the bounded documented grace period and no operation can re-enter a completed run. +5. Preserve path, SQL, row, byte, transaction, and authorizer limits. + +### Milestone 4: Establish HTTP lifecycle identity + +**Files:** +- Modify: `src/builtins/runtime/http.rs` +- Modify: HTTP tests + +1. Allocate HTTP-visible wait identities through the shared operation registry. +2. Connect run cancellation/reset/drop to the host async bridge. +3. Do not store or poll HTTP futures through HTTP-specific resources. +4. Remove HTTP-specific pending dispatch from `runtime/mod.rs` during the async host migration. + +### Milestone 5: Migrate IO and other existing resources + +**Files:** +- Modify IO runtime modules and VM host polling + +Keep blocking file/iterator/callback resources in the shared arena/registry. Async IO futures use the embedding host bridge under the `async` feature. Delete subsystem counters/maps and pollers after migration. + +### Milestone 6: Centralize wait/poll/cancel + +**Files:** +- Modify: `src/builtins/runtime/mod.rs` +- Modify: `src/vm/host.rs` +- Modify: `src/vm/mod.rs` or new component files + +1. Replace subsystem `if` chains with one bridge dispatch for async host operations. +2. Let Instance wait on an operation ID while HostRuntime owns lifecycle state and the embedding owns the future. +3. Route run cancellation, deadline, resource close, reset, and drop through one cancellation API. +4. Guarantee one terminal transition and one cleanup execution. + +### Milestone 7: Verification + +```bash +cargo fmt --all -- --check +cargo test --locked --test runtime_context_tests +cargo test --locked --test runtime_host_tests +cargo test --locked --test http_host_tests --features http-client +cargo test --locked --test sqlite_host_tests --features sqlite +cargo test --locked --workspace --all-features +cargo clippy --locked --workspace --all-targets --all-features -- -D warnings +git diff --check +``` + +## Target criteria + +- Production blocking IO/SQLite paths use the shared arena and registry; async IO/HTTP use shared lifecycle IDs plus the embedding host bridge. +- No subsystem owns an independent public operation-ID namespace. +- Cancellation reasons remain structured from run through cleanup. +- Close/reset/drop invoke cleanup once and reject stale handles afterward. +- A terminal run cannot receive a late operation result. +- Generic resource/operation code has production callers and no broad dead-code warnings. +- Per-subsystem async polling chains are removed. +- No HTTP/IO path creates a private thread, runtime, oneshot completion scheduler, or executor. diff --git a/plans/2026-08-09_vm-runtime-decomposition.md b/plans/2026-08-09_vm-runtime-decomposition.md new file mode 100644 index 00000000..2c80b101 --- /dev/null +++ b/plans/2026-08-09_vm-runtime-decomposition.md @@ -0,0 +1,150 @@ +# VM Runtime Decomposition Implementation Plan + +**Goal:** Split the current monolithic `Vm` state into explicit engine, program, instance, run-context, and host-runtime ownership layers. + +**Architecture:** Immutable compiled artifacts and backend caches live outside per-run execution state. An `Instance` owns interpreter state, a `RunContext` owns one execution's input/budgets/events/cancellation, and `HostRuntime` owns capabilities/resources/operations. The migration preserves observable execution behavior while removing subsystem-specific fields from the central VM object. + +**Tech Stack:** Rust 2024, `pd-vm` interpreter/JIT/AOT integration, existing compiler and runtime tests. + +--- + +## Independence and dependency + +- Static builtin IDs should land first so decomposition does not move an unstable wire catalog. +- Defines ownership required by the unified host-lifecycle and RunOutcome plans. +- Independent of agent providers, gateway routes, module semantics, and new host functions. + +## Scope boundary + +### In scope + +- Ownership split for program, backend cache, interpreter instance, run-scoped context, and host runtime. +- Explicit reset/drop semantics for each layer. +- Removal of subsystem fields from the top-level VM facade. +- Migration of embedding entry points to the new ownership model. + +### Out of scope + +- New language syntax or bytecode operations. +- New host capabilities. +- Compatibility adapters for every prior internal API. +- JIT/AOT optimization redesign. +- Agent-specific execution policy. + +## Target model + +```text +Engine + backend configuration + decoded/JIT/AOT caches + code-generation telemetry + +Program + immutable bytecode + constants and metadata + import requirements + +Instance + instruction pointer + stack, locals, frames, captures + yield/wait state + +RunContext + input + event channel + fuel/deadline/cancellation + usage accounting + +HostRuntime + capability profile + resources + operations + executor +``` + +The public facade may be renamed or retained, but ownership must follow this model. + +## Implementation route + +### Milestone 1: Add ownership tests + +**Files:** +- Add focused tests under `tests/vm/` +- Modify reset/reuse tests + +Prove: + +- one immutable program can create multiple isolated instances; +- run input/events/budgets never leak between runs; +- backend cache may be shared without sharing stacks/resources; +- reset closes run-scoped state and retains only documented reusable state. + +### Milestone 2: Extract immutable Program and Engine state + +**Files:** +- Modify: `src/vm/mod.rs` +- Create: `src/vm/engine.rs` +- Create or refine: `src/vm/program.rs` +- Move backend cache ownership from VM fields + +1. Remove raw program pointer/cache duplication from per-run state. +2. Give Engine explicit cache keys and invalidation rules. +3. Keep Program immutable after validation/binding metadata construction. +4. Test program sharing across interpreter-only, JIT, and AOT configurations. + +### Milestone 3: Extract Instance state + +**Files:** +- Create: `src/vm/instance.rs` +- Modify interpreter dispatch and frame helpers + +Move IP, stack, locals, frames, captures, callbacks, waiting/yield state, and instance-only counters. Define one lifecycle from new to halted/failed/cancelled. + +### Milestone 4: Introduce RunContext + +**Files:** +- Create: `src/vm/run_context.rs` +- Move runtime input, event sink, fuel, epoch/deadline, cancellation, and usage state + +1. Create a fresh RunContext per execution. +2. Make cancellation and deadline mandatory run-owned data, with explicit unlimited settings where allowed. +3. Remove source injection and embedding-global event ownership from execution paths. +4. Make run completion consume/finalize the context. + +### Milestone 5: Extract HostRuntime shell + +**Files:** +- Create: `src/vm/host_runtime.rs` +- Modify: `src/vm/host.rs` +- Modify: `src/builtins/runtime/mod.rs` + +Move capability profile, host bindings, resource tables, operation registry, and executor references behind HostRuntime. Subsystem migration proceeds in the separate host-lifecycle plan. + +### Milestone 6: Remove duplicate lifecycle paths + +1. Replace central constructor/reset/drop field lists with component lifecycle methods. +2. Remove fields that exist only as transitional mirrors. +3. Remove old internal APIs once all callers move; no long-lived compatibility layer. +4. Document thread-safety and clone semantics for Engine, Program, Instance, RunContext, and HostRuntime. + +### Milestone 7: Verification + +```bash +cargo fmt --all -- --check +cargo test --locked --workspace --all-features +cargo test --locked -p pd-vm-nostd +cargo clippy --locked --workspace --all-targets --all-features -- -D warnings +git diff --check +``` + +Add behavioral comparison fixtures that run the same program before and after each extraction milestone. + +## Target criteria + +- Immutable Program data and backend caches are not owned by per-run state. +- Stack/frame/wait state is isolated in Instance. +- Input/events/budget/cancellation are isolated in RunContext. +- Capabilities/resources/operations are isolated in HostRuntime. +- Reset and drop no longer enumerate every runtime subsystem in one central method. +- Multiple instances from one program cannot share mutable run or host resources. +- Existing interpreter/JIT/AOT/no-std behavior tests remain passing.