refactor: rewrite plugin system from extism/WASM to stabby native dylibs#7
Closed
markovejnovic wants to merge 63 commits into
Closed
refactor: rewrite plugin system from extism/WASM to stabby native dylibs#7markovejnovic wants to merge 63 commits into
markovejnovic wants to merge 63 commits into
Conversation
Add stabby =72.1.1 and borsh 1 as workspace dependencies. Replace extism-pdk with stabby+borsh in hm-plugin-sdk. Define the two core FFI boundary traits (RawPlugin for plugin→host, RawHostApi for host→plugin) using #[stabby::stabby] with extern "C" + DynFuture for async methods. Compile-time assertions verify both traits produce valid stabby trait objects.
Introduce PluginContext that wraps stabby's DynRef<RawHostApi> with ergonomic Rust-native methods (log, kv_get/kv_set, emit_event, emit_step_log, should_cancel, write_stdout/stderr, archive I/O, fs_read_config). Rewrite the four capability traits (StepExecutor, LifecycleHook, SubcommandPlugin, OutputFormatter) to be async: each now requires Send + Sync + Default, takes &PluginContext as first arg, and returns impl Future<Output = Result<..., PluginError>> + Send + '_ (RPITIT). Remove stale pub mod manifest (re-exports already covered by pub use hm_plugin_protocol::*) and redundant SubcommandInput re-export. Update lib.rs module doc to describe the new stabby-based SDK.
Review feedback: archive_read/archive_total_size now accept &ArchiveId instead of raw &[u8], matching the ergonomic wrapper pattern. Added emit_step_log(StdStream, &[u8]) alongside the convenience emit_step_log_stdout/stderr methods. Documented emit_event's temporary use of serde_json (switches to borsh when BuildEvent gets derives).
New hm-plugin-macros proc-macro crate. Parses manifest + capability keyword args, generates __HmPluginImpl struct with RawPlugin impl and #[stabby::export] hm_load_plugin entry point. Replaces the old register_plugin!/extism-based declarative macros. Integration test validates all 4 capability types compile through the macro.
Replace the extism-based WASM plugin loading in crates/hm/ with stabby-based native shared-library (.dylib/.so/.dll) loading. Key changes: - host.rs: LoadedPlugin now wraps stabby's Library + Dyn<RawPlugin> trait object instead of extism's PluginPool. Typed async methods (execute_step, on_hook_event, run_subcommand, on_output_event, finalize_output) replace the generic call_capability<I,O>. - host_api.rs: New HostApiImpl struct implementing RawHostApi with all 11 FFI methods (log, kv_get/set, emit_event, emit_step_log, should_cancel, write_stdout/stderr, archive_read/total_size, fs_read_config). Minimal but compilable implementations. - registry.rs: Discovers .dylib/.so/.dll files instead of .wasm. Removed embedded/pool_sizes fields from RegistryConfig. Passes Arc<HostApiImpl> to each LoadedPlugin::load(). - manifest.rs: Removed MissingHostFn variant and host-fn validation (no longer needed with stabby trait objects). - paths.rs: Changed user_plugins_dir to ~/.harmont/plugins/. - Deleted build.rs (no more WASM compilation at build time). - Deleted embedded.rs (no more include_bytes! embedding). Remaining errors are all caller-side (dispatcher.rs, scheduler.rs, output_subscriber.rs, install.rs) referencing the removed old API. Those callers will be updated in Tasks 5-8.
Spec compliance cleanup for Task 4. These modules were kept alive by the previous commit but the spec requires deleting them now and accepting caller-side compilation errors until Tasks 5-8.
- Log warning on poisoned mutex in kv_set - Add TODO marker for nil step_id in emit_step_log - Clarify staticify_slice safety with cross-crate reference
Replace extism-pdk host functions with bollard Docker API calls. The plugin now drives the Docker daemon directly through bollard, streaming exec output via PluginContext::emit_step_log_stdout/stderr. Key changes: - Delete extism_host.rs; add docker.rs with bollard-based DockerClient - Rewrite lib.rs to use StepExecutor trait with PluginContext - Switch from register_plugin! (extism) to hm_plugin! (stabby) - Update scheduler.rs: replace call_capability with execute_step - Change RawPlugin FFI from DynFuture to DynFutureUnsync (futures holding bollard streams are Send but not Sync) - Update StepExecutor trait to use named lifetimes so ctx can be captured in async futures
Replace extism-pdk with native dylib approach:
- HTTP: extism_pdk::http::request -> async reqwest
- Credentials: host::keyring_* -> direct file I/O (~/.harmont/credentials.toml)
- KV storage: host::kv_* -> ctx.kv_get()/ctx.kv_set()
- Output: host::write_stdout/stderr -> ctx.write_stdout()/ctx.write_stderr()
- Logging: host::log -> ctx.log()
- OAuth loopback: host::spawn_loopback/loopback_recv -> axum one-shot server
- Browser: host::browser_open -> webbrowser::open()
- TTY prompts: host::tty_prompt -> dialoguer
- PKCE: insecure clock-based seed -> rand::thread_rng()
All functions are now async, threading &PluginContext<'_> through the
entire call chain. The SubcommandPlugin trait gains explicit 'a lifetime
(matching StepExecutor) so the future can capture ctx. The dispatcher
switches from call_capability("hm_subcommand_run") to run_subcommand().
The build watch loop uses tokio::time::sleep instead of busy-waiting.
Replace the single `crates/hm-fixtures` WASM crate with 6 individual cdylib crates under `tests/fixtures/`: - hm-fixture-noop-executor - hm-fixture-recording-hook - hm-fixture-failing-subcommand - hm-fixture-host-fn-probe - hm-fixture-bad-api-version - hm-fixture-freestyle-runner Each fixture now uses `hm_plugin!` macro, RPITIT trait impls with `PluginContext`, and compiles to a native shared library instead of WASM. The `LifecycleHook::on_event` trait signature is updated to use a named lifetime `'a` (matching `StepExecutor` and `SubcommandPlugin`) so hook impls can use `ctx` in their async blocks. Keyring host-fn tests are removed from host-fn-probe since `RawHostApi` no longer exposes keyring operations. Test call sites updated to use new hyphenated crate names.
Migrate all integration tests from the old extism/WASM API to the new stabby native dylib surface. Fix production callers (scheduler, install) that still referenced removed fields (embedded, pool_sizes, signal module, docker_host_fns module). - tests: use typed method calls instead of call_capability strings - tests: use HostApiImpl KV directly instead of filesystem probes - scheduler: inline ctrl-c handler, construct RegistryConfig with host_api - install: validate via LoadedPlugin::load instead of from_bytes - events: expose sender() for scheduler HostApiImpl construction - remove dead docker_host_fns module declaration
Strip host_abi.rs down to Level, KvScope, and ArchiveReadArgs — all other types (Socket*, Loopback*, Keyring*, Tty*, Docker*) were only needed for extism host-function calls and are dead code in the stabby native-dylib world. Remove `required_host_fns` and `allowed_hosts` from PluginManifest (extism-era concepts; native plugins call APIs directly). Bump HM_PLUGIN_API_VERSION from 1 to 2. Delete dead files: docker_host_fns.rs (module decl already removed), hm-plugin-sdk/src/host.rs (extism host wrappers, never referenced by stabby SDK). Remove Docker schema snapshot tests and their .snap files. Update all 16 manifest construction sites across plugin crates, test fixtures, and host-side tests.
- Remove extism/extism-pdk from workspace dependencies - Remove axum/webbrowser from hm (moved to cloud plugin) - Strip wasm32-wasip1 targets from CI/release/examples workflows - Remove dead "Build embedded WASM plugins" step from release.yml - Update CLAUDE.md, README.md, crates/hm/CLAUDE.md, crates/hm/README.md to describe stabby native dylib plugin architecture - Fix builtin/plugin.rs remove command to use DLL_EXTENSION not .wasm - Fix stale .wasm doc comments in cli.rs and install.rs - Remove "no Extism" from protocol crate doc comment (Extism is gone) - Mark cloud integration tests as #[ignore] (need plugins installed)
Resolve two modify/delete conflicts: - cli.rs: accept incoming cli/ directory split, apply stabby doc fixes - plugin/host_fns.rs: keep deleted (dead extism code) Auto-merged cli/external.rs correctly picked up stabby API from dispatcher.rs rename detection.
Relocate docker, cloud, output-human, and output-json plugin crates from crates/ top level into crates/hm/plugins/ for better organization. Protocol, SDK, and macros stay at crates/ level as shared infrastructure.
Port BuildEvent rendering logic from the output-human plugin into a new BuildEventRenderer struct in crates/hm/src/output/build_events.rs. The renderer owns its step-key map directly (HashMap<Uuid, String>) instead of using a static Mutex, and adds a render_json method for JSON-line output. All three original tests are ported and passing.
… plugins Replace the plugin-dispatch loop in output_subscriber with direct rendering via BuildEventRenderer. The scheduler now accepts OutputMode instead of a format name string, eliminating the runtime format validation against the plugin registry. Human output writes to stderr, JSON to stdout, matching the previous plugin behavior.
The OutputFormatter capability was removed in earlier commits. Delete the now-unused hm-plugin-output-human and hm-plugin-output-json crate directories, remove their commented-out workspace entries, and clean up stale references in docs (CLAUDE.md, README.md, crates/hm/README.md, RELEASING.md). Also fixes register_plugin! -> hm_plugin! in READMEs.
…ources Replace six plugin-specific HmError variants (PluginLoad, PluginManifest, PluginMissingHostFn, PluginPanic, PluginTimeout, PluginConflict) with a single PluginRuntime(#[from] RuntimeError) wrapper that delegates to hm-plugin-runtime. The category() method maps the inner variants to the same ErrorCategory values as before. crates/hm/src/plugin/mod.rs is now a re-export shim; the six original module files are deleted. Integration tests in plugin_manifest.rs updated to downcast to RuntimeError directly.
No more insert-then-check-conflict — Entry::Vacant/Occupied ensures the map is never silently overwritten. Indexes are built and returned rather than mutating self piecemeal.
# Conflicts: # crates/hm-plugin-docker/src/lib.rs # crates/hm-plugin-protocol/src/host_abi.rs # crates/hm/src/orchestrator/docker_host_fns.rs
Two-phase CLI parsing: load plugin registry, augment clap Command with plugin SubcommandSpecs, parse once, route built-in commands through derive types and plugin commands through clap_bridge extraction.
Replace clap-based CLI parsing inside the cloud plugin with JSON arg extraction from SubcommandInput. The host now parses all args via clap_bridge; the plugin just reads structured JSON. Full ArgSpec manifest declared so hm --help shows all cloud verbs. Removes clap dependency from the cloud plugin.
Plugin authors can define their CLI schema with clap derive macros and convert to SubcommandSpec automatically for the manifest.
…lper Replace hand-written ArgSpec manifest with a manifest_schema module containing clap derive types. spec_from_command introspects these at init time to produce the SubcommandSpec tree automatically.
- PluginRegistry::subcommand_specs() replaces free function in main.rs - Cli::command_with_plugins() replaces build_augmented_command() - Builtin subcommand names derived from Cli::command() introspection instead of hand-maintained BUILTIN_SUBCOMMANDS const
FfiValue is an ABI-stable recursive enum that replaces serde_json::Value at the plugin FFI boundary. Uses #[repr(C, u8)] instead of #[stabby::stabby] because stabby's IStable trait resolver overflows on recursive types (E0275). Inner types (stabby::string::String, stabby::vec::Vec) are still stabby-stable, so the layout is deterministic across cdylib boundaries.
This reverts commit cb36047.
Add borsh serialization support to all FFI-boundary types in hm-plugin-protocol, preparing for the switch from serde_json to borsh binary serialization across the plugin FFI. - Create `Value` enum (borsh + serde + schemars) replacing `serde_json::Value` in `SubcommandInput.args`, `CommandStep.runner_args`, and the `JsonSchema` type alias - Add `BorshSerialize`/`BorshDeserialize` to every protocol type - Add `borsh_helpers` module with custom serializers for `DateTime<Utc>`, `semver::Version`, and `char` (types without native borsh support) - Enable `borsh` feature on the `uuid` crate (native support) - Fix downstream callsites in hm-plugin-runtime, hm-plugin-cloud, and harmont-cli that constructed `SubcommandInput` with serde_json::Value
- Return &[Value] from as_array instead of &Vec<Value> - Remove unused serialize/deserialize_option_datetime helpers - Add borsh round-trip tests for all composed protocol types (BuildEvent, HookEvent, HookOutcome, ExecutorInput, StepResult, SubcommandInput, PluginManifest)
emit_event, archive_read, archive_total_size in PluginContext and the emit_event handler in HostApiImpl now use borsh serialization.
- Drop serde_json from hm-plugin-sdk, hm-plugin-docker, and 4 test fixtures that no longer use it - Delete obsolete stabby-ffi-types plan (superseded by borsh plan)
Contributor
Author
|
too much slop |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
libloading+ stabby ABI-stable trait objects. Eliminates WASM sandbox overhead, enables direct async code, and removes ~4000 lines of boilerplate.BuildEventRenderer). TheOutputFormattercapability, SDK trait, proc-macro codegen, and both output plugin crates are deleted.--formatis now a compile-timeclap::ValueEnumenum.docker,cloud) moved fromcrates/tocrates/hm/plugins/. Test fixtures moved totests/fixtures/as native cdylib crates.Key changes
hm-plugin-sdk: newRawPlugintrait (4 methods:manifest,execute_step,on_hook_event,run_subcommand) +RawHostApitrait (11extern "C"methods)hm-plugin-macros:hm_plugin!proc macro generates all stabby FFI gluehm-plugin-protocol: removed WASM-erahost_abitypes,OutputFormatterSpecoutput_subscriberrenders directly viaBuildEventRenderer, scheduler takesOutputModeenum instead of string=72.1.1for ABI stabilityTest plan
cargo check --workspacepassescargo test --workspacepasses (excluding pre-existing env-dependent failures)hm run --format humanand--format jsonproduce correct outputhm plugin list/hm plugin infoworkhm cloud login, etc.)