From 67107ea5032eb836db2eaf02844fa81298bbf58b Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 16:33:35 -0400 Subject: [PATCH 01/28] Add research docs for SyntaxKit-driven codegen CLI (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1 documents how Tuist evaluates user manifests via xcrun swift + token-delimited JSON over stdout, with citations into tuist/tuist and swiftlang/swift-package-manager. Phase 2 sketches the SyntaxKit equivalent: pure-DSL input files wrapped into a Group { … } closure before spawning swift, with a 7-step POC ladder that retires cold-start cost first. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/codegen-cli-design.md | 154 +++++++++++++ Docs/research/tuist-manifest-pipeline.md | 265 +++++++++++++++++++++++ 2 files changed, 419 insertions(+) create mode 100644 Docs/research/codegen-cli-design.md create mode 100644 Docs/research/tuist-manifest-pipeline.md diff --git a/Docs/research/codegen-cli-design.md b/Docs/research/codegen-cli-design.md new file mode 100644 index 0000000..cae86e1 --- /dev/null +++ b/Docs/research/codegen-cli-design.md @@ -0,0 +1,154 @@ +# Design sketch: a CLI for SyntaxKit-driven codegen + +> Phase 2 deliverable for [issue #154](https://github.com/brightdigit/SyntaxKit/issues/154). Builds on [`tuist-manifest-pipeline.md`](./tuist-manifest-pipeline.md). Nothing here is implemented — this is the design we'd validate with the POC in §6. + +## 1. What we're borrowing from Tuist — and what we're not + +From Phase 1, Tuist's manifest pipeline reduces to four moving parts: + +1. A **public DSL framework** that ships next to the CLI binary (`ProjectDescription.framework`). +2. A **script runner** that invokes `xcrun swift ` with `-I/-L/-F` pointing at that framework, captures stdout, and slices out a token-delimited payload. +3. A **helpers compiler** that pre-builds `Tuist/ProjectDescriptionHelpers/*.swift` into a sibling dylib so manifests can `import ProjectDescriptionHelpers`. +4. A **two-tier cache** (helpers module + decoded manifest) keyed on source hashes + toolchain/tool versions. + +We borrow (1), (2), (3), and (4). We **don't** borrow Tuist's "manifest" framing — no `Project.swift`-style wrapper, no `Output(...)` value, no token-delimited stdout payload. Tuist needs the wrapper because its host has to re-interpret the description into an `xcodeproj`. SyntaxKit doesn't: the input file is *pure DSL* — a series of `CodeBlock` expressions — and the CLI generates the boilerplate that turns it into a runnable Swift program. + +## 2. CLI shape + +The CLI is `stdin → stdout`-shaped, with a SyntaxKit-aware `swift` invocation as the engine: + +``` +syntaxkit run Input.swift # rendered Swift source to stdout +syntaxkit run Input.swift -o Output.swift # write to a file (atomic) +syntaxkit run InputDir/ -o OutputDir/ # walk InputDir/*.swift, mirror paths into OutputDir/ +``` + +**Input file:** pure DSL. A series of `CodeBlock` expressions, optionally preceded by `import` declarations. No `print`, no `@main`, no boilerplate. Example: + +```swift +// Person.swift +import SyntaxKit // optional — only needed for IDE / autocomplete + +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} +``` + +That's the entire file. Top-level expressions form an implicit `@CodeBlockBuilder` body that the CLI wraps for execution — see §3. + +**Output:** the rendered Swift source produced by `generateCode()`. The CLI does not reshape it. + +**Stderr:** forwarded to the user's terminal. The captured output is stdout-only. (The wrapper writes to stdout via a single `print`; if helpers or the user's DSL want to log debug info, they should use `FileHandle.standardError.write(...)`.) + +**Folder mode:** when the input is a directory, the CLI walks `**/*.swift` and produces a parallel tree of outputs (one input file → one output file, mirrored relative path). Files starting with `_` are skipped (convention for shared helpers — see §4). + +**Exit codes:** child `swift` non-zero → CLI non-zero with stderr preserved. No retries. + +## 3. Process model: wrap, then spawn + +Because the input is pure DSL rather than a runnable Swift program, the CLI does a tiny **wrap** step before spawning `swift`. + +**Wrap.** Read the input file. Use SwiftSyntax (already a dep — `Docs/SwiftSyntax-LLM.md`) to split the top of the file into (a) any leading `import` declarations and (b) the remaining body. Generate a temporary `Input.wrapped.swift`: + +```swift +import SyntaxKit + + +let __syntaxkit_root = Group { + +} +print(__syntaxkit_root.generateCode()) +``` + +`Group` (`Sources/SyntaxKit/Utilities/Group.swift`) already uses `@CodeBlockBuilder`, so its closure body accepts a series of `CodeBlock` expressions exactly the way the user wrote them. `import SyntaxKit` is always injected; duplicates from the input are harmless. + +**Spawn.** Adopt Tuist's model `(b)` from Phase 1 §2 — `swift` in script mode against the wrapped file, pipe through: + +``` +/usr/bin/env swift \ + -suppress-warnings \ + -I \ + -L \ + -F \ + -lSyntaxKit -framework SyntaxKit \ + -I … -L … -F … -l # optional, when helpers exist + /Input.wrapped.swift +``` + +No `--syntaxkit-dump` flag, no start/end tokens. The child's stdout is the rendered Swift source verbatim. The CLI's job is wrap → `Process` → capture stdout → atomic write to destination → clean up temp wrapper. + +Use `/usr/bin/env swift` rather than `/usr/bin/xcrun swift` so the same code path runs on Linux. `xcrun` is mac-only and is implicit when `env swift` resolves to Xcode's swift on macOS. + +**Why wrap instead of requiring `print()` in the input.** Two reasons. First, the user's authoring surface is *just* DSL — declarative, no I/O verbs. Second, error reporting: when the child `swift` reports a diagnostic at `Input.wrapped.swift:42`, the CLI can map that line back to the original `Input.swift` (the wrapper is line-faithful aside from a known prefix offset) and rewrite the path in stderr before forwarding. + +## 4. Helpers + +Same mechanism as Tuist (Phase 1 §4), folder name TBD — `Helpers/` adjacent to the input file or input directory is the obvious choice. The CLI walks up from the input path looking for a `Helpers/` directory, globs `**/*.swift` (excluding files prefixed with `_` to allow private helpers within helpers), and pre-compiles them into `lib.dylib` via: + +``` +swiftc -module-name SyntaxKitHelpers \ + -emit-module -emit-module-path /SyntaxKitHelpers.swiftmodule \ + -parse-as-library -emit-library \ + -suppress-warnings \ + -I … -L … -F … -lSyntaxKit -framework SyntaxKit \ + Helpers/**/*.swift +``` + +The output dylib is then added to the input script's invocation via `-I/-L/-F/-l`. Scripts can `import SyntaxKitHelpers` and use shared codegen utilities. + +Compile into a `tmp..` staging directory and atomic-rename into the cache path, mirroring `ProjectDescriptionHelpersBuilder.swift:204-244` — concurrent CLI invocations need to be safe. + +## 5. Caching + +Two layers, both mirroring Tuist (Phase 1 §5). + +**Helpers cache.** Keyed by: + +| Field | Source | +| --- | --- | +| per-file SHA-256s | `Helpers/**/*.swift`, sorted | +| `syntaxkitVersion` | bundled SyntaxKit dylib version | +| `swiftlangVersion` | `swift --version` | +| `osVersion` | `uname -r` (macOS or Linux) | +| `cacheSchemaVersion` | bumped on layout changes | + +Hash → directory name → reuse if present. + +**Output cache.** Skip the swift spawn entirely when nothing has changed. Keyed by: + +| Field | Source | +| --- | --- | +| `inputHash` | SHA-256 of the input `.swift` file | +| `helpersHash` | the helpers cache key above, or empty | +| `syntaxkitVersion` | bundled SyntaxKit dylib version | +| `swiftlangVersion` | `swift --version` | +| `envHash` | md5 of `SYNTAXKIT_*` env vars | +| `cacheSchemaVersion` | bumped on layout changes | + +On hit, copy the cached rendered output directly to the destination — no `swift` spawn. On miss, run and re-cache. + +Cache location: `~/.cache/syntaxkit/` on Linux, `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS (XDG-aware via `XDG_CACHE_HOME`). + +## 6. Smallest possible proof-of-concept steps + +Each step is independently shippable and de-risks the next. + +1. **Hand-driven wrap + spawn.** Hand-write a pure-DSL `Input.swift` and a hand-rolled `Input.wrapped.swift` that imports SyntaxKit, splices the body into `Group { … }`, and prints `generateCode()`. Invoke it manually with `swift Input.wrapped.swift -I … -L … -F … -lSyntaxKit -framework SyntaxKit` against a local `swift build` of SyntaxKit. **Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit at all, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost.** Cold-start is the single biggest risk to retire — if it's 20+ seconds, the design needs rethinking. +2. **CLI subcommand for single-file mode.** Add `syntaxkit run `: parse out top-level imports with SwiftSyntax, write `Input.wrapped.swift` to a temp dir, spawn `swift` via `Foundation.Process`, capture stdout, write to `-o ` (or stdout if no `-o`). Stderr is forwarded; rewrite `Input.wrapped.swift:LINE` references back to `Input.swift:LINE` before forwarding. +3. **Folder mode.** Walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`. Add `_`-prefix skip rule. Parallelize cautiously (`swift` spawns aren't free — gate on a small concurrency limit, maybe `ProcessInfo.activeProcessorCount`). +4. **Ship a bundled-binary release.** Build SyntaxKit + SwiftSyntax dylibs, drop them in `lib/` next to the CLI binary, write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). Now the CLI is self-contained — no `swift build` required at the call site. +5. **Helpers directory.** Discovery + compile + flag-splicing. +6. **Output cache.** Add the cache, keyed as in §5. Add `--no-cache` for debugging. +7. **Linux smoke test.** Confirm `/usr/bin/env swift` works on Linux with the bundled dylib layout (no framework search path, but `-I + -L + -lSyntaxKit` should be sufficient, matching Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch). + +## 7. What we still need to verify + +- **Cold-start cost.** Single biggest unknown. Step 1 of §6 answers this. +- **Splice fidelity.** When the input body lives inside a `Group { … }` closure, is everything users naturally write in the DSL still legal? Result-builder closures don't allow `import`, top-level type decls, or top-level `let`/`var` outside the builder DSL. The wrapper hoists `import`s; we need to confirm there's no other top-level construct users would reasonably write that the wrap step would break. Verify in step 1 with a few realistic inputs (large struct, nested types, conditionals via `if`-in-builder). +- **`Process` stdout/stderr separation.** Tuist captures stdout-only and merges stderr via `CommandError`. Foundation's `Process` has the same split — confirm it doesn't interleave under load, and confirm the CLI doesn't accidentally swallow stderr. +- **`swift` script-mode quirks.** `swift ` runs in interpret/`-frontend -interpret` mode. Some features (`@main`, certain attributes) behave differently than in compile mode. Top-level `print` statements are fine. Verify in step 1. +- **SwiftSyntax linkage stability across toolchains.** SwiftSyntax pins to specific Swift toolchain versions. The bundled dylib is built against a particular toolchain; if the user's `swift` is from a newer or older release, ABI breakage is possible. Mitigation: cache key includes `swiftlangVersion`, plus a clear error when the gap is too wide. +- **What if a single input script needs to produce multiple files?** Out of scope for v1. Split into multiple inputs and use folder mode. If demand materializes, we can layer in a `--multi` envelope mode (a script writes a small JSON manifest to stdout, CLI fans out to multiple files) without breaking single-file semantics. +- **Sandboxing.** Out of scope for v1. Input scripts are user-owned code in their own repo — running them has the same threat model as running `swift Input.swift` by hand. Revisit if/when this CLI runs untrusted scripts (CI for OSS contributions, etc.). +- **Timeout.** Add one. Tuist's omission (Phase 1 §7) is a bug, not a feature. 60s default `Process` wait with `SIGTERM` → 5s grace → `SIGKILL`. Override via `--timeout `. diff --git a/Docs/research/tuist-manifest-pipeline.md b/Docs/research/tuist-manifest-pipeline.md new file mode 100644 index 0000000..313464b --- /dev/null +++ b/Docs/research/tuist-manifest-pipeline.md @@ -0,0 +1,265 @@ +# How Tuist Evaluates Manifests + +> Phase 1 deliverable for [issue #154](https://github.com/brightdigit/SyntaxKit/issues/154). +> Source: read of `tuist/tuist@main` and `swiftlang/swift-package-manager@main` via GitHub. Every behavioral claim cites a file/line in those repos. + +## TL;DR + +- **Manifest is run, not pre-compiled to an artifact.** Tuist invokes `/usr/bin/xcrun swift ` in Swift script/interpreter mode with `-I/-L/-F` pointing at `ProjectDescription.framework` (`cli/Sources/TuistLoader/Loaders/ManifestLoader.swift:374-528`). There is no separate `.o`/executable for the manifest itself. +- **JSON over stdout, token-delimited.** The manifest's `Project(...)` initializer calls `dumpIfNeeded(self)` (`cli/Sources/ProjectDescription/Dump.swift:3-14`) which `JSONEncoder`s `self` and prints it bracketed by `TUIST_MANIFEST_START` / `TUIST_MANIFEST_END`; the host scans stdout for those tokens and `JSONDecoder`s the slice between them (`ManifestLoader.swift:119-120, 347-365`). +- **Helpers are real dylibs.** Files under `/Tuist/ProjectDescriptionHelpers/` are compiled by a separate `swiftc -emit-module -emit-library -parse-as-library` invocation into `lib.dylib`, cached on disk, and added to the manifest's `swift` invocation via `-I/-L/-F/-l` (`cli/Sources/TuistLoader/ProjectDescriptionHelpers/ProjectDescriptionHelpersBuilder.swift:174-298`). +- **Two-layer cache.** The helpers module is keyed by an md5 of (file SHA-256s + Tuist version + Swift toolchain version + macOS version + macOS SDK version + tuist env vars + DEBUG flag) (`ProjectDescriptionHelpersHasher.swift:34-55`). The decoded manifest JSON itself is cached in `~/.tuist/Cache/Manifests`, keyed by (manifest SHA-256 + helpers hash + plugins hash + env hash + sandbox flag + Tuist version + cache schema version) (`CachedManifestLoader.swift:181-275`). +- **Sandboxed.** On macOS the whole `xcrun swift …` command is wrapped in `sandbox-exec -p ` by default; the profile denies everything and re-allows only read access to the project path, Xcode, the `ProjectDescription` search paths, and `/private/tmp` + `/private/var` for writes (`ManifestLoader.swift:492-574`). + +## 1. Compilation pipeline + +The manifest is **not** compiled by Tuist to a separate artifact — it is run with `swift` in interpreter mode. Argument construction lives in `ManifestLoader.buildArguments(_:at:disableSandbox:)`: + +```swift +// cli/Sources/TuistLoader/Loaders/ManifestLoader.swift:392-401 +var arguments = [ + "/usr/bin/xcrun", + "swift", + "-suppress-warnings", + "-I", searchPaths.includeSearchPath.pathString, + "-L", searchPaths.librarySearchPath.pathString, + "-F", searchPaths.frameworkSearchPath.pathString, + "-l\(frameworkName)", + "-framework", frameworkName, +] +``` + +Then helpers are stitched in (`ManifestLoader.swift:423-428`): + +```swift +let projectDescriptionHelperArguments = projectDescriptionHelperModules.flatMap { [ + "-I", $0.path.parentDirectory.pathString, + "-L", $0.path.parentDirectory.pathString, + "-F", $0.path.parentDirectory.pathString, + "-l\($0.name)", +] } +``` + +The manifest path itself is appended last (`ManifestLoader.swift:478`), and `--tuist-dump` is added by the caller (`ManifestLoader.swift:344`). There is **no** `-target`, `-sdk`, `-module-name`, `-emit-executable`, or `-emit-library` on the manifest run — those are SwiftPM-style flags that Tuist deliberately avoids for the manifest. The implicit target/SDK come from `xcrun swift` itself (i.e. from the active Xcode selected via `xcode-select`). + +**For `Package.swift` (the `.packageSettings` case)**, Tuist additionally points `-I/-L/-F` at `XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI`, links `-lPackageDescription`, and passes `-package-description-version -D TUIST` and a JSON-encoded `-context ` argument that SwiftPM's `PackageDescription` runtime expects (`ManifestLoader.swift:430-490`). + +**Helpers compilation** uses `swiftc` directly (`ProjectDescriptionHelpersBuilder.swift:264-298`): + +```swift +var command: [String] = [ + "/usr/bin/xcrun", "swiftc", + "-module-name", moduleName, + "-emit-module", + "-emit-module-path", outputDirectory.appending(component: "\(moduleName).swiftmodule").pathString, + "-parse-as-library", + "-emit-library", + "-suppress-warnings", + "-I", projectDescriptionSearchPaths.includeSearchPath.pathString, + "-L", projectDescriptionSearchPaths.librarySearchPath.pathString, + "-F", projectDescriptionSearchPaths.frameworkSearchPath.pathString, + "-working-directory", outputDirectory.pathString, +] +// + helper-module flags + `-framework ProjectDescription` (or `-lProjectDescription` when dylib) + all *.swift sources +``` + +The helper artifact is a dylib named `lib.dylib` (`ProjectDescriptionHelpersBuilder.swift:188-190`). + +## 2. Execution model + +It's model **(b) — `Process`-spawn + parse stdout**. There is no `dlopen`, no exported C symbol, no plugin entry point. `loadDataForManifest` simply runs the constructed `arguments` via `CommandRunner.capture(...)`: + +```swift +// cli/Sources/TuistLoader/Loaders/ManifestLoader.swift:347-365 +let string = try await commandRunner.capture( + arguments: arguments, + environment: Environment.current.manifestLoadingVariables +) + +guard let startTokenRange = string.range(of: ManifestLoader.startManifestToken, options: .literal), + let endTokenRange = string.range(of: ManifestLoader.endManifestToken, options: [.literal, .backwards]) +else { + return string.data(using: .utf8)! +} +// ... slice out the JSON ... +let manifest = string[startTokenRange.upperBound ..< endTokenRange.lowerBound] +return manifest.data(using: .utf8)! +``` + +The host scans the entire captured stdout for `TUIST_MANIFEST_START` and `TUIST_MANIFEST_END` (`ManifestLoader.swift:119-120`). Anything *before* the start token and *after* the end token is treated as user-visible log output and re-emitted via `Logger.current.notice(...)` (`ManifestLoader.swift:358-362`). This means a manifest can `print(…)` debug output *and* still be parsed correctly — a useful side-effect of the token-delimited design. + +The slice between the tokens is utf8-decoded and passed straight to `JSONDecoder().decode(T.self, from: data)` (`ManifestLoader.swift:256-257`). + +## 3. Bridging format + +JSON, generated by `JSONEncoder` on the manifest side and `JSONDecoder` on the host side. + +**Encode site** (manifest process): `cli/Sources/ProjectDescription/Dump.swift:3-14`: + +```swift +func dumpIfNeeded(_ entity: some Encodable) { + guard !ProcessInfo.processInfo.arguments.isEmpty, + ProcessInfo.processInfo.arguments.contains("--tuist-dump") + else { return } + let encoder = JSONEncoder() + let data = try! encoder.encode(entity) + let manifest = String(data: data, encoding: .utf8)! + print("TUIST_MANIFEST_START") + print(manifest) + print("TUIST_MANIFEST_END") +} +``` + +`dumpIfNeeded(self)` is invoked from inside the *initializer* of each top-level manifest type. For example, `Project.init(...)` ends with `dumpIfNeeded(self)` (`cli/Sources/ProjectDescription/Project.swift:109`). This is what makes a bare `let _ = Project(...)` self-publishing — the user never has to call anything; constructing the value at the top level *is* the side-effect. + +**Decode site** (host process): `cli/Sources/TuistLoader/Loaders/ManifestLoader.swift:127, 175, 257`: + +```swift +private let decoder: JSONDecoder +… +return try decoder.decode(T.self, from: data) +``` + +All top-level `ProjectDescription` types (`Project`, `Workspace`, `Config`, `Template`, `Plugin`, `PackageSettings`) are `Codable` — that's what allows the same `Decodable` constraint on `loadManifest(...)` (`ManifestLoader.swift:244`) to handle every manifest kind. + +Note the asymmetry: the manifest side imports `ProjectDescription` and produces `ProjectDescription.Project`; the host side decodes into the **same** `ProjectDescription.Project` Swift type (`ManifestLoading.loadProject` returns `ProjectDescription.Project`), and only later does `ManifestModelConverter` / `Project+ManifestMapper.swift` map that into `XcodeGraph.Project` (the internal model). So `ProjectDescription` plays a dual role: API surface for users + DTO/IR for the bridge. + +## 4. ProjectDescriptionHelpers + +The flow is: locate → hash → compile-or-reuse → splice flags into the manifest `swift` command. + +**Locate** (`HelpersDirectoryLocator.swift:37-44`): + +```swift +public func locate(at: AbsolutePath) async throws -> AbsolutePath? { + guard let rootDirectory = try await rootDirectoryLocator.locate(from: at) else { return nil } + let helpersDirectory = rootDirectory + .appending(component: Constants.tuistDirectoryName) // "Tuist" + .appending(component: Constants.helpersDirectoryName) // "ProjectDescriptionHelpers" + if try await !fileSystem.exists(helpersDirectory) { return nil } + return helpersDirectory +} +``` + +So the discovery rule is literally: walk up from the manifest's directory to the project root, then look for `Tuist/ProjectDescriptionHelpers/`. If absent, helpers are skipped. + +**Build** (`ProjectDescriptionHelpersBuilder.swift:174-255`): each helpers directory is hashed (see section 5), the hash becomes a subdirectory name under the on-disk cache, and the build is gated on whether that directory already exists. If not, files are globbed (`**/*.swift`), `swiftc` runs into a sibling **staging directory** (`.tmp..`) and is then atomically `rename(2)`'d into place (`ProjectDescriptionHelpersBuilder.swift:204-244`) — explicitly designed to be safe under concurrent Tuist processes. + +**Plugin helpers** (`ProjectDescriptionHelpersBuilder.swift:132-144`) get built first because local helpers may import plugin helpers but not vice-versa. The plugin helper modules are then passed in as `customProjectDescriptionHelperModules` to the local helpers compile, so the local helpers can `import `. + +**Splicing into the manifest invocation** (`ManifestLoader.swift:405-428`): for `.project`, `.template`, `.workspace`, `.packageSettings` manifests, the helpers builder is called; for `.config`, `.plugin`, `.package`, helpers are *not* built (so you can't `import ProjectDescriptionHelpers` from `Tuist.swift`/`Plugin.swift` — the loader logs a clear error if you try, see `logUnexpectedImportErrorIfNeeded`, `ManifestLoader.swift:576-592`). + +In-process the builder also memoizes via `builtHelpers: ThreadSafe<[AbsolutePath: Task<…>]>` (`ProjectDescriptionHelpersBuilder.swift:69, 180-252`), so within one Tuist invocation a helpers directory is compiled at most once even if many manifests are loaded. + +## 5. Caching + +There are **two** caches; both contribute to the "manifest didn't change → skip work" property. + +**Helpers cache** (`ProjectDescriptionHelpersHasher.swift:34-55`): + +```swift +let fileHashes = try await fileSystem + .glob(directory: helpersDirectory, include: ["**/*.swift"]) + .collect() + .sorted() + .compactMap { $0.sha256() } + .compactMap { $0.compactMap { byte in String(format: "%02x", byte) }.joined() } +let tuistEnvVariables = Environment.current.manifestLoadingVariables.map { "\($0.key)=\($0.value)" }.sorted() +let swiftlangVersion = try await SwiftVersionProvider.current.swiftlangVersion() +let macosVersion = machineEnvironment.macOSVersion +let macosSDKVersion = try await MacOSSDKVersionProvider.current.macOSSDKVersion() +… +let identifiers = + [macosVersion, macosSDKVersion, swiftlangVersion, tuistVersion] + fileHashes + tuistEnvVariables + ["\(debug)"] +return identifiers.joined(separator: "-").md5 +``` + +So the helpers cache key is sensitive to: per-file SHA-256s (sorted), macOS version, macOS SDK version, Swift compiler/toolchain version, Tuist version, every `TUIST_*` env var, and DEBUG vs release Tuist build. The on-disk location is `cacheDirectoriesProvider.cacheDirectory(for: .projectDescriptionHelpers)` (`ManifestLoader.swift:402-403`), with the hash as the directory name — defaults under `~/.cache/tuist/…` on macOS / Linux per Tuist's CacheDirectoriesProvider conventions. + +**Decoded-manifest cache** (`CachedManifestLoader.swift:181-275`): after the manifest has been compiled-and-run once, the resulting JSON-encoded `ProjectDescription.Project` value is itself cached at `cacheDirectoriesProvider.cacheDirectory(for: .manifests)` (default `~/.tuist/Cache/Manifests` per the class doc, `CachedManifestLoader.swift:15`), keyed by: + +```swift +// CachedManifestLoader.swift:192-198 +return Hashes( + manifestHash: manifestHash, // SHA-256 of Project.swift + helpersHash: helpersHash, // md5 from ProjectDescriptionHelpersHasher + pluginsHash: try await pluginsHashCache.value?.value, + environmentHash: environmentHash, // md5 of TUIST_* env vars + disableSandboxHash: disableSandboxHash +) +``` + +Plus a `cacheVersion` integer (`CachedManifest.currentCacheVersion = 1`, `CachedManifestLoader.swift:314`) and the Tuist version (`CachedManifestLoader.swift:268-273`) — all five must match for a cache hit. On a hit, Tuist skips spawning `swift` entirely and decodes the cached JSON directly. + +## 6. Toolchain resolution + +Two paths. + +**For `swift` / `swiftc`**: Tuist hard-codes `/usr/bin/xcrun` (`ManifestLoader.swift:393`, `ProjectDescriptionHelpersBuilder.swift:267`) and lets xcrun resolve the toolchain. So the active Xcode (set by `xcode-select -s` or by `DEVELOPER_DIR`) determines `swift`/`swiftc`. Tuist does not appear to honor a `TOOLCHAINS=` or a custom `.xctoolchain` indirectly beyond whatever xcrun itself does. + +**For the SDK / Xcode path** (only needed for `.packageSettings`): `ManifestLoader.swift:432-457`: + +```swift +let xcodePath = try await { + if let developerDir = Environment.current.variables["DEVELOPER_DIR"] { + let developerDirPath = try AbsolutePath(validating: developerDir) + let resolvedXcodePath = if developerDirPath.components.suffix(2) == ["Contents", "Developer"] { + developerDirPath.parentDirectory.parentDirectory + } else { + developerDirPath + } + let manifestPath = resolvedXcodePath + .appending(components: "Contents", "Developer", "Toolchains", + "XcodeDefault.xctoolchain", "usr", "lib", "swift", "pm", "ManifestAPI") + if try await fileSystem.exists(manifestPath) { + return resolvedXcodePath + } + } + return try await XcodeController.current.selected().path +}() +``` + +So `DEVELOPER_DIR` is consulted first (with normalization for either form — pointing at `Xcode.app` or at `Xcode.app/Contents/Developer`), falling back to whatever `xcode-select` reports. + +**For `ProjectDescription.framework` / `libProjectDescription.dylib`**: `ResourceLocator.frameworkPath` (`cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`) searches relative to the Tuist binary's `Bundle.bundleURL`, including a Homebrew-style `bin/` ↔ `lib/` adjacency, and honors `TUIST_FRAMEWORK_SEARCH_PATHS` (space-separated). It will accept any of `libProjectDescription.dylib`, `ProjectDescription.framework`, or `PackageFrameworks/ProjectDescription.framework` — and `ProjectDescriptionSearchPaths.pathStyle(for:)` (`cli/Sources/TuistLoader/Utils/ProjectDescriptionPaths.swift:85-93`) decides whether downstream include/library/framework search paths should be derived dylib-style or framework-style. + +## 7. Failure modes + +- **Syntax error / compile failure**: `swift` exits non-zero, `CommandRunner` throws `CommandError.terminated(exitCode, standardError, command)`, and the host wraps it. Two hooks (`logUnexpectedImportErrorIfNeeded`, `logPluginHelperBuildErrorIfNeeded`, `ManifestLoader.swift:576-604`) inspect stderr and surface friendlier errors for common cases (e.g. importing `ProjectDescriptionHelpers` from `Config.swift`/`Plugin.swift`, or a missing plugin helper module). +- **Runtime crash / non-zero exit**: same path — `CommandRunner.capture(...)` throws, and the loader bubbles it up. +- **No start/end tokens in stdout**: the loader does **not** error; it falls back to treating the entire stdout as the JSON payload (`ManifestLoader.swift:352-356`). The JSON decode will then fail and produce a `ManifestLoaderError.manifestLoadingFailed` with the captured payload included for debugging (`ManifestLoader.swift:259-265`). +- **Decode failure**: each `DecodingError` case (`typeMismatch`, `valueNotFound`, `keyNotFound`, `dataCorrupted`) is mapped to a tailored `manifestLoadingFailed(context:)` error message that includes the offending JSON (`ManifestLoader.swift:267-315`). +- **Missing import / unresolved module**: stderr from `swift` carries the message; the two `logXxxErrorIfNeeded` functions detect the most common culprit (default helpers from a forbidden manifest, or a plugin helper that failed its own build) and emit a targeted log line. Beyond that the raw compile error reaches the user. +- **Hang / timeout**: no process-wait timeout in `ManifestLoader` or `CommandRunner` usage. Tuist appears to wait indefinitely for the `swift` subprocess to finish. (Inferred — `grep` for `timeout|terminate|sleep` in the loader files surfaced nothing.) +- **Empty `Package.swift`** is special-cased: `loadPackageSettings` catches `manifestLoadingFailed` with `data.count == 0` and returns a default `PackageSettings()` (`ManifestLoader.swift:217-230`). +- **Sandbox**: by default on macOS, the `swift` call is wrapped in `sandbox-exec -p ` (`ManifestLoader.swift:516-521`). The profile (`ManifestLoader.swift:547-574`) denies by default, then allows `process*`, `file-read-metadata`, RW under `/private/tmp` and `/private/var`, and read-only access to: the project path, the selected Xcode path, `com.apple.dt.Xcode.plist`, the three ProjectDescription search paths, the resolved `DEVELOPER_DIR`, and every helpers-module parent directory. Manifests can opt out via `disableSandbox: true` (config and plugin always run unsandboxed: `ManifestLoader.swift:195, 207, 234`). + +## 8. SwiftPM comparison + +SwiftPM uses model **(b)** as well but goes one extra step: it actually compiles the manifest into a **temporary executable on disk** and then `Process`-spawns *that* (`Sources/PackageLoading/ManifestLoader.swift:779-908` in `swiftlang/swift-package-manager`). In `evaluateManifest(...)`: + +```swift +// swift-package-manager/Sources/PackageLoading/ManifestLoader.swift:779-786 +let compiledManifestFile = tmpDir.appending("\(packageIdentity)-manifest\(executableSuffix)") +cmd += ["-o", compiledManifestFile.pathString] +``` + +then later (`:840, :906-908`): + +```swift +var runCmd = [compiledManifestFile.pathString] +… +runResult = try await AsyncProcess.popen(arguments: runCmd, environment: environment) +``` + +The framework search path for `PackageDescription` comes from `self.toolchain.swiftPMLibrariesLocation.manifestLibraryPath` and is added with `-F -Xlinker -rpath -Xlinker -framework PackageDescription` (or the `-L/-lPackageDescription/-rpath` dylib variant), plus an `-target` derived from `swiftPMLibrariesLocation.manifestLibraryMinimumDeploymentTarget` (`Sources/PackageLoading/ManifestLoader.swift:721-755`). Tuist by contrast (a) skips the explicit `-target` (lets `swift` interpreter pick its default), (b) does not write an intermediate executable to disk, (c) does not use `-Xlinker -rpath` for the manifest, and (d) wraps the whole thing in `sandbox-exec`. The net effect is the same — JSON on stdout, host decodes — but Tuist's path is one process and one disk-write fewer per manifest load (offset by the heavier helpers cache). + +The framework-search-path mechanics are the most directly transferable to a SyntaxKit equivalent: both tools resolve a *bundled, versioned* library that ships next to the CLI (Tuist's `ResourceLocator`, SwiftPM's `swiftPMLibrariesLocation`) and pass it via `-I/-L/-F + -framework ` (or `-l` for dylib distributions) to a single Swift compiler invocation that *also* takes the user's manifest path as a positional argument. + +## Open questions / things I couldn't determine + +- **Exact `CommandRunner.capture` semantics.** Call sites read but not implementation; *assuming* it returns stdout — the token-scanning code clearly operates on stdout-style content, but it isn't verified whether `print(…)` logs from user code (stdout) and compiler errors (stderr) end up in the same string. Inferred from `logUnexpectedImportErrorIfNeeded`'s use of `CommandError.terminated(_, standardError, command)` (`ManifestLoader.swift:577`) that stderr is captured separately. +- **Timeout.** No `terminate(after:)` / `timeout:` wiring around the manifest-spawn. If the user manifest infinite-loops, Tuist appears to wait forever. Not 100% certain because the `Command` package is a separate dependency not opened here. +- **Linux behavior.** The sandbox branch is `#if os(macOS)`; on Linux the manifest runs un-sandboxed (`ManifestLoader.swift:522-524`). Did not check whether `ResourceLocator` finds the framework on Linux or only the dylib — the `ProjectDescriptionSearchPaths.Style.commandLine` branch (which derives `-I` from a sibling `Modules/` directory) suggests Linux relies on `libProjectDescription.dylib + Modules/ProjectDescription.swiftmodule`. +- **Module cache.** SwiftPM passes `-module-cache-path` (`swift-package-manager Sources/PackageLoading/ManifestLoader.swift:759-761`); Tuist does **not** set one for the manifest invocation, so the manifest run inherits the default global Swift module cache. Significance for SyntaxKit's use case not chased. +- **`Config.swift` / `Tuist.swift` special path.** Both fall through `loadConfig` → `loadManifest(.config, …, disableSandbox: true)` → same `swift` invocation, but skip helpers entirely (`ManifestLoader.swift:407-408`). Not verified whether `Tuist.swift` accepts a different framework name (the switch in `buildArguments` lumps `.config` under `frameworkName = "ProjectDescription"`). From e4925038ac5847ed6b8a59ad9fe026c157c586de Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 16:40:49 -0400 Subject: [PATCH 02/28] POC step 1: wrap+spawn flow works at ~720ms cold (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran the hand-driven wrap+spawn flow against a temporarily-dynamic SyntaxKit dylib. Pure-DSL input spliced into a Group {…} wrapper compiles cleanly under xcrun swift with -I/-L/-l + an -Xcc include for SwiftSyntax's C shims. Cold start 0.72s, warm 0.11s. Dylib weight 25MB debug. Hoisted imports work; `if`-in-Group hits a type-checker crash that's a separate SyntaxKit bug to file. Design doc updated with the -Xcc requirement, the rpath flag, the bundled-binary layout's new C-shims include directory, and the POC findings in §7. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/codegen-cli-design.md | 9 +- Docs/research/poc-step1-results.md | 138 ++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+), 2 deletions(-) create mode 100644 Docs/research/poc-step1-results.md diff --git a/Docs/research/codegen-cli-design.md b/Docs/research/codegen-cli-design.md index cae86e1..c2d3bd5 100644 --- a/Docs/research/codegen-cli-design.md +++ b/Docs/research/codegen-cli-design.md @@ -72,10 +72,14 @@ print(__syntaxkit_root.generateCode()) -L \ -F \ -lSyntaxKit -framework SyntaxKit \ + -Xcc -I -Xcc \ + -Xlinker -rpath -Xlinker \ -I … -L … -F … -l # optional, when helpers exist /Input.wrapped.swift ``` +The `-Xcc -I -Xcc <…>` flag is non-obvious but required (POC step 1 finding): SyntaxKit transitively depends on `_SwiftSyntaxCShims` whose module map lives in `swift-syntax/Sources/_SwiftSyntaxCShims/include/`. The bundled-binary release must ship this header directory alongside the dylib and the SwiftSyntax `.swiftmodule` files. Without the flag, the script compile fails with `missing required module '_SwiftSyntaxCShims'`. The `-Xlinker -rpath -Xlinker <…>` flag tells the just-built script's dylib loader where to find `libSyntaxKit.dylib` at runtime. + No `--syntaxkit-dump` flag, no start/end tokens. The child's stdout is the rendered Swift source verbatim. The CLI's job is wrap → `Process` → capture stdout → atomic write to destination → clean up temp wrapper. Use `/usr/bin/env swift` rather than `/usr/bin/xcrun swift` so the same code path runs on Linux. `xcrun` is mac-only and is implicit when `env swift` resolves to Xcode's swift on macOS. @@ -137,14 +141,15 @@ Each step is independently shippable and de-risks the next. 1. **Hand-driven wrap + spawn.** Hand-write a pure-DSL `Input.swift` and a hand-rolled `Input.wrapped.swift` that imports SyntaxKit, splices the body into `Group { … }`, and prints `generateCode()`. Invoke it manually with `swift Input.wrapped.swift -I … -L … -F … -lSyntaxKit -framework SyntaxKit` against a local `swift build` of SyntaxKit. **Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit at all, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost.** Cold-start is the single biggest risk to retire — if it's 20+ seconds, the design needs rethinking. 2. **CLI subcommand for single-file mode.** Add `syntaxkit run `: parse out top-level imports with SwiftSyntax, write `Input.wrapped.swift` to a temp dir, spawn `swift` via `Foundation.Process`, capture stdout, write to `-o ` (or stdout if no `-o`). Stderr is forwarded; rewrite `Input.wrapped.swift:LINE` references back to `Input.swift:LINE` before forwarding. 3. **Folder mode.** Walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`. Add `_`-prefix skip rule. Parallelize cautiously (`swift` spawns aren't free — gate on a small concurrency limit, maybe `ProcessInfo.activeProcessorCount`). -4. **Ship a bundled-binary release.** Build SyntaxKit + SwiftSyntax dylibs, drop them in `lib/` next to the CLI binary, write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). Now the CLI is self-contained — no `swift build` required at the call site. +4. **Ship a bundled-binary release.** Build SyntaxKit as a `type: .dynamic` library, then bundle the `libSyntaxKit.dylib` + every `.swiftmodule` SyntaxKit publicly re-exports (SwiftSyntax + version-suffixed variants, SwiftOperators, SwiftParser) + the `_SwiftSyntaxCShims/include/` header dir in a `lib/` directory next to the CLI binary. Write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). POC step 1 confirmed cold-start with this layout is ~720ms. Debug dylib size is ~25 MB — re-measure under release config. 5. **Helpers directory.** Discovery + compile + flag-splicing. 6. **Output cache.** Add the cache, keyed as in §5. Add `--no-cache` for debugging. 7. **Linux smoke test.** Confirm `/usr/bin/env swift` works on Linux with the bundled dylib layout (no framework search path, but `-I + -L + -lSyntaxKit` should be sufficient, matching Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch). ## 7. What we still need to verify -- **Cold-start cost.** Single biggest unknown. Step 1 of §6 answers this. +- **Cold-start cost.** ~~Single biggest unknown.~~ Answered by POC step 1: ~720ms cold, ~110ms warm. See [`poc-step1-results.md`](./poc-step1-results.md). +- **SyntaxKit `if`-in-`Group` compiler crash.** POC step 1 surfaced this: `CodeBlockBuilderResult` claims `buildEither`/`buildOptional` support but conditionals trigger a type-checker failure-to-diagnose. Independent of the CLI design but blocks users writing conditional codegen. File as a separate SyntaxKit bug. - **Splice fidelity.** When the input body lives inside a `Group { … }` closure, is everything users naturally write in the DSL still legal? Result-builder closures don't allow `import`, top-level type decls, or top-level `let`/`var` outside the builder DSL. The wrapper hoists `import`s; we need to confirm there's no other top-level construct users would reasonably write that the wrap step would break. Verify in step 1 with a few realistic inputs (large struct, nested types, conditionals via `if`-in-builder). - **`Process` stdout/stderr separation.** Tuist captures stdout-only and merges stderr via `CommandError`. Foundation's `Process` has the same split — confirm it doesn't interleave under load, and confirm the CLI doesn't accidentally swallow stderr. - **`swift` script-mode quirks.** `swift ` runs in interpret/`-frontend -interpret` mode. Some features (`@main`, certain attributes) behave differently than in compile mode. Top-level `print` statements are fine. Verify in step 1. diff --git a/Docs/research/poc-step1-results.md b/Docs/research/poc-step1-results.md new file mode 100644 index 0000000..1958b44 --- /dev/null +++ b/Docs/research/poc-step1-results.md @@ -0,0 +1,138 @@ +# POC Step 1 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 1. Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost. + +## TL;DR — Approach is viable + +- **Cold start: ~720ms** real wall-clock for `xcrun swift Input.wrapped.swift -lSyntaxKit …` on M-series macOS with the dylib + module files unbacked by the OS page cache. +- **Warm: ~110ms** for subsequent runs. +- **Pure-DSL `Input.swift` spliced into `Group { … }`** in a generated wrapper compiles and runs end-to-end, producing the expected Swift source. +- **Bundled-dylib distribution is real and viable.** The only flags the CLI has to assemble are `-I`, `-L`, `-lSyntaxKit`, `-Xlinker -rpath -Xlinker ` and one new requirement: `-Xcc -I -Xcc ` (see §3 finding). +- **SyntaxKit dylib weight: 25.3 MB** in debug. Release build will be smaller. The CLI release artifact is dominated by this and the SwiftSyntax `.swiftmodule` files. + +## 1. What was run + +Built the dylib by temporarily flipping the SyntaxKit library product to `type: .dynamic` in `Package.swift`, ran `swift build`, copied the artifacts into a `/tmp/syntaxkit-poc/lib/` staging dir, then reverted the package manifest. + +Wrote a pure-DSL `Input.swift`: + +```swift +import SyntaxKit // optional; only for IDE + +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} + +Struct("Pet") { + Variable(.let, name: "kind", type: "String") +} +``` + +And a hand-rolled `Input.wrapped.swift`: + +```swift +import SyntaxKit + +let __syntaxkit_root = Group { + Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") + } + Struct("Pet") { + Variable(.let, name: "kind", type: "String") + } +} + +print(__syntaxkit_root.generateCode()) +``` + +Invoked with: + +``` +xcrun swift \ + -I lib -L lib -lSyntaxKit \ + -Xcc -I -Xcc lib/_SwiftSyntaxCShims-include \ + -Xlinker -rpath -Xlinker $(pwd)/lib \ + Input.wrapped.swift +``` + +Output (verbatim): + +``` +struct Person { +let name : String +let age : Int + +} +struct Pet { +let kind : String + +} +``` + +(Whitespace artifacts are SyntaxKit's `generateCode()` output as-is — out of scope for this POC.) + +## 2. Timings + +Three back-to-back runs after a cold first run: + +| Run | real | user | sys | +| --- | ---: | ---: | ---: | +| cold (first) | 0.72s | 0.77s | 0.29s | +| warm 1 | 0.14s | 0.08s | 0.04s | +| warm 2 | 0.11s | 0.07s | 0.02s | +| warm 3 | 0.11s | 0.07s | 0.02s | + +Hardware: Apple Silicon mac. Cold start is dominated by loading SyntaxKit + SwiftSyntax dylibs from disk; once cached, the swift interpreter just compiles a tiny script. For a per-file CLI invocation this is well inside the "feels instant" budget. + +## 3. New design finding: C-shim include path + +Without `-Xcc -I -Xcc <_SwiftSyntaxCShims/include>`, the script compile fails with: + +``` +:0: error: missing required module '_SwiftSyntaxCShims' +``` + +SyntaxKit transitively depends on SwiftSyntax which has a C-shims target whose module map lives at `swift-syntax/Sources/_SwiftSyntaxCShims/include/module.modulemap`. The CLI's bundled-binary distribution layout (§5 of the design doc) must include this header directory, not just the `.dylib` and `.swiftmodule` files. Updated the design doc to reflect this. + +## 4. New design finding: `if` inside `Group` is broken in SyntaxKit today + +A wrapped input containing a conditional in the builder: + +```swift +let __syntaxkit_root = Group { + if true { + Struct("A") { Variable(.let, name: "x", type: "Int") } + } +} +``` + +fails with: + +``` +error: failed to produce diagnostic for expression; please submit a bug report +let __syntaxkit_root = Group { + ^ +``` + +`CodeBlockBuilderResult` declares both `buildEither(first:)` / `buildEither(second:)` and `buildOptional` (`Sources/SyntaxKit/CodeBlocks/CodeBlockBuilderResult.swift:46-58`), so the API surface *says* `if`/`else` is supported. The compiler crash is a Swift type-checker timeout — likely from the `any CodeBlock...` variadic overload combined with `buildEither` overload resolution. No test in `Tests/SyntaxKitTests/Unit/` exercises an `if` inside `Group { … }`, which is why this hasn't been caught. + +**Implication for the CLI design:** non-blocking. v1 can document "no conditionals in input files yet" and ship; the underlying SyntaxKit fix is independent. Worth filing as a separate issue. + +## 5. Confirmed: hoisted imports work + +A wrapped input with both `import SyntaxKit` and `import Foundation` at the top compiles fine and `UUID`/`Date` resolve in the rendered struct fields. The CLI's hoist-imports step (design §3) is safe. + +## 6. Not yet retired + +- **Stderr/stdout interleaving under load.** Need a load test (large input → tons of output → confirm captured stdout is intact and stderr doesn't bleed in). +- **Linux behavior.** Step 7 of the POC ladder. Same `swift -I -L -l` flag set should work; framework-search paths (`-F`) become a no-op. +- **`@main` and other top-level forms.** Not relevant for the wrapper because we always control the wrapper's shape, but if users ever paste a class/extension declaration directly into `Input.swift` we need to reject it cleanly. + +## 7. Updates to the design doc to make from this POC + +- §5 distribution layout: add `Sources/_SwiftSyntaxCShims/include/` to the bundled `lib/` contents. +- §3 spawn command: include the `-Xcc -I -Xcc <…>` flag. +- §7 open questions: SwiftSyntax dylib size confirmed at ~25 MB (debug). Re-measure release. +- §7 open questions: add tracking note for the `if`-in-`Group` Swift compiler bug. From cf9f02a581b6be7deb686ad6c7b58f184ed17e54 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 16:50:17 -0400 Subject: [PATCH 03/28] Add POC step 1 reproducer script (#154) One-command reproduction: flips Package.swift to a dynamic SyntaxKit library, builds, stages dylib + swiftmodules + _SwiftSyntaxCShims headers into /tmp/syntaxkit-poc/, writes a pure-DSL input + hand-rolled wrapper, and runs one cold + three warm timings. Restores Package.swift on exit via trap. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step1.sh | 117 +++++++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100755 Docs/research/poc-step1.sh diff --git a/Docs/research/poc-step1.sh b/Docs/research/poc-step1.sh new file mode 100755 index 0000000..fe41e3b --- /dev/null +++ b/Docs/research/poc-step1.sh @@ -0,0 +1,117 @@ +#!/usr/bin/env bash +# +# POC step 1 reproducer for issue #154. +# Run from anywhere; resolves the repo root from its own location. +# +# What it does: +# 1. Backs up Package.swift, flips the SyntaxKit library to type: .dynamic. +# 2. swift build (produces libSyntaxKit.dylib). +# 3. Stages dylib + swiftmodules + _SwiftSyntaxCShims headers into /tmp/syntaxkit-poc/lib/. +# 4. Writes a pure-DSL Input.swift and a hand-rolled Input.wrapped.swift. +# 5. Runs the wrapped script once cold + three times warm, printing timings. +# 6. Restores Package.swift on exit (even on failure). + +set -euo pipefail + +if [[ "$(uname -s)" != "Darwin" ]]; then + echo "This reproducer is macOS-only. Linux smoke test is POC step 7." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +POC_DIR="/tmp/syntaxkit-poc" +PACKAGE_FILE="$REPO_ROOT/Package.swift" +PACKAGE_BACKUP="$(mktemp)" + +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if old not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found — has Package.swift changed shape?") +p.write_text(src.replace(old, new, 1)) +PY + +cd "$REPO_ROOT" +echo "==> swift build" +swift build + +BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1 || true)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate .build//debug" >&2 + exit 1 +fi + +echo "==> Staging $POC_DIR/lib/" +rm -rf "$POC_DIR" +mkdir -p "$POC_DIR/lib" +cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" +cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" +cp -r ".build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ + "$POC_DIR/lib/_SwiftSyntaxCShims-include" + +cat > "$POC_DIR/Input.swift" <<'SWIFT' +// Pure-DSL input. No print, no @main, no boilerplate. +import SyntaxKit // optional; only for IDE autocomplete + +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} + +Struct("Pet") { + Variable(.let, name: "kind", type: "String") +} +SWIFT + +cat > "$POC_DIR/Input.wrapped.swift" <<'SWIFT' +import SyntaxKit + +let __syntaxkit_root = Group { + Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") + } + Struct("Pet") { + Variable(.let, name: "kind", type: "String") + } +} + +print(__syntaxkit_root.generateCode()) +SWIFT + +cd "$POC_DIR" + +SWIFT_ARGS=( + -I lib -L lib -lSyntaxKit + -Xcc -I -Xcc lib/_SwiftSyntaxCShims-include + -Xlinker -rpath -Xlinker "$POC_DIR/lib" + Input.wrapped.swift +) + +echo +echo "==> Cold run (full output + timing):" +/usr/bin/time -p xcrun swift "${SWIFT_ARGS[@]}" + +echo +echo "==> Warm runs (timings only, output discarded):" +for _ in 1 2 3; do + /usr/bin/time -p xcrun swift "${SWIFT_ARGS[@]}" >/dev/null +done + +echo +echo "==> Done. Staging dir kept at $POC_DIR for further poking." From 840cb0c64aaa6ac51d62da0ad0997aff6316baee Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 16:56:48 -0400 Subject: [PATCH 04/28] Release-config dylib size: 9.3 MB stripped (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured SyntaxKit dylib under -c release with strip -x: 25 MB debug → 18 MB release → 9.3 MB stripped. Warm execution timings identical to debug. Distribution sizing is comfortable. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/codegen-cli-design.md | 2 +- Docs/research/poc-step1-results.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Docs/research/codegen-cli-design.md b/Docs/research/codegen-cli-design.md index c2d3bd5..0056f96 100644 --- a/Docs/research/codegen-cli-design.md +++ b/Docs/research/codegen-cli-design.md @@ -141,7 +141,7 @@ Each step is independently shippable and de-risks the next. 1. **Hand-driven wrap + spawn.** Hand-write a pure-DSL `Input.swift` and a hand-rolled `Input.wrapped.swift` that imports SyntaxKit, splices the body into `Group { … }`, and prints `generateCode()`. Invoke it manually with `swift Input.wrapped.swift -I … -L … -F … -lSyntaxKit -framework SyntaxKit` against a local `swift build` of SyntaxKit. **Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit at all, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost.** Cold-start is the single biggest risk to retire — if it's 20+ seconds, the design needs rethinking. 2. **CLI subcommand for single-file mode.** Add `syntaxkit run `: parse out top-level imports with SwiftSyntax, write `Input.wrapped.swift` to a temp dir, spawn `swift` via `Foundation.Process`, capture stdout, write to `-o ` (or stdout if no `-o`). Stderr is forwarded; rewrite `Input.wrapped.swift:LINE` references back to `Input.swift:LINE` before forwarding. 3. **Folder mode.** Walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`. Add `_`-prefix skip rule. Parallelize cautiously (`swift` spawns aren't free — gate on a small concurrency limit, maybe `ProcessInfo.activeProcessorCount`). -4. **Ship a bundled-binary release.** Build SyntaxKit as a `type: .dynamic` library, then bundle the `libSyntaxKit.dylib` + every `.swiftmodule` SyntaxKit publicly re-exports (SwiftSyntax + version-suffixed variants, SwiftOperators, SwiftParser) + the `_SwiftSyntaxCShims/include/` header dir in a `lib/` directory next to the CLI binary. Write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). POC step 1 confirmed cold-start with this layout is ~720ms. Debug dylib size is ~25 MB — re-measure under release config. +4. **Ship a bundled-binary release.** Build SyntaxKit as a `type: .dynamic` library under `-c release`, then `strip -x` the resulting dylib. Bundle the stripped `libSyntaxKit.dylib` + every `.swiftmodule` SyntaxKit publicly re-exports (SwiftSyntax + version-suffixed variants, SwiftOperators, SwiftParser) + the `_SwiftSyntaxCShims/include/` header dir in a `lib/` directory next to the CLI binary. Write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). POC step 1 confirmed cold-start with this layout is ~720ms; stripped release dylib is **9.3 MB** on Apple Silicon (down from 25 MB debug / 18 MB unstripped release). 5. **Helpers directory.** Discovery + compile + flag-splicing. 6. **Output cache.** Add the cache, keyed as in §5. Add `--no-cache` for debugging. 7. **Linux smoke test.** Confirm `/usr/bin/env swift` works on Linux with the bundled dylib layout (no framework search path, but `-I + -L + -lSyntaxKit` should be sufficient, matching Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch). diff --git a/Docs/research/poc-step1-results.md b/Docs/research/poc-step1-results.md index 1958b44..1940027 100644 --- a/Docs/research/poc-step1-results.md +++ b/Docs/research/poc-step1-results.md @@ -8,7 +8,7 @@ - **Warm: ~110ms** for subsequent runs. - **Pure-DSL `Input.swift` spliced into `Group { … }`** in a generated wrapper compiles and runs end-to-end, producing the expected Swift source. - **Bundled-dylib distribution is real and viable.** The only flags the CLI has to assemble are `-I`, `-L`, `-lSyntaxKit`, `-Xlinker -rpath -Xlinker ` and one new requirement: `-Xcc -I -Xcc ` (see §3 finding). -- **SyntaxKit dylib weight: 25.3 MB** in debug. Release build will be smaller. The CLI release artifact is dominated by this and the SwiftSyntax `.swiftmodule` files. +- **SyntaxKit dylib weight:** 25 MB debug → 18 MB release → **9.3 MB stripped release**. The 9.3 MB number is the one that matters for distribution and is well within range of a normal CLI binary. ## 1. What was run @@ -134,5 +134,5 @@ A wrapped input with both `import SyntaxKit` and `import Foundation` at the top - §5 distribution layout: add `Sources/_SwiftSyntaxCShims/include/` to the bundled `lib/` contents. - §3 spawn command: include the `-Xcc -I -Xcc <…>` flag. -- §7 open questions: SwiftSyntax dylib size confirmed at ~25 MB (debug). Re-measure release. +- §7 open questions: SyntaxKit dylib size measured at 25 MB debug, 18 MB release, 9.3 MB release+stripped (`strip -x`). Warm performance identical between debug and release builds. - §7 open questions: add tracking note for the `if`-in-`Group` Swift compiler bug. From 9e5affa136805c06ea16a371fa9c9be2cecb645f Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 17:01:01 -0400 Subject: [PATCH 05/28] POC step 2: skitrun CLI wraps + spawns SyntaxKit DSL inputs (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New executable target `skitrun` (POC name) wraps the hand-driven step-1 flow in real code: parse input with SwiftSyntax, hoist top-level imports, emit a Group { … } wrapper fenced in #sourceLocation directives, spawn `swift` via Foundation.Process, pipe stdout through, forward stderr with literal-path fix-up. #sourceLocation does the heavy lifting for diagnostic fidelity — a NonexistentType in InputError.swift:4 now reports as InputError.swift:4:37 from the spawned swift, with no manual stderr arithmetic. Verified default-stdout, -o file output, hoisted Foundation import, and error path rewriting all work end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step2-results.md | 70 +++++++++ Package.swift | 12 ++ Sources/skitrun/Main.swift | 243 +++++++++++++++++++++++++++++ 3 files changed, 325 insertions(+) create mode 100644 Docs/research/poc-step2-results.md create mode 100644 Sources/skitrun/Main.swift diff --git a/Docs/research/poc-step2-results.md b/Docs/research/poc-step2-results.md new file mode 100644 index 0000000..5bd6cd2 --- /dev/null +++ b/Docs/research/poc-step2-results.md @@ -0,0 +1,70 @@ +# POC Step 2 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 2. Goal: wrap the hand-driven flow from step 1 inside an actual CLI executable that parses imports with SwiftSyntax, generates the wrapper, spawns `swift`, and forwards output/errors. + +## What landed + +New executable target `skitrun` (POC name; final CLI name TBD) at `Sources/skitrun/Main.swift`. Single file, ~180 lines. Depends only on `SwiftSyntax` + `SwiftParser` — does **not** depend on SyntaxKit, since the host doesn't render anything itself. + +## Usage + +``` +skitrun [-o ] [--lib ] +``` + +`--lib` defaults to `/tmp/syntaxkit-poc/lib` so it works against the artifacts produced by [`poc-step1.sh`](./poc-step1.sh) without further flags. + +Build it once: `swift build --product skitrun`. The binary lands at `.build//debug/skitrun`. + +## Verified flows + +1. **Default (stdout):** `skitrun Input.swift` prints rendered Swift to stdout. +2. **File output:** `skitrun Input.swift -o Out.swift` writes the file atomically. +3. **Hoisted imports:** an input with `import Foundation` at the top compiles cleanly; `UUID`/`Date` resolve in the rendered struct. +4. **Compiler diagnostics map to the input file.** A deliberate `type: NonexistentType` in `InputError.swift:4` produces: + ``` + /tmp/syntaxkit-poc/InputError.swift:4:37: error: cannot find 'NonexistentType' in scope + ``` + The path and line are correct — confirming `#sourceLocation` is doing the work end-to-end. + +## How the wrap works + +`SwiftParser.Parser.parse(source:)` produces a `SourceFileSyntax`. We walk `tree.statements`: + +- Every leading `ImportDeclSyntax` is collected for hoisting. +- The first non-import statement marks the start of the body. +- Everything from that byte offset forward is the body, copied verbatim. + +The wrapper is then: + +```swift +import SyntaxKit + + +let __skitrun_root = Group { +#sourceLocation(file: "", line: ) + +#sourceLocation() +} + +print(__skitrun_root.generateCode()) +``` + +`#sourceLocation` is what gives us diagnostic fidelity for free — the Swift compiler honors it and rewrites file/line in every error/warning emitted from the body range. No manual stderr line-number arithmetic needed. + +## Spawn shape + +`Foundation.Process` invoking `/usr/bin/env swift` with the exact flag set from POC step 1, captured into `stdoutPipe` and `stderrPipe`. Stdout is written verbatim to the output destination. Stderr is forwarded after one fix-up: any remaining literal `//skitrun-/Input.wrapped.swift` references (those outside the `#sourceLocation` range — i.e. errors in the preamble itself) get rewritten to the input path. + +## Surface limits worth knowing + +- **Snippet gutter line numbers in diagnostics show wrapper line numbers, not input line numbers.** The compiler maps the *file/line* in the diagnostic header via `#sourceLocation` but shows the surrounding source snippet from the actual file with its actual line numbers. The path and starting line are correct (navigable), but the gutter `7 |` / `8 |` markers may not match the input's line numbering. Cosmetic; doesn't affect navigation. +- **No timeout yet.** The design calls for a 60s default. Adding `Process.terminate(after:)` is a step 6 (cache) sibling concern. +- **Stdin / stderr interleaving under load not tested.** Step 7 territory. +- **`if`-in-`Group` still crashes the compiler** ([#155](https://github.com/brightdigit/SyntaxKit/issues/155)). Independent SyntaxKit bug; `skitrun` would happily pass such an input through, but the spawned `swift` would fail with the same opaque diagnostic from step 1. + +## What's next + +The natural step 3 is folder mode (walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`). All the per-file work is already in place — folder mode is just iteration + concurrency-limited fan-out + a `_`-prefix skip rule. Modest engineering, low risk. + +After that, step 4 (bundled-binary release) is where the design hits its biggest remaining systems-integration question: how to actually ship the `lib/` directory next to the binary across SwiftPM build, install, and `brew` distribution. diff --git a/Package.swift b/Package.swift index ea48d24..1087718 100644 --- a/Package.swift +++ b/Package.swift @@ -95,6 +95,10 @@ let package = Package( name: "skit", targets: ["skit"] ), + .executable( + name: "skitrun", + targets: ["skitrun"] + ), ], dependencies: [ .package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.1"), @@ -144,6 +148,14 @@ let package = Package( dependencies: ["SyntaxParser"], swiftSettings: swiftSettings ), + .executableTarget( + name: "skitrun", + dependencies: [ + .product(name: "SwiftSyntax", package: "swift-syntax"), + .product(name: "SwiftParser", package: "swift-syntax") + ], + swiftSettings: swiftSettings + ), .testTarget( name: "SyntaxKitTests", dependencies: ["SyntaxKit"], diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift new file mode 100644 index 0000000..4c3d2ee --- /dev/null +++ b/Sources/skitrun/Main.swift @@ -0,0 +1,243 @@ +// +// Main.swift +// SyntaxKit — skitrun (POC for issue #154) +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation +import SwiftParser +import SwiftSyntax + +@main +internal enum SkitRun { + internal static func main() throws { + let args = try CLIArgs.parse(CommandLine.arguments) + + let inputURL = URL(fileURLWithPath: args.inputPath) + let absoluteInputPath = inputURL.standardizedFileURL.path + let source = try String(contentsOf: inputURL, encoding: .utf8) + let wrapped = wrap(source: source, originalPath: absoluteInputPath) + + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("skitrun-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") + try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) + + let result = try runSwift( + wrappedPath: wrappedURL.path, + libPath: args.libPath + ) + + if !result.stderr.isEmpty { + // #sourceLocation already maps body diagnostics to the input file. + // For diagnostics in the preamble (lines outside the body) the path + // still references the wrapper — rewrite verbatim path occurrences so + // users see something coherent. + let rewritten = result.stderr.replacingOccurrences( + of: wrappedURL.path, + with: absoluteInputPath + ) + FileHandle.standardError.write(Data(rewritten.utf8)) + } + + guard result.exitCode == 0 else { + exit(result.exitCode) + } + + if let outputPath = args.outputPath { + try result.stdout.write(to: URL(fileURLWithPath: outputPath)) + } else { + FileHandle.standardOutput.write(result.stdout) + } + } +} + +// MARK: - Arg parsing + +private struct CLIArgs { + let inputPath: String + let outputPath: String? + let libPath: String + + static func parse(_ argv: [String]) throws -> CLIArgs { + var inputPath: String? + var outputPath: String? + var libPath = "/tmp/syntaxkit-poc/lib" + + var i = 1 + while i < argv.count { + let arg = argv[i] + switch arg { + case "-o", "--output": + guard i + 1 < argv.count else { throw usage("-o requires a value") } + outputPath = argv[i + 1] + i += 2 + case "--lib": + guard i + 1 < argv.count else { throw usage("--lib requires a value") } + libPath = argv[i + 1] + i += 2 + case "-h", "--help": + FileHandle.standardError.write(Data(helpText.utf8)) + exit(0) + case _ where arg.hasPrefix("-"): + throw usage("unknown flag: \(arg)") + default: + guard inputPath == nil else { throw usage("only one input file is supported") } + inputPath = arg + i += 1 + } + } + + guard let inputPath else { throw usage("missing input file") } + return CLIArgs(inputPath: inputPath, outputPath: outputPath, libPath: libPath) + } +} + +private let helpText = """ + skitrun [-o ] [--lib ] + + POC for issue #154 — runs a SyntaxKit DSL input file by wrapping it in a + Group { … } closure and spawning `swift`. + + Options: + -o, --output Write rendered Swift to (default: stdout). + --lib Directory containing libSyntaxKit.dylib + module files. + (default: /tmp/syntaxkit-poc/lib, produced by + Docs/research/poc-step1.sh) + """ + +private func usage(_ message: String) -> CLIError { + CLIError(message: "\(message)\n\n\(helpText)\n") +} + +private struct CLIError: Error, CustomStringConvertible { + let message: String + var description: String { message } +} + +// MARK: - Wrapping + +/// Splits the input into hoisted `import` declarations and a verbatim body, +/// returning a complete Swift program that runs SyntaxKit on the body. +/// +/// The body is fenced in `#sourceLocation` directives so compiler diagnostics +/// in the body reference the original input file and line numbers. +internal func wrap(source: String, originalPath: String) -> String { + let tree = Parser.parse(source: source) + let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) + + // Find the first non-import top-level statement; everything before it that + // is an import gets hoisted, anything before that which is *not* an import + // stays in the body (e.g. a top-level `// comment` is left alone). + var hoisted: [String] = [] + var firstBodyByte: AbsolutePosition? + + for item in tree.statements { + if let importDecl = item.item.as(ImportDeclSyntax.self), + firstBodyByte == nil { + hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) + continue + } + firstBodyByte = item.position + break + } + + let body: String + let firstBodyLine: Int + if let firstBodyByte { + let start = source.utf8.index(source.utf8.startIndex, offsetBy: firstBodyByte.utf8Offset) + body = String(source[start...]) + firstBodyLine = locConverter.location(for: firstBodyByte).line + } else { + body = "" + firstBodyLine = 1 + } + + let hoistedBlock = hoisted.isEmpty ? "" : hoisted.joined(separator: "\n") + "\n" + + // #sourceLocation must use a forward-slash path; escape backslashes/quotes + // defensively even though macOS paths shouldn't contain them. + let escapedPath = originalPath + .replacingOccurrences(of: "\\", with: "\\\\") + .replacingOccurrences(of: "\"", with: "\\\"") + + return """ + import SyntaxKit + \(hoistedBlock) + let __skitrun_root = Group { + #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) + \(body) + #sourceLocation() + } + + print(__skitrun_root.generateCode()) + """ +} + +// MARK: - Spawning swift + +private struct RunResult { + let exitCode: Int32 + let stdout: Data + let stderr: String +} + +private func runSwift(wrappedPath: String, libPath: String) throws -> RunResult { + let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/env") + process.arguments = [ + "swift", + "-suppress-warnings", + "-I", libPath, + "-L", libPath, + "-lSyntaxKit", + "-Xcc", "-I", "-Xcc", cShimsInclude, + "-Xlinker", "-rpath", "-Xlinker", libPath, + wrappedPath + ] + + let stdoutPipe = Pipe() + let stderrPipe = Pipe() + process.standardOutput = stdoutPipe + process.standardError = stderrPipe + + try process.run() + process.waitUntilExit() + + let stdoutData = stdoutPipe.fileHandleForReading.readDataToEndOfFile() + let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() + + return RunResult( + exitCode: process.terminationStatus, + stdout: stdoutData, + stderr: String(decoding: stderrData, as: UTF8.self) + ) +} From dd209c9f8ed8301a48e9b2092b4f58ee5410ce9e Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 21:08:08 -0400 Subject: [PATCH 06/28] POC step 3: skitrun folder mode (#154) skitrun InputDir/ -o OutDir/ walks **/*.swift (skipping `_`-prefixed files), runs the per-file wrap+spawn over withTaskGroup capped at activeProcessorCount, and writes successes into mirrored paths under OutDir/. Failures don't abort the batch: successes are still written and the CLI exits non-zero with a `skitrun: N/M succeeded` summary. Verified happy path (3 files, 1.41s wall vs 0.72s cold baseline), skip rule (deliberately-invalid `_HelperShouldBeSkipped.swift` is not visited), partial failure (3/4 succeeded with exit 1), and single-file regression. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step3-results.md | 48 ++++++ Sources/skitrun/Main.swift | 254 +++++++++++++++++++++++------ 2 files changed, 252 insertions(+), 50 deletions(-) create mode 100644 Docs/research/poc-step3-results.md diff --git a/Docs/research/poc-step3-results.md b/Docs/research/poc-step3-results.md new file mode 100644 index 0000000..be524bf --- /dev/null +++ b/Docs/research/poc-step3-results.md @@ -0,0 +1,48 @@ +# POC Step 3 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 3. Goal: extend `skitrun` to walk a directory of `.swift` inputs, mirror the relative paths into an output directory, and run the per-file work concurrently with a sane cap. + +## What changed + +`skitrun` now accepts a directory as input: + +``` +skitrun InputDir/ -o OutDir/ +``` + +When the input is a directory, the existing single-file work is hoisted into a `processFile(inputPath:libPath:)` helper. The new `runDirectory(...)` driver walks the input with `FileManager.enumerator`, fan-outs the per-file work over `withTaskGroup`, and writes successes into mirrored paths under `OutDir/`. Single-file mode is unchanged. + +Three small conventions ride along: + +- **`_`-prefix skip rule.** `_Helpers.swift`, `_Shared.swift`, etc. are not processed. (Confirmed against `_HelperShouldBeSkipped.swift` containing deliberately-invalid Swift — skitrun didn't try to compile it.) +- **`activeProcessorCount` concurrency cap.** The task group keeps that many in-flight `swift` spawns at a time, draining + refilling as each finishes. +- **Tuist-analog partial semantics.** Successful files are *always* written, even when other files in the same batch fail. The CLI exits non-zero if any failed and prints a `skitrun: N/M succeeded` summary to stderr. + +## Verified flows + +1. **Happy path.** A `codegen/` tree with `Models/Person.swift`, `Models/Pet.swift`, `Audit/Snapshot.swift` (the last with a hoisted `import Foundation`) produces a mirrored `out/Models/{Person,Pet}.swift` + `out/Audit/Snapshot.swift`. Total wall time 1.41s for 3 files (vs. 0.72s baseline cold-start for one). +2. **Skip rule.** `codegen/_HelperShouldBeSkipped.swift` contains the literal line `this is not valid swift`. It is not visited, the rest of the tree processes cleanly. +3. **Partial failure.** Adding a `Models/Bad.swift` with `type: TypeThatDoesNotExist` produces: + ``` + ---- /tmp/skitrun-folder-test/codegen/Models/Bad.swift ---- + /tmp/skitrun-folder-test/codegen/Models/Bad.swift:4:37: error: cannot find 'TypeThatDoesNotExist' in scope + … + skitrun: 3/4 succeeded + ``` + Exit code 1. Person/Pet/Snapshot still written. +4. **Single-file regression.** Both `skitrun Input.swift` (stdout) and `skitrun Input.swift -o Out.swift` (file) still work after the refactor. + +## Parallelism observations + +A quick timing on 3 parallel files vs. 1 cold-start baseline: + +| | wall time | +| --- | ---: | +| 1 file, cold | 0.72s | +| 3 files, cold, `withTaskGroup` cap = `activeProcessorCount` | 1.41s | + +That's well below 3×0.72 = 2.16s, confirming the parallelism is buying something — but also clearly slower than 3×0.11 = 0.33s warm, meaning successive `swift` invocations don't fully share OS file-cache benefits within a single batch run. (Each spawn still pays its own compile cost; the dylib pages are warm after the first, but compile work isn't deduplicated.) For larger batches we'd want to measure where the curve goes — and eventually pull the work into a single long-lived `swift` process driving all inputs, to skip the per-file compile overhead entirely. Out of scope for v1. + +## What's next + +Step 4 — bundled-binary release. The first real systems-integration challenge: how the `lib/` directory ships next to the `skitrun` binary across `swift build`, `swift run`, and `brew install`. Today users have to run `Docs/research/poc-step1.sh` to stage `/tmp/syntaxkit-poc/lib/`; that has to become "user installs the CLI, it just works." diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index 4c3d2ee..0365afe 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -33,56 +33,193 @@ import SwiftSyntax @main internal enum SkitRun { - internal static func main() throws { + internal static func main() async throws { let args = try CLIArgs.parse(CommandLine.arguments) - let inputURL = URL(fileURLWithPath: args.inputPath) - let absoluteInputPath = inputURL.standardizedFileURL.path - let source = try String(contentsOf: inputURL, encoding: .utf8) - let wrapped = wrap(source: source, originalPath: absoluteInputPath) - - let tmpDir = FileManager.default.temporaryDirectory - .appendingPathComponent("skitrun-\(UUID().uuidString)") - try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) - defer { try? FileManager.default.removeItem(at: tmpDir) } - - let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") - try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) - - let result = try runSwift( - wrappedPath: wrappedURL.path, - libPath: args.libPath - ) - - if !result.stderr.isEmpty { - // #sourceLocation already maps body diagnostics to the input file. - // For diagnostics in the preamble (lines outside the body) the path - // still references the wrapper — rewrite verbatim path occurrences so - // users see something coherent. - let rewritten = result.stderr.replacingOccurrences( - of: wrappedURL.path, - with: absoluteInputPath + switch args.mode { + case .singleFile(let input, let output): + try runSingleFile(inputPath: input, outputPath: output, libPath: args.libPath) + case .directory(let inputDir, let outputDir): + let exitCode = await runDirectory( + inputDir: inputDir, + outputDir: outputDir, + libPath: args.libPath ) - FileHandle.standardError.write(Data(rewritten.utf8)) + exit(exitCode) } + } +} + +// MARK: - Single-file mode + +private func runSingleFile(inputPath: String, outputPath: String?, libPath: String) throws { + let result = try processFile(inputPath: inputPath, libPath: libPath) + if !result.stderr.isEmpty { + FileHandle.standardError.write(Data(result.stderr.utf8)) + } + guard result.exitCode == 0 else { + exit(result.exitCode) + } + if let outputPath { + try result.stdout.write(to: URL(fileURLWithPath: outputPath)) + } else { + FileHandle.standardOutput.write(result.stdout) + } +} + +// MARK: - Folder mode + +private func runDirectory(inputDir: String, outputDir: String, libPath: String) async -> Int32 { + let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL + let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL + + let inputs: [URL] + do { + inputs = try collectInputs(at: inputURL) + } catch { + FileHandle.standardError.write(Data("skitrun: failed to walk \(inputDir): \(error)\n".utf8)) + return 1 + } + + if inputs.isEmpty { + FileHandle.standardError.write(Data("skitrun: no .swift inputs under \(inputDir)\n".utf8)) + return 0 + } - guard result.exitCode == 0 else { - exit(result.exitCode) + let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) + + var outcomes: [FileOutcome] = [] + var iterator = inputs.makeIterator() + + await withTaskGroup(of: FileOutcome.self) { group in + for _ in 0.. +} + +private func runOne(_ input: URL, libPath: String) -> FileOutcome { + do { + let result = try processFile(inputPath: input.path, libPath: libPath) + return FileOutcome(input: input, result: .success(result)) + } catch { + return FileOutcome(input: input, result: .failure(error)) + } +} + +private func collectInputs(at inputDir: URL) throws -> [URL] { + guard let enumerator = FileManager.default.enumerator( + at: inputDir, + includingPropertiesForKeys: [.isRegularFileKey], + options: [.skipsHiddenFiles] + ) else { + throw CLIError(message: "could not enumerate \(inputDir.path)") + } + + var result: [URL] = [] + for case let url as URL in enumerator { + let values = try url.resourceValues(forKeys: [.isRegularFileKey]) + guard values.isRegularFile == true else { continue } + guard url.pathExtension == "swift" else { continue } + guard !url.lastPathComponent.hasPrefix("_") else { continue } + result.append(url.standardizedFileURL) + } + return result.sorted { $0.path < $1.path } +} + +// MARK: - Per-file work + +private struct ProcessResult { + let exitCode: Int32 + let stdout: Data + let stderr: String +} + +private func processFile(inputPath: String, libPath: String) throws -> ProcessResult { + let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL + let absoluteInputPath = inputURL.path + let source = try String(contentsOf: inputURL, encoding: .utf8) + let wrapped = wrap(source: source, originalPath: absoluteInputPath) + + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("skitrun-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") + try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) + + let raw = try runSwift(wrappedPath: wrappedURL.path, libPath: libPath) + // #sourceLocation maps body diagnostics back to the input file. Errors in + // the preamble (lines outside the body) still reference the wrapper — + // rewrite literal occurrences of its path so users see something coherent. + let stderr = raw.stderr.replacingOccurrences( + of: wrappedURL.path, + with: absoluteInputPath + ) + return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } // MARK: - Arg parsing private struct CLIArgs { - let inputPath: String - let outputPath: String? + enum Mode { + case singleFile(input: String, output: String?) + case directory(input: String, output: String) + } + + let mode: Mode let libPath: String static func parse(_ argv: [String]) throws -> CLIArgs { @@ -108,25 +245,48 @@ private struct CLIArgs { case _ where arg.hasPrefix("-"): throw usage("unknown flag: \(arg)") default: - guard inputPath == nil else { throw usage("only one input file is supported") } + guard inputPath == nil else { throw usage("only one input path is supported") } inputPath = arg i += 1 } } - guard let inputPath else { throw usage("missing input file") } - return CLIArgs(inputPath: inputPath, outputPath: outputPath, libPath: libPath) + guard let inputPath else { throw usage("missing input path") } + + var isDirectory: ObjCBool = false + guard FileManager.default.fileExists(atPath: inputPath, isDirectory: &isDirectory) else { + throw usage("input does not exist: \(inputPath)") + } + + let mode: Mode + if isDirectory.boolValue { + guard let outputPath else { + throw usage("directory inputs require -o ") + } + mode = .directory(input: inputPath, output: outputPath) + } else { + mode = .singleFile(input: inputPath, output: outputPath) + } + + return CLIArgs(mode: mode, libPath: libPath) } } private let helpText = """ - skitrun [-o ] [--lib ] + skitrun [-o ] [--lib ] - POC for issue #154 — runs a SyntaxKit DSL input file by wrapping it in a + POC for issue #154 — runs SyntaxKit DSL input(s) by wrapping each in a Group { … } closure and spawning `swift`. + Forms: + skitrun Input.swift — render to stdout + skitrun Input.swift -o Out.swift — render to a file + skitrun InputDir/ -o OutDir/ — walk **/*.swift (skipping files + prefixed with '_') and mirror + rendered output into OutDir/ + Options: - -o, --output Write rendered Swift to (default: stdout). + -o, --output Output file (single-file mode) or directory (folder mode). --lib Directory containing libSyntaxKit.dylib + module files. (default: /tmp/syntaxkit-poc/lib, produced by Docs/research/poc-step1.sh) @@ -202,13 +362,7 @@ internal func wrap(source: String, originalPath: String) -> String { // MARK: - Spawning swift -private struct RunResult { - let exitCode: Int32 - let stdout: Data - let stderr: String -} - -private func runSwift(wrappedPath: String, libPath: String) throws -> RunResult { +private func runSwift(wrappedPath: String, libPath: String) throws -> ProcessResult { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" let process = Process() @@ -235,7 +389,7 @@ private func runSwift(wrappedPath: String, libPath: String) throws -> RunResult let stdoutData = stdoutPipe.fileHandleForReading.readDataToEndOfFile() let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() - return RunResult( + return ProcessResult( exitCode: process.terminationStatus, stdout: stdoutData, stderr: String(decoding: stderrData, as: UTF8.self) From c6daced94f5de8c125e4249ccbf1ea703404b3d3 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 06:14:22 -0400 Subject: [PATCH 07/28] POC step 4: self-contained skitrun release bundle (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit skitrun now resolves its lib/ directory automatically: --lib flag, then $SKITRUN_LIB_DIR, then /lib/, then /../lib/skitrun/ (Homebrew layout). No more /tmp/syntaxkit-poc/lib fallback. Clear diagnostic when no lib is found, naming all four search paths. poc-step4-release.sh builds a portable bundle under .build/skitrun-release/: release-config + stripped dylib (9.3 MB), modules, C-shims headers, and the binary itself (17 MB — SwiftSyntax statically linked, follow-up to deduplicate). Tested copying the bundle to unrelated directories, both same-dir and Homebrew layouts work zero-config. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step4-release.sh | 87 ++++++++++++++++++++++++++++++ Docs/research/poc-step4-results.md | 55 +++++++++++++++++++ Sources/skitrun/Main.swift | 71 +++++++++++++++++++++--- 3 files changed, 207 insertions(+), 6 deletions(-) create mode 100755 Docs/research/poc-step4-release.sh create mode 100644 Docs/research/poc-step4-results.md diff --git a/Docs/research/poc-step4-release.sh b/Docs/research/poc-step4-release.sh new file mode 100755 index 0000000..36b2035 --- /dev/null +++ b/Docs/research/poc-step4-release.sh @@ -0,0 +1,87 @@ +#!/usr/bin/env bash +# +# POC step 4: build a self-contained skitrun release bundle. +# +# Output: .build/skitrun-release/ +# skitrun ← the CLI binary +# lib/ +# libSyntaxKit.dylib ← release + strip -x +# *.swiftmodule ← SyntaxKit + transitively re-exported modules +# _SwiftSyntaxCShims-include/ ← C-shims headers (module map + .h files) +# +# Once produced, the binary is portable: copy the whole .build/skitrun-release/ +# directory anywhere, and `./skitrun-release/skitrun ` Just Works — no +# flags, no env vars, no SyntaxKit checkout required. + +set -euo pipefail + +if [[ "$(uname -s)" != "Darwin" ]]; then + echo "macOS-only for now. Linux smoke test is POC step 7." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +OUTPUT_DIR="$REPO_ROOT/.build/skitrun-release" +PACKAGE_FILE="$REPO_ROOT/Package.swift" +PACKAGE_BACKUP="$(mktemp)" + +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if old not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found") +p.write_text(src.replace(old, new, 1)) +PY + +cd "$REPO_ROOT" + +echo "==> swift build -c release --product skitrun" +swift build -c release --product skitrun + +BUILD_DIR="$(ls -d .build/*-apple-macosx*/release 2>/dev/null | head -1)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate release build dir under .build//release" >&2 + exit 1 +fi + +echo "==> Staging $OUTPUT_DIR" +rm -rf "$OUTPUT_DIR" +mkdir -p "$OUTPUT_DIR/lib" + +cp "$BUILD_DIR/skitrun" "$OUTPUT_DIR/skitrun" +cp "$BUILD_DIR/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/" +strip -x "$OUTPUT_DIR/lib/libSyntaxKit.dylib" +cp -r "$BUILD_DIR/Modules/." "$OUTPUT_DIR/lib/" +cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ + "$OUTPUT_DIR/lib/_SwiftSyntaxCShims-include" + +# Ensure the dylib's install_name uses @rpath so it's portable. +install_name_tool -id "@rpath/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/libSyntaxKit.dylib" 2>/dev/null || true + +BINARY_SIZE=$(ls -lh "$OUTPUT_DIR/skitrun" | awk '{print $5}') +DYLIB_SIZE=$(ls -lh "$OUTPUT_DIR/lib/libSyntaxKit.dylib" | awk '{print $5}') +TOTAL_SIZE=$(du -sh "$OUTPUT_DIR" | awk '{print $1}') + +echo +echo "==> Release bundle ready:" +echo " Binary: $BINARY_SIZE" +echo " Dylib: $DYLIB_SIZE" +echo " Total: $TOTAL_SIZE" +echo +echo "==> Try it:" +echo " $OUTPUT_DIR/skitrun " diff --git a/Docs/research/poc-step4-results.md b/Docs/research/poc-step4-results.md new file mode 100644 index 0000000..c5248ac --- /dev/null +++ b/Docs/research/poc-step4-results.md @@ -0,0 +1,55 @@ +# POC Step 4 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 4. Goal: produce a self-contained `skitrun` release bundle so users don't need a SyntaxKit checkout or `Docs/research/poc-step1.sh`. The binary finds its own `lib/` directory. + +## What landed + +1. **`resolveLibPath(override:)` in `Sources/skitrun/Main.swift`.** Search order: + 1. `--lib ` flag + 2. `$SKITRUN_LIB_DIR` env var + 3. `/lib/` — same-directory layout (the release bundle ships this way) + 4. `/../lib/skitrun/` — Homebrew layout (`bin/skitrun` ↔ `lib/skitrun/`) + + When none match, the CLI errors with a message enumerating all four paths and pointing at this script. The `/tmp/syntaxkit-poc/lib` fallback is gone. + +2. **`Docs/research/poc-step4-release.sh`.** Builds a self-contained bundle: + ``` + .build/skitrun-release/ + skitrun ← the CLI binary + lib/ + libSyntaxKit.dylib ← release + strip -x + *.swiftmodule ← SyntaxKit + transitively re-exported modules + _SwiftSyntaxCShims-include/ ← C-shims headers + ``` + Same trap-based Package.swift backup/restore as `poc-step1.sh`. `install_name_tool -id @rpath/libSyntaxKit.dylib` ensures the dylib install name is portable. + +## Verified flows + +Built bundle → copied to three unrelated locations → all worked with no flags, no env vars, no SyntaxKit checkout: + +1. **Same-directory layout.** `cp -r .build/skitrun-release /tmp/portable && /tmp/portable/skitrun Input.swift` → correct output. +2. **Homebrew layout.** `bin/skitrun + lib/skitrun/` arrangement → correct output. +3. **Error case.** `skitrun` alone in `/tmp/lonely/` (no lib anywhere) → clear diagnostic: + ``` + Could not locate SyntaxKit lib directory. Looked for: + 1. --lib (not provided) + 2. $SKITRUN_LIB_DIR (not set) + 3. /lib/ (not found) + 4. /../lib/skitrun/ (not found) + ``` +4. **Folder mode** from the portable bundle works end-to-end with partial-failure semantics intact. + +## Bundle weight + +| Component | Size | +| --- | ---: | +| `skitrun` (binary) | 17 MB | +| `libSyntaxKit.dylib` (release + stripped) | 9.3 MB | +| `lib/*` (modules + headers + dylib) | ~28 MB | +| **Total bundle** | **45 MB** | + +The 17 MB binary is unexpectedly heavy: it links SwiftSyntax statically because `skitrun` uses SwiftSyntax directly for parsing input files. So SwiftSyntax ships **twice** — once statically inside `skitrun`, once dynamically as part of the SyntaxKit dylib stack. Worth a follow-up: make `skitrun` itself dlopen SyntaxKit / share a dynamic SwiftSyntax with the dylib path. For v1 this is acceptable but is the largest single thing standing between the CLI and a "feels small" download. + +## What's next + +Step 5: helpers directory. Today users can `import Foundation` and `import SyntaxKit` from input files; step 5 lets them factor reusable codegen into a `Helpers/` directory that gets pre-compiled into `lib.dylib` and made importable from inputs. Modest engineering, but the first time `skitrun` itself invokes `swiftc` rather than `swift` (the helpers compile, distinct from the input run). See [`codegen-cli-design.md` §4](./codegen-cli-design.md#4-helpers) for the shape. diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index 0365afe..ebaf3a3 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -36,20 +36,77 @@ internal enum SkitRun { internal static func main() async throws { let args = try CLIArgs.parse(CommandLine.arguments) + let libPath: String + do { + libPath = try resolveLibPath(override: args.libPath) + } catch { + FileHandle.standardError.write(Data("\(error)\n".utf8)) + exit(2) + } + switch args.mode { case .singleFile(let input, let output): - try runSingleFile(inputPath: input, outputPath: output, libPath: args.libPath) + try runSingleFile(inputPath: input, outputPath: output, libPath: libPath) case .directory(let inputDir, let outputDir): let exitCode = await runDirectory( inputDir: inputDir, outputDir: outputDir, - libPath: args.libPath + libPath: libPath ) exit(exitCode) } } } +// MARK: - Resource location + +/// Resolves the directory containing `libSyntaxKit.dylib` + module files, +/// in priority order: explicit flag → env var → adjacent-to-binary +/// (`/lib/`) → Homebrew layout (`/../lib/skitrun/`). +internal func resolveLibPath(override: String?) throws -> String { + if let override { + guard isLibDir(override) else { + throw CLIError(message: "--lib path does not look like a SyntaxKit lib dir: \(override)") + } + return override + } + + if let env = ProcessInfo.processInfo.environment["SKITRUN_LIB_DIR"], !env.isEmpty { + guard isLibDir(env) else { + throw CLIError(message: "SKITRUN_LIB_DIR is set but path is not a lib dir: \(env)") + } + return env + } + + if let execURL = Bundle.main.executableURL?.resolvingSymlinksInPath() { + let execDir = execURL.deletingLastPathComponent() + + let adjacent = execDir.appendingPathComponent("lib").path + if isLibDir(adjacent) { return adjacent } + + let brewLayout = execDir.deletingLastPathComponent() + .appendingPathComponent("lib/skitrun").path + if isLibDir(brewLayout) { return brewLayout } + } + + throw CLIError(message: """ + Could not locate SyntaxKit lib directory. Looked for: + 1. --lib (not provided) + 2. $SKITRUN_LIB_DIR (not set) + 3. /lib/ (not found) + 4. /../lib/skitrun/ (not found) + Run Docs/research/poc-step4-release.sh to produce a self-contained + release bundle under .build/skitrun-release/. + """) +} + +private func isLibDir(_ path: String) -> Bool { + let fm = FileManager.default + var isDir: ObjCBool = false + guard fm.fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { return false } + return fm.fileExists(atPath: "\(path)/libSyntaxKit.dylib") +} + // MARK: - Single-file mode private func runSingleFile(inputPath: String, outputPath: String?, libPath: String) throws { @@ -220,12 +277,12 @@ private struct CLIArgs { } let mode: Mode - let libPath: String + let libPath: String? static func parse(_ argv: [String]) throws -> CLIArgs { var inputPath: String? var outputPath: String? - var libPath = "/tmp/syntaxkit-poc/lib" + var libPath: String? var i = 1 while i < argv.count { @@ -288,8 +345,10 @@ private let helpText = """ Options: -o, --output Output file (single-file mode) or directory (folder mode). --lib Directory containing libSyntaxKit.dylib + module files. - (default: /tmp/syntaxkit-poc/lib, produced by - Docs/research/poc-step1.sh) + When omitted, skitrun searches: $SKITRUN_LIB_DIR, + then /lib/, then /../lib/skitrun/. + Build a self-contained bundle with + Docs/research/poc-step4-release.sh. """ private func usage(_ message: String) -> CLIError { From 3d3b631d2cf77943ab050314f72b905d3bf624dc Mon Sep 17 00:00:00 2001 From: leogdion Date: Tue, 12 May 2026 08:32:31 -0400 Subject: [PATCH 08/28] POC step 5: Helpers/ discovery + per-toolchain cache (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit skitrun now walks up from the input looking for a Helpers/ directory. On hit, helpers compile via swiftc into libSyntaxKitHelpers.dylib under ~/Library/Caches/com.brightdigit.SyntaxKit/helpers//, keyed on helper-source hashes + swift --version + libSyntaxKit.dylib stamp + cache schema version. Compile lands in a tmp../ staging dir then atomic-renames into the cache path so concurrent invocations are safe. Inputs then `import SyntaxKitHelpers` and call into the compiled module; runSwift splices -I/-L/-lSyntaxKitHelpers -Xlinker -rpath onto the spawn. Two new flags: --helpers overrides discovery, --no-helpers skips it entirely. Folder mode's enumerator now also yields directories so it can skipDescendants() on a Helpers/ directly under the input root — without it the helpers would be reprocessed as inputs. Verified end-to-end via Docs/research/poc-step5.sh: cold compile 2.96s, warm cache hit 0.54s (matches step-1 warm baseline), folder mode 2/2 with Helpers/ excluded, --no-helpers errors with `no such module 'SyntaxKitHelpers'` as expected. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step5-results.md | 58 +++++++ Docs/research/poc-step5.sh | 135 ++++++++++++++++ Sources/skitrun/Helpers.swift | 247 +++++++++++++++++++++++++++++ Sources/skitrun/Main.swift | 211 +++++++++++++++++++----- 4 files changed, 610 insertions(+), 41 deletions(-) create mode 100644 Docs/research/poc-step5-results.md create mode 100755 Docs/research/poc-step5.sh create mode 100644 Sources/skitrun/Helpers.swift diff --git a/Docs/research/poc-step5-results.md b/Docs/research/poc-step5-results.md new file mode 100644 index 0000000..746570e --- /dev/null +++ b/Docs/research/poc-step5-results.md @@ -0,0 +1,58 @@ +# POC Step 5 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §4 + §6 step 5. Goal: let inputs `import SyntaxKitHelpers` from a `Helpers/` directory adjacent to the input, with discovery + on-demand `swiftc` compile + a per-toolchain cache. + +## What landed + +1. **`Sources/skitrun/Helpers.swift`.** New module with three responsibilities: + - `discoverHelpersDir(near:)` walks up from the input path looking for a `Helpers/` directory. Single-file mode starts from the file's parent; folder mode starts from the input directory. Walk-up stops at the filesystem root. + - `collectHelperSources(in:)` globs `**/*.swift` under the helpers dir, skipping `_`-prefixed files (same convention as input enumeration). + - `buildHelpers(helpersDir:libPath:)` hashes the sources + Swift version + dylib stamp into a cache key under `~/Library/Caches/com.brightdigit.SyntaxKit/helpers//`. On a cache miss, it shells out to `swiftc` into a `tmp../` staging dir, then atomic-renames into the cache path (`ProjectDescriptionHelpersBuilder` pattern from Tuist). + +2. **`Sources/skitrun/Main.swift` wiring.** + - `CLIArgs` gains `--helpers ` (explicit override) and `--no-helpers` (skip discovery). Default is auto. + - `runSwift` splices `-I/-L/-lSyntaxKitHelpers -Xlinker -rpath -Xlinker ` when helpers are present. + - Folder mode's `collectInputs` now also yields directories so it can call `enumerator.skipDescendants()` when it hits a `Helpers/` directly under the input root — otherwise the helpers would be re-processed as inputs. + - Helpers compile happens **once per invocation** in folder mode (not per input file). + +3. **`Docs/research/poc-step5.sh`.** Standalone demo: builds skitrun, stages a runtime lib, writes a tiny `Helpers/Models.swift` exporting `equatableModel(_:fields:)`, and two `inputs/*.swift` files that `import SyntaxKitHelpers` and call the helper. + +## Verified flows + +| Flow | Result | +| --- | --- | +| Cold run (cache cleared, helpers compile from scratch) | ✓ 2.96s real | +| Warm run (cache hit, same helper sources) | ✓ 0.54s real | +| Folder mode against `demo/` containing `Helpers/` + `inputs/` | ✓ 2/2 succeeded, `Helpers/*.swift` not enumerated as input | +| `--no-helpers` with an input that imports `SyntaxKitHelpers` | ✓ child `swift` errors with `no such module 'SyntaxKitHelpers'`, exit non-zero | + +The cached layout for a single helper file: + +``` +~/Library/Caches/com.brightdigit.SyntaxKit/helpers// + libSyntaxKitHelpers.dylib + SyntaxKitHelpers.swiftmodule + SyntaxKitHelpers.swiftdoc + SyntaxKitHelpers.abi.json + SyntaxKitHelpers.swiftsourceinfo +``` + +## Cache key + +SHA-256 over (in order): +- Cache schema version string (`v1`). +- For each helper source (sorted by absolute path): `lastPathComponent` + file bytes. +- `swift --version` output. +- `libSyntaxKit.dylib` size and modification time (proxy for SyntaxKit version until the bundle is versioned). + +Mutating any helper source, switching toolchains, or rebuilding SyntaxKit invalidates the cache. Adding a `cacheSchemaVersion` bump constant covers future layout changes. + +## Known rough edges + +- **Helpers cold compile is the dominant cost.** 2.96s vs 0.54s warm — the helper compile is ~2.5s on top of the ~0.5s `swift` interpret cost. Once cached it's free, but the first run after a clean checkout is noticeably slow. Acceptable for v1; could be sped up by caching the helpers `.o` files separately, but that's a step-6+ optimization. +- **Walk-up false positives.** If a user happens to have an unrelated `Helpers/` somewhere up-tree (e.g. a sibling library), skitrun will try to compile it. `--helpers ` or `--no-helpers` is the escape hatch. A future heuristic could require a sentinel file (`Helpers/.syntaxkit-helpers`) before claiming the directory. +- **Import-line diagnostics off by one.** When the user's input has `import Foo` on line 1, the wrap step puts an injected `import SyntaxKit` above it, so a child-compile error on the user's import reports `:2:8` instead of `:1:8`. `#sourceLocation` directives only wrap the body, not the hoisted imports. Easy follow-up: emit a `#sourceLocation` directive per hoisted import too. + +## What's next + +Step 6: **output cache.** Today every `skitrun` invocation re-spawns `swift` to render the input, even when nothing has changed. Add the per-input output cache from [`codegen-cli-design.md` §5](./codegen-cli-design.md#5-caching), keyed by input hash + helpers-cache key + Swift version + envHash. On a hit, skip the spawn entirely and copy the rendered output to the destination. Add `--no-cache` for debugging. diff --git a/Docs/research/poc-step5.sh b/Docs/research/poc-step5.sh new file mode 100755 index 0000000..ea633a3 --- /dev/null +++ b/Docs/research/poc-step5.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +# +# POC step 5 demo: Helpers/ discovery + compile + import in input scripts. +# +# Builds skitrun, stages a runtime lib/ next to it, then runs skitrun against +# a demo project that uses `import SyntaxKitHelpers`. Demonstrates: +# 1. Cold path — Helpers/ compiles to libSyntaxKitHelpers.dylib. +# 2. Warm path — second invocation reuses the cached helpers dylib. +# 3. Folder mode — skitrun ignores Helpers/ when walking the input tree. +# 4. --no-helpers — disables discovery; the import then fails as expected. + +set -euo pipefail + +if [[ "$(uname -s)" != "Darwin" ]]; then + echo "macOS-only. Linux smoke test is POC step 7." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +POC_DIR="/tmp/syntaxkit-poc-step5" +DEMO_DIR="$POC_DIR/demo" +CACHE_DIR="$HOME/Library/Caches/com.brightdigit.SyntaxKit/helpers" +PACKAGE_FILE="$REPO_ROOT/Package.swift" +PACKAGE_BACKUP="$(mktemp)" + +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if old not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found") +p.write_text(src.replace(old, new, 1)) +PY + +cd "$REPO_ROOT" + +echo "==> swift build" +swift build + +BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate .build//debug" >&2 + exit 1 +fi + +echo "==> Staging $POC_DIR" +rm -rf "$POC_DIR" +mkdir -p "$POC_DIR/lib" "$DEMO_DIR/Helpers" "$DEMO_DIR/inputs" + +cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" +cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" +cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" +cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ + "$POC_DIR/lib/_SwiftSyntaxCShims-include" + +cat > "$DEMO_DIR/Helpers/Models.swift" <<'SWIFT' +import SyntaxKit + +public func equatableModel( + _ name: String, + fields: [(name: String, type: String)] +) -> any CodeBlock { + Struct(name) { + for field in fields { + Variable(.let, name: field.name, type: field.type) + } + }.inherits("Equatable") +} +SWIFT + +cat > "$DEMO_DIR/inputs/Person.swift" <<'SWIFT' +import SyntaxKitHelpers + +equatableModel("Person", fields: [ + ("name", "String"), + ("age", "Int"), +]) +SWIFT + +cat > "$DEMO_DIR/inputs/Pet.swift" <<'SWIFT' +import SyntaxKitHelpers + +equatableModel("Pet", fields: [ + ("kind", "String"), + ("owner", "String"), +]) +SWIFT + +echo "==> Clearing helpers cache to force cold compile" +rm -rf "$CACHE_DIR" + +echo +echo "==> Cold run (helpers compile from scratch):" +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/inputs/Person.swift" + +echo +echo "==> Warm run (helpers cache hit):" +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/inputs/Person.swift" >/dev/null + +echo +echo "==> Cached helper artifacts:" +find "$CACHE_DIR" -maxdepth 3 -type f | sed "s|$CACHE_DIR||" | sort + +echo +echo "==> Folder mode (Helpers/ excluded from input enumeration):" +rm -rf "$POC_DIR/out" +"$POC_DIR/skitrun" "$DEMO_DIR" -o "$POC_DIR/out" +echo " Generated files:" +find "$POC_DIR/out" -type f | sed "s|$POC_DIR/| |" + +echo +echo "==> --no-helpers should fail with an unresolved import:" +if "$POC_DIR/skitrun" --no-helpers "$DEMO_DIR/inputs/Person.swift" >/dev/null 2>&1; then + echo "FAIL: --no-helpers should have errored" >&2 + exit 1 +else + echo " ✓ skitrun returned non-zero as expected" +fi + +echo +echo "==> Done. Demo project kept at $DEMO_DIR; cache at $CACHE_DIR." diff --git a/Sources/skitrun/Helpers.swift b/Sources/skitrun/Helpers.swift new file mode 100644 index 0000000..dfccba3 --- /dev/null +++ b/Sources/skitrun/Helpers.swift @@ -0,0 +1,247 @@ +// +// Helpers.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import CryptoKit +import Foundation + +/// Hardcoded module name for the user's `Helpers/` compilation output. Inputs +/// reach the compiled helpers via `import SyntaxKitHelpers`. +internal let helpersModuleName = "SyntaxKitHelpers" + +/// Bumped when the cache layout changes in a way that requires invalidation. +private let helpersCacheSchemaVersion = "v1" + +/// A compiled `Helpers/` directory ready to splice into the input spawn. +internal struct CompiledHelpers: Sendable { + /// Directory containing `libSyntaxKitHelpers.dylib` + `.swiftmodule` files. + let outputDir: URL + /// Whether the build was reused from cache (false = freshly compiled). + let cacheHit: Bool +} + +// MARK: - Discovery + +/// Walks up from `inputURL` looking for a `Helpers/` directory. Returns the +/// first one found, or nil if no ancestor contains one. +/// +/// When `inputURL` is a file, the search starts from its parent. When it's a +/// directory, the search starts from the directory itself. +internal func discoverHelpersDir(near inputURL: URL) -> URL? { + let fm = FileManager.default + var isDirectory: ObjCBool = false + let exists = fm.fileExists(atPath: inputURL.path, isDirectory: &isDirectory) + var dir = (exists && isDirectory.boolValue) ? inputURL : inputURL.deletingLastPathComponent() + dir = dir.standardizedFileURL + + while true { + let candidate = dir.appendingPathComponent("Helpers") + var isDir: ObjCBool = false + if fm.fileExists(atPath: candidate.path, isDirectory: &isDir), isDir.boolValue { + return candidate.standardizedFileURL + } + let parent = dir.deletingLastPathComponent().standardizedFileURL + if parent.path == dir.path { return nil } + dir = parent + } +} + +/// Globs `**/*.swift` under `helpersDir`, skipping files prefixed with `_`. +internal func collectHelperSources(in helpersDir: URL) throws -> [URL] { + guard + let enumerator = FileManager.default.enumerator( + at: helpersDir, + includingPropertiesForKeys: [.isRegularFileKey], + options: [.skipsHiddenFiles] + ) + else { + throw CLIError(message: "could not enumerate \(helpersDir.path)") + } + + var result: [URL] = [] + for case let url as URL in enumerator { + let values = try url.resourceValues(forKeys: [.isRegularFileKey]) + guard values.isRegularFile == true else { continue } + guard url.pathExtension == "swift" else { continue } + guard !url.lastPathComponent.hasPrefix("_") else { continue } + result.append(url.standardizedFileURL) + } + return result.sorted { $0.path < $1.path } +} + +// MARK: - Build pipeline + +/// Compiles helper sources into a per-key cache directory and returns the +/// directory plus a cache-hit flag. Returns nil when the helpers dir is empty. +internal func buildHelpers( + helpersDir: URL, + libPath: String +) throws -> CompiledHelpers? { + let sources = try collectHelperSources(in: helpersDir) + if sources.isEmpty { return nil } + + let key = try helpersCacheKey(sources: sources, libPath: libPath) + let cacheRoot = try syntaxKitCacheRoot() + .appendingPathComponent("helpers") + .appendingPathComponent(key) + let dylibPath = cacheRoot.appendingPathComponent("lib\(helpersModuleName).dylib").path + + let fm = FileManager.default + if fm.fileExists(atPath: dylibPath) { + return CompiledHelpers(outputDir: cacheRoot, cacheHit: true) + } + + try fm.createDirectory( + at: cacheRoot.deletingLastPathComponent(), + withIntermediateDirectories: true + ) + + let staging = cacheRoot.deletingLastPathComponent() + .appendingPathComponent("tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)") + try fm.createDirectory(at: staging, withIntermediateDirectories: true) + + do { + try compileHelpers(sources: sources, into: staging, libPath: libPath) + } catch { + try? fm.removeItem(at: staging) + throw error + } + + // Atomic rename into the cache path. If a peer beat us to it (rename failed + // because the destination now exists), keep theirs and drop ours. + do { + try fm.moveItem(at: staging, to: cacheRoot) + } catch { + try? fm.removeItem(at: staging) + if !fm.fileExists(atPath: dylibPath) { + throw error + } + } + + return CompiledHelpers(outputDir: cacheRoot, cacheHit: false) +} + +private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) throws { + let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + let dylib = outDir.appendingPathComponent("lib\(helpersModuleName).dylib").path + let modulePath = outDir.appendingPathComponent("\(helpersModuleName).swiftmodule").path + + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/env") + var args: [String] = [ + "swiftc", + "-module-name", helpersModuleName, + "-emit-module", + "-emit-module-path", modulePath, + "-parse-as-library", + "-emit-library", + "-o", dylib, + "-suppress-warnings", + "-I", libPath, + "-L", libPath, + "-lSyntaxKit", + "-Xcc", "-I", "-Xcc", cShimsInclude, + "-Xlinker", "-install_name", + "-Xlinker", "@rpath/lib\(helpersModuleName).dylib", + "-Xlinker", "-rpath", "-Xlinker", libPath, + ] + args.append(contentsOf: sources.map(\.path)) + process.arguments = args + + let stderrPipe = Pipe() + process.standardOutput = FileHandle.nullDevice + process.standardError = stderrPipe + try process.run() + process.waitUntilExit() + + let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() + guard process.terminationStatus == 0 else { + let stderr = String(decoding: stderrData, as: UTF8.self) + throw CLIError( + message: """ + skitrun: failed to compile Helpers/ (exit \(process.terminationStatus)) + \(stderr) + """) + } +} + +// MARK: - Cache key + +private func helpersCacheKey(sources: [URL], libPath: String) throws -> String { + var hasher = SHA256() + hasher.update(data: Data(helpersCacheSchemaVersion.utf8)) + + for source in sources { + let data = try Data(contentsOf: source) + hasher.update(data: Data(source.lastPathComponent.utf8)) + hasher.update(data: data) + } + + if let swiftVersion = captureSwiftVersion() { + hasher.update(data: Data(swiftVersion.utf8)) + } + if let stamp = libStamp(libPath: libPath) { + hasher.update(data: Data(stamp.utf8)) + } + + return hasher.finalize().map { String(format: "%02x", $0) }.joined() +} + +private func captureSwiftVersion() -> String? { + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/env") + process.arguments = ["swift", "--version"] + let pipe = Pipe() + process.standardOutput = pipe + process.standardError = FileHandle.nullDevice + do { try process.run() } catch { return nil } + process.waitUntilExit() + let data = pipe.fileHandleForReading.readDataToEndOfFile() + return String(decoding: data, as: UTF8.self) +} + +private func libStamp(libPath: String) -> String? { + let dylib = "\(libPath)/libSyntaxKit.dylib" + guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } + let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 + let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 + return "\(size)/\(Int(mtime))" +} + +private func syntaxKitCacheRoot() throws -> URL { + if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { + return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") + } + let home = NSHomeDirectory() + #if os(macOS) + return URL(fileURLWithPath: home) + .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") + #else + return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") + #endif +} diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index ebaf3a3..52cd1ed 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -1,6 +1,6 @@ // // Main.swift -// SyntaxKit — skitrun (POC for issue #154) +// SyntaxKit // // Created by Leo Dion. // Copyright © 2026 BrightDigit. @@ -46,18 +46,70 @@ internal enum SkitRun { switch args.mode { case .singleFile(let input, let output): - try runSingleFile(inputPath: input, outputPath: output, libPath: libPath) + let helpers = try resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: args.helpers + ) + try runSingleFile( + inputPath: input, + outputPath: output, + libPath: libPath, + helpers: helpers + ) case .directory(let inputDir, let outputDir): + let helpers = try resolveHelpers( + nearInputPath: inputDir, + libPath: libPath, + options: args.helpers + ) let exitCode = await runDirectory( inputDir: inputDir, outputDir: outputDir, - libPath: libPath + libPath: libPath, + helpers: helpers ) exit(exitCode) } } } +// MARK: - Helpers resolution + +private func resolveHelpers( + nearInputPath path: String, + libPath: String, + options: HelpersOptions +) throws -> CompiledHelpers? { + let helpersDir: URL? + switch options { + case .disabled: + return nil + case .auto: + helpersDir = discoverHelpersDir(near: URL(fileURLWithPath: path).standardizedFileURL) + case .explicit(let dir): + let url = URL(fileURLWithPath: dir).standardizedFileURL + var isDir: ObjCBool = false + guard FileManager.default.fileExists(atPath: url.path, isDirectory: &isDir), + isDir.boolValue + else { + throw CLIError(message: "--helpers path is not a directory: \(dir)") + } + helpersDir = url + } + guard let helpersDir else { return nil } + + guard let compiled = try buildHelpers(helpersDir: helpersDir, libPath: libPath) else { + return nil + } + let suffix = compiled.cacheHit ? "cached" : "compiled" + FileHandle.standardError.write( + Data( + "skitrun: helpers \(suffix) at \(helpersDir.path)\n".utf8 + )) + return compiled +} + // MARK: - Resource location /// Resolves the directory containing `libSyntaxKit.dylib` + module files, @@ -89,15 +141,16 @@ internal func resolveLibPath(override: String?) throws -> String { if isLibDir(brewLayout) { return brewLayout } } - throw CLIError(message: """ - Could not locate SyntaxKit lib directory. Looked for: - 1. --lib (not provided) - 2. $SKITRUN_LIB_DIR (not set) - 3. /lib/ (not found) - 4. /../lib/skitrun/ (not found) - Run Docs/research/poc-step4-release.sh to produce a self-contained - release bundle under .build/skitrun-release/. - """) + throw CLIError( + message: """ + Could not locate SyntaxKit lib directory. Looked for: + 1. --lib (not provided) + 2. $SKITRUN_LIB_DIR (not set) + 3. /lib/ (not found) + 4. /../lib/skitrun/ (not found) + Run Docs/research/poc-step4-release.sh to produce a self-contained + release bundle under .build/skitrun-release/. + """) } private func isLibDir(_ path: String) -> Bool { @@ -109,8 +162,13 @@ private func isLibDir(_ path: String) -> Bool { // MARK: - Single-file mode -private func runSingleFile(inputPath: String, outputPath: String?, libPath: String) throws { - let result = try processFile(inputPath: inputPath, libPath: libPath) +private func runSingleFile( + inputPath: String, + outputPath: String?, + libPath: String, + helpers: CompiledHelpers? +) throws { + let result = try processFile(inputPath: inputPath, libPath: libPath, helpers: helpers) if !result.stderr.isEmpty { FileHandle.standardError.write(Data(result.stderr.utf8)) } @@ -126,13 +184,18 @@ private func runSingleFile(inputPath: String, outputPath: String?, libPath: Stri // MARK: - Folder mode -private func runDirectory(inputDir: String, outputDir: String, libPath: String) async -> Int32 { +private func runDirectory( + inputDir: String, + outputDir: String, + libPath: String, + helpers: CompiledHelpers? +) async -> Int32 { let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL let inputs: [URL] do { - inputs = try collectInputs(at: inputURL) + inputs = try collectInputs(at: inputURL, excluding: helpersExcludePath(inputDir: inputURL)) } catch { FileHandle.standardError.write(Data("skitrun: failed to walk \(inputDir): \(error)\n".utf8)) return 1 @@ -151,12 +214,12 @@ private func runDirectory(inputDir: String, outputDir: String, libPath: String) await withTaskGroup(of: FileOutcome.self) { group in for _ in 0.. } -private func runOne(_ input: URL, libPath: String) -> FileOutcome { +private func runOne(_ input: URL, libPath: String, helpers: CompiledHelpers?) -> FileOutcome { do { - let result = try processFile(inputPath: input.path, libPath: libPath) + let result = try processFile(inputPath: input.path, libPath: libPath, helpers: helpers) return FileOutcome(input: input, result: .success(result)) } catch { return FileOutcome(input: input, result: .failure(error)) } } -private func collectInputs(at inputDir: URL) throws -> [URL] { - guard let enumerator = FileManager.default.enumerator( - at: inputDir, - includingPropertiesForKeys: [.isRegularFileKey], - options: [.skipsHiddenFiles] - ) else { +/// Returns the path of a `Helpers/` directory living directly under `inputDir`, +/// so the folder-mode enumerator can skip its descendants. Helpers that live +/// outside the input tree don't need to be excluded (they aren't enumerated). +private func helpersExcludePath(inputDir: URL) -> String? { + let candidate = inputDir.appendingPathComponent("Helpers").standardizedFileURL + var isDir: ObjCBool = false + guard FileManager.default.fileExists(atPath: candidate.path, isDirectory: &isDir), + isDir.boolValue + else { + return nil + } + return candidate.path +} + +private func collectInputs(at inputDir: URL, excluding excludedDir: String?) throws -> [URL] { + guard + let enumerator = FileManager.default.enumerator( + at: inputDir, + includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], + options: [.skipsHiddenFiles] + ) + else { throw CLIError(message: "could not enumerate \(inputDir.path)") } var result: [URL] = [] for case let url as URL in enumerator { - let values = try url.resourceValues(forKeys: [.isRegularFileKey]) + let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + if values.isDirectory == true { + if let excludedDir, url.standardizedFileURL.path == excludedDir { + enumerator.skipDescendants() + } + continue + } guard values.isRegularFile == true else { continue } guard url.pathExtension == "swift" else { continue } guard !url.lastPathComponent.hasPrefix("_") else { continue } @@ -243,7 +329,11 @@ private struct ProcessResult { let stderr: String } -private func processFile(inputPath: String, libPath: String) throws -> ProcessResult { +private func processFile( + inputPath: String, + libPath: String, + helpers: CompiledHelpers? +) throws -> ProcessResult { let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL let absoluteInputPath = inputURL.path let source = try String(contentsOf: inputURL, encoding: .utf8) @@ -257,7 +347,7 @@ private func processFile(inputPath: String, libPath: String) throws -> ProcessRe let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) - let raw = try runSwift(wrappedPath: wrappedURL.path, libPath: libPath) + let raw = try runSwift(wrappedPath: wrappedURL.path, libPath: libPath, helpers: helpers) // #sourceLocation maps body diagnostics back to the input file. Errors in // the preamble (lines outside the body) still reference the wrapper — // rewrite literal occurrences of its path so users see something coherent. @@ -270,6 +360,12 @@ private func processFile(inputPath: String, libPath: String) throws -> ProcessRe // MARK: - Arg parsing +internal enum HelpersOptions { + case auto + case disabled + case explicit(String) +} + private struct CLIArgs { enum Mode { case singleFile(input: String, output: String?) @@ -278,11 +374,13 @@ private struct CLIArgs { let mode: Mode let libPath: String? + let helpers: HelpersOptions static func parse(_ argv: [String]) throws -> CLIArgs { var inputPath: String? var outputPath: String? var libPath: String? + var helpers: HelpersOptions = .auto var i = 1 while i < argv.count { @@ -296,6 +394,13 @@ private struct CLIArgs { guard i + 1 < argv.count else { throw usage("--lib requires a value") } libPath = argv[i + 1] i += 2 + case "--helpers": + guard i + 1 < argv.count else { throw usage("--helpers requires a value") } + helpers = .explicit(argv[i + 1]) + i += 2 + case "--no-helpers": + helpers = .disabled + i += 1 case "-h", "--help": FileHandle.standardError.write(Data(helpText.utf8)) exit(0) @@ -325,7 +430,7 @@ private struct CLIArgs { mode = .singleFile(input: inputPath, output: outputPath) } - return CLIArgs(mode: mode, libPath: libPath) + return CLIArgs(mode: mode, libPath: libPath, helpers: helpers) } } @@ -349,13 +454,18 @@ private let helpText = """ then /lib/, then /../lib/skitrun/. Build a self-contained bundle with Docs/research/poc-step4-release.sh. + --helpers Override Helpers/ directory location. By default, + skitrun walks up from the input looking for one. + Compiled into libSyntaxKitHelpers.dylib and made + importable via `import SyntaxKitHelpers`. + --no-helpers Skip helpers discovery entirely. """ private func usage(_ message: String) -> CLIError { CLIError(message: "\(message)\n\n\(helpText)\n") } -private struct CLIError: Error, CustomStringConvertible { +internal struct CLIError: Error, CustomStringConvertible { let message: String var description: String { message } } @@ -379,7 +489,8 @@ internal func wrap(source: String, originalPath: String) -> String { for item in tree.statements { if let importDecl = item.item.as(ImportDeclSyntax.self), - firstBodyByte == nil { + firstBodyByte == nil + { hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) continue } @@ -402,7 +513,8 @@ internal func wrap(source: String, originalPath: String) -> String { // #sourceLocation must use a forward-slash path; escape backslashes/quotes // defensively even though macOS paths shouldn't contain them. - let escapedPath = originalPath + let escapedPath = + originalPath .replacingOccurrences(of: "\\", with: "\\\\") .replacingOccurrences(of: "\"", with: "\\\"") @@ -421,12 +533,14 @@ internal func wrap(source: String, originalPath: String) -> String { // MARK: - Spawning swift -private func runSwift(wrappedPath: String, libPath: String) throws -> ProcessResult { +private func runSwift( + wrappedPath: String, + libPath: String, + helpers: CompiledHelpers? +) throws -> ProcessResult { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - let process = Process() - process.executableURL = URL(fileURLWithPath: "/usr/bin/env") - process.arguments = [ + var arguments: [String] = [ "swift", "-suppress-warnings", "-I", libPath, @@ -434,9 +548,24 @@ private func runSwift(wrappedPath: String, libPath: String) throws -> ProcessRes "-lSyntaxKit", "-Xcc", "-I", "-Xcc", cShimsInclude, "-Xlinker", "-rpath", "-Xlinker", libPath, - wrappedPath ] + if let helpers { + let helpersPath = helpers.outputDir.path + arguments.append(contentsOf: [ + "-I", helpersPath, + "-L", helpersPath, + "-l\(helpersModuleName)", + "-Xlinker", "-rpath", "-Xlinker", helpersPath, + ]) + } + + arguments.append(wrappedPath) + + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/env") + process.arguments = arguments + let stdoutPipe = Pipe() let stderrPipe = Pipe() process.standardOutput = stdoutPipe From 2adc3cd4ada399ede9140c46200bcd11ee6e5305 Mon Sep 17 00:00:00 2001 From: leogdion Date: Tue, 12 May 2026 08:46:04 -0400 Subject: [PATCH 09/28] POC step 6: rendered-output cache + --no-cache (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit skitrun now hashes (input bytes + helpers cache key + swift --version + libSyntaxKit stamp + sorted SKITRUN_*/SYNTAXKIT_* env vars) into a SHA-256 key under ~/Library/Caches/com.brightdigit.SyntaxKit/outputs/. On hit, processFile short-circuits — no temp wrapper, no swift spawn, just return the cached bytes. On miss the normal compile path runs and stores the rendered output. Only exit-0 runs are cached so failures always re-spawn with fresh diagnostics. Atomic write through a tmp../ staging dir + rename mirrors the helpers cache; concurrent writers race safely, the loser drops their copy when the destination already exists. --no-cache disables the lookup wholesale (debugging, after manual cache deletion). Threads through runSingleFile + runDirectory so folder mode can opt out batch-wide. Helpers.swift's syntaxKitCacheRoot/captureSwiftVersion/libStamp are now internal so OutputCache.swift can reuse them without duplication. Verified via Docs/research/poc-step6.sh: cold 0.55s → warm 0.14s (4× faster, no swift spawn), --no-cache 0.27s, post-mutation miss 0.41s then 0.14s on the new key, two cache entries persist. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step6-results.md | 58 +++++++++++++++ Docs/research/poc-step6.sh | 116 +++++++++++++++++++++++++++++ Sources/skitrun/Helpers.swift | 6 +- Sources/skitrun/Main.swift | 65 +++++++++++++--- Sources/skitrun/OutputCache.swift | 111 +++++++++++++++++++++++++++ 5 files changed, 342 insertions(+), 14 deletions(-) create mode 100644 Docs/research/poc-step6-results.md create mode 100755 Docs/research/poc-step6.sh create mode 100644 Sources/skitrun/OutputCache.swift diff --git a/Docs/research/poc-step6-results.md b/Docs/research/poc-step6-results.md new file mode 100644 index 0000000..f50fc85 --- /dev/null +++ b/Docs/research/poc-step6-results.md @@ -0,0 +1,58 @@ +# POC Step 6 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §5 + §6 step 6. Goal: skip the `swift` spawn entirely when the rendered output for an input is already cached, with a key that captures everything that could change the output. + +## What landed + +1. **`Sources/skitrun/OutputCache.swift`.** Three functions: + - `outputCacheKey(inputSource:helpers:libPath:)` — SHA-256 over cache schema version + input bytes + helpers cache key (or `"no-helpers"`) + `swift --version` + libSyntaxKit dylib stamp + sorted `SKITRUN_*` / `SYNTAXKIT_*` env vars. + - `lookupCachedOutput(key:)` — returns the cached `output.swift` bytes from `~/Library/Caches/com.brightdigit.SyntaxKit/outputs//output.swift`, or `nil` on miss. + - `storeCachedOutput(key:data:)` — atomic write via `tmp../` staging dir + rename. Concurrent writers race safely; the loser drops their copy if the destination already exists. + +2. **`Sources/skitrun/Main.swift` wiring.** + - `processFile` reads the input source, computes the cache key, and short-circuits on hit — no temp wrapper, no `swift` spawn, just return the cached bytes with `exitCode = 0`. + - On miss, the normal wrap + spawn path runs; if `swift` returns 0, the rendered output is stored under the key. + - Only successful (exit 0) runs are cached. Failed runs always re-spawn so the user sees fresh diagnostics. + - `CLIArgs` gains `--no-cache` to skip the cache entirely (still useful when chasing flaky output or after manually deleting the cache). + - The flag threads through `runSingleFile` and `runDirectory` so folder mode can opt out wholesale. + +3. **Helpers.swift internal exposure.** `syntaxKitCacheRoot`, `captureSwiftVersion`, and `libStamp` were `private`; they're now `internal` so OutputCache can reuse them rather than duplicating. + +## Verified flows + +From `Docs/research/poc-step6.sh` (single input, no helpers): + +| Run | Real time | Notes | +| --- | ---: | --- | +| Cold (cache cleared) | 0.55s | swift spawn + compile + store | +| Warm (cache hit) | 0.14s | FS read only, no swift spawn | +| `--no-cache` | 0.27s | always spawn, ignore cache | +| After mutation (miss) | 0.41s | new key, recompile, store | +| Warm after mutation | 0.14s | second key cached | + +After mutation, the cache directory contains **two** entries (one per input version) — old keys aren't evicted, which is fine for a per-toolchain cache where stale entries are dead weight, not correctness risks. Eviction can be a follow-up if cache size becomes a complaint. + +## Cache key, written out + +``` +SHA-256( + "v1" // cache schema version + + input.swift bytes // verbatim user input + + helpersCacheKey || "no-helpers" // helpers fingerprint (sibling cache) + + swift --version stdout // toolchain fingerprint + + "/" of libSyntaxKit.dylib // SyntaxKit fingerprint proxy + + sorted SKITRUN_*/SYNTAXKIT_* env (k=v\0) // env override sensitivity +) +``` + +Two cooperating cache layers — helpers (step 5) and outputs (step 6) — sit side-by-side under `~/Library/Caches/com.brightdigit.SyntaxKit/{helpers,outputs}//`. Helpers cache hits are reused across many inputs; output cache hits are per-input. + +## Known rough edges + +- **Output cache stores stdout only.** Stderr from a successful run (e.g. warnings even with `-suppress-warnings` off) is discarded on a hit. With `-suppress-warnings` in `runSwift` this is rarely visible, but it does mean cache hits suppress warnings that would have appeared on a fresh run. Acceptable for a generator; revisit if SyntaxKit grows runtime-side warnings. +- **`libStamp` is a coarse proxy.** Size + mtime catches normal rebuilds but a deterministic rebuild that preserves both would slip past. Hashing the dylib is correct but slow (9.3 MB per invocation defeats the cache). The right long-term fix is embedding a SyntaxKit version constant the bundle exports. +- **No size cap or eviction.** A repo that touches inputs frequently will accrete cache entries. Each is small (a few hundred bytes of rendered Swift) so the practical ceiling is high, but a `--prune` subcommand is a reasonable v1.1 addition. + +## What's next + +Step 7: **Linux smoke test.** Confirm `/usr/bin/env swift` + the bundled-dylib layout works on Linux without `-F` framework search paths — `-I + -L + -lSyntaxKit` should be sufficient per Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch. CryptoKit is macOS-only; Linux will need `swift-crypto` or a small fallback hash impl behind a `#if canImport(CryptoKit)` shim. After that, the 7-step ladder is complete. diff --git a/Docs/research/poc-step6.sh b/Docs/research/poc-step6.sh new file mode 100755 index 0000000..ad90c70 --- /dev/null +++ b/Docs/research/poc-step6.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# +# POC step 6 demo: rendered-output cache. skitrun skips the `swift` spawn +# when input + helpers + toolchain are unchanged. +# +# Builds skitrun, stages a runtime lib next to it, runs an input, then +# replays it three ways: +# 1. with cache — should be near-instant (no swift spawn) +# 2. --no-cache — always spawns swift +# 3. mutated input — invalidates the cache key, falls back to swift + +set -euo pipefail + +if [[ "$(uname -s)" != "Darwin" ]]; then + echo "macOS-only. Linux smoke test is POC step 7." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +POC_DIR="/tmp/syntaxkit-poc-step6" +DEMO_DIR="$POC_DIR/demo" +OUTPUT_CACHE="$HOME/Library/Caches/com.brightdigit.SyntaxKit/outputs" +PACKAGE_FILE="$REPO_ROOT/Package.swift" +PACKAGE_BACKUP="$(mktemp)" + +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if old not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found") +p.write_text(src.replace(old, new, 1)) +PY + +cd "$REPO_ROOT" + +echo "==> swift build" +swift build + +BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate .build//debug" >&2 + exit 1 +fi + +echo "==> Staging $POC_DIR" +rm -rf "$POC_DIR" +mkdir -p "$POC_DIR/lib" "$DEMO_DIR" + +cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" +cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" +cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" +cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ + "$POC_DIR/lib/_SwiftSyntaxCShims-include" + +cat > "$DEMO_DIR/Input.swift" <<'SWIFT' +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} +SWIFT + +echo "==> Clearing output cache to force a cold run" +rm -rf "$OUTPUT_CACHE" + +echo +echo "==> Cold run (cache miss → swift spawn → store):" +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> Warm run (cache hit → no swift spawn):" +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> --no-cache (always spawn swift, even with cache present):" +/usr/bin/time -p "$POC_DIR/skitrun" --no-cache "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> Output cache contents:" +find "$OUTPUT_CACHE" -maxdepth 3 -type f | sed "s|$OUTPUT_CACHE||" | sort + +echo +echo "==> Mutating input invalidates the cache:" +cat > "$DEMO_DIR/Input.swift" <<'SWIFT' +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") + Variable(.let, name: "email", type: "String") +} +SWIFT +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> Warm run after mutation:" +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> Cache now contains two distinct keys:" +find "$OUTPUT_CACHE" -maxdepth 1 -mindepth 1 -type d | wc -l | xargs -I {} echo " {} cache entries" + +echo +echo "==> Done. Cache at $OUTPUT_CACHE." diff --git a/Sources/skitrun/Helpers.swift b/Sources/skitrun/Helpers.swift index dfccba3..d8c3d7f 100644 --- a/Sources/skitrun/Helpers.swift +++ b/Sources/skitrun/Helpers.swift @@ -212,7 +212,7 @@ private func helpersCacheKey(sources: [URL], libPath: String) throws -> String { return hasher.finalize().map { String(format: "%02x", $0) }.joined() } -private func captureSwiftVersion() -> String? { +internal func captureSwiftVersion() -> String? { let process = Process() process.executableURL = URL(fileURLWithPath: "/usr/bin/env") process.arguments = ["swift", "--version"] @@ -225,7 +225,7 @@ private func captureSwiftVersion() -> String? { return String(decoding: data, as: UTF8.self) } -private func libStamp(libPath: String) -> String? { +internal func libStamp(libPath: String) -> String? { let dylib = "\(libPath)/libSyntaxKit.dylib" guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 @@ -233,7 +233,7 @@ private func libStamp(libPath: String) -> String? { return "\(size)/\(Int(mtime))" } -private func syntaxKitCacheRoot() throws -> URL { +internal func syntaxKitCacheRoot() throws -> URL { if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") } diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index 52cd1ed..9d8baf0 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -55,7 +55,8 @@ internal enum SkitRun { inputPath: input, outputPath: output, libPath: libPath, - helpers: helpers + helpers: helpers, + useCache: args.useCache ) case .directory(let inputDir, let outputDir): let helpers = try resolveHelpers( @@ -67,7 +68,8 @@ internal enum SkitRun { inputDir: inputDir, outputDir: outputDir, libPath: libPath, - helpers: helpers + helpers: helpers, + useCache: args.useCache ) exit(exitCode) } @@ -166,9 +168,15 @@ private func runSingleFile( inputPath: String, outputPath: String?, libPath: String, - helpers: CompiledHelpers? + helpers: CompiledHelpers?, + useCache: Bool ) throws { - let result = try processFile(inputPath: inputPath, libPath: libPath, helpers: helpers) + let result = try processFile( + inputPath: inputPath, + libPath: libPath, + helpers: helpers, + useCache: useCache + ) if !result.stderr.isEmpty { FileHandle.standardError.write(Data(result.stderr.utf8)) } @@ -188,7 +196,8 @@ private func runDirectory( inputDir: String, outputDir: String, libPath: String, - helpers: CompiledHelpers? + helpers: CompiledHelpers?, + useCache: Bool ) async -> Int32 { let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL @@ -214,12 +223,12 @@ private func runDirectory( await withTaskGroup(of: FileOutcome.self) { group in for _ in 0.. } -private func runOne(_ input: URL, libPath: String, helpers: CompiledHelpers?) -> FileOutcome { +private func runOne( + _ input: URL, + libPath: String, + helpers: CompiledHelpers?, + useCache: Bool +) -> FileOutcome { do { - let result = try processFile(inputPath: input.path, libPath: libPath, helpers: helpers) + let result = try processFile( + inputPath: input.path, + libPath: libPath, + helpers: helpers, + useCache: useCache + ) return FileOutcome(input: input, result: .success(result)) } catch { return FileOutcome(input: input, result: .failure(error)) @@ -332,11 +351,21 @@ private struct ProcessResult { private func processFile( inputPath: String, libPath: String, - helpers: CompiledHelpers? + helpers: CompiledHelpers?, + useCache: Bool ) throws -> ProcessResult { let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL let absoluteInputPath = inputURL.path let source = try String(contentsOf: inputURL, encoding: .utf8) + + let cacheKey: String? = + useCache + ? outputCacheKey(inputSource: source, helpers: helpers, libPath: libPath) + : nil + if let cacheKey, let cached = lookupCachedOutput(key: cacheKey) { + return ProcessResult(exitCode: 0, stdout: cached, stderr: "") + } + let wrapped = wrap(source: source, originalPath: absoluteInputPath) let tmpDir = FileManager.default.temporaryDirectory @@ -355,6 +384,11 @@ private func processFile( of: wrappedURL.path, with: absoluteInputPath ) + + if let cacheKey, raw.exitCode == 0 { + try? storeCachedOutput(key: cacheKey, data: raw.stdout) + } + return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } @@ -375,12 +409,14 @@ private struct CLIArgs { let mode: Mode let libPath: String? let helpers: HelpersOptions + let useCache: Bool static func parse(_ argv: [String]) throws -> CLIArgs { var inputPath: String? var outputPath: String? var libPath: String? var helpers: HelpersOptions = .auto + var useCache = true var i = 1 while i < argv.count { @@ -401,6 +437,9 @@ private struct CLIArgs { case "--no-helpers": helpers = .disabled i += 1 + case "--no-cache": + useCache = false + i += 1 case "-h", "--help": FileHandle.standardError.write(Data(helpText.utf8)) exit(0) @@ -430,7 +469,7 @@ private struct CLIArgs { mode = .singleFile(input: inputPath, output: outputPath) } - return CLIArgs(mode: mode, libPath: libPath, helpers: helpers) + return CLIArgs(mode: mode, libPath: libPath, helpers: helpers, useCache: useCache) } } @@ -459,6 +498,10 @@ private let helpText = """ Compiled into libSyntaxKitHelpers.dylib and made importable via `import SyntaxKitHelpers`. --no-helpers Skip helpers discovery entirely. + --no-cache Skip the rendered-output cache (always run swift). + The cache lives at /outputs// + and is keyed on input bytes, helpers, swift version, + libSyntaxKit stamp, and SKITRUN_*/SYNTAXKIT_* env. """ private func usage(_ message: String) -> CLIError { diff --git a/Sources/skitrun/OutputCache.swift b/Sources/skitrun/OutputCache.swift new file mode 100644 index 0000000..b951bff --- /dev/null +++ b/Sources/skitrun/OutputCache.swift @@ -0,0 +1,111 @@ +// +// OutputCache.swift +// SyntaxKit — skitrun (POC step 6 for issue #154) +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import CryptoKit +import Foundation + +/// Bumped when the output cache layout changes in a way that requires invalidation. +private let outputCacheSchemaVersion = "v1" + +/// SHA-256 over (cache schema, input source bytes, helpers key, swift version, +/// libSyntaxKit stamp, sorted SKITRUN_*/SYNTAXKIT_* env vars). Any change in +/// these inputs produces a fresh key and forces a recompile. +internal func outputCacheKey( + inputSource: String, + helpers: CompiledHelpers?, + libPath: String +) -> String { + var hasher = SHA256() + hasher.update(data: Data(outputCacheSchemaVersion.utf8)) + hasher.update(data: Data(inputSource.utf8)) + + if let helpers { + // Helpers cache dir name *is* the helpers cache key (per Helpers.swift). + hasher.update(data: Data(helpers.outputDir.lastPathComponent.utf8)) + } else { + hasher.update(data: Data("no-helpers".utf8)) + } + + if let version = captureSwiftVersion() { + hasher.update(data: Data(version.utf8)) + } + if let stamp = libStamp(libPath: libPath) { + hasher.update(data: Data(stamp.utf8)) + } + + let env = ProcessInfo.processInfo.environment + .filter { $0.key.hasPrefix("SKITRUN_") || $0.key.hasPrefix("SYNTAXKIT_") } + .sorted { $0.key < $1.key } + for (key, value) in env { + hasher.update(data: Data("\(key)=\(value)\0".utf8)) + } + + return hasher.finalize().map { String(format: "%02x", $0) }.joined() +} + +/// Returns the cached rendered output for `key`, or nil on miss. +internal func lookupCachedOutput(key: String) -> Data? { + guard let dir = try? outputCacheDir(for: key) else { return nil } + return try? Data(contentsOf: dir.appendingPathComponent("output.swift")) +} + +/// Atomically stores `data` under `key`. Concurrent writers race via a +/// `tmp../` staging dir + rename; the loser drops their copy. +internal func storeCachedOutput(key: String, data: Data) throws { + let cacheRoot = try outputCacheDir(for: key) + let final = cacheRoot.appendingPathComponent("output.swift") + let fm = FileManager.default + + try fm.createDirectory( + at: cacheRoot.deletingLastPathComponent(), + withIntermediateDirectories: true + ) + + let staging = cacheRoot.deletingLastPathComponent() + .appendingPathComponent( + "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)" + ) + try fm.createDirectory(at: staging, withIntermediateDirectories: true) + try data.write(to: staging.appendingPathComponent("output.swift")) + + do { + try fm.moveItem(at: staging, to: cacheRoot) + } catch { + try? fm.removeItem(at: staging) + if !fm.fileExists(atPath: final.path) { + throw error + } + } +} + +private func outputCacheDir(for key: String) throws -> URL { + try syntaxKitCacheRoot() + .appendingPathComponent("outputs") + .appendingPathComponent(key) +} From 68e49f0af7136abccc9de7c2eb772b477c3da785 Mon Sep 17 00:00:00 2001 From: leogdion Date: Tue, 12 May 2026 08:49:24 -0400 Subject: [PATCH 10/28] Note web-server form as a post-CLI follow-up (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a §7 bullet recording that a long-lived HTTP server variant is on the table after the 7-step CLI ladder finishes — warm `swift` reuse across requests, shared helpers/output caches across tenants, but new isolation/request-shape questions. Captured here so it doesn't get lost between now and step 7. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/codegen-cli-design.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Docs/research/codegen-cli-design.md b/Docs/research/codegen-cli-design.md index 0056f96..601b4e0 100644 --- a/Docs/research/codegen-cli-design.md +++ b/Docs/research/codegen-cli-design.md @@ -157,3 +157,4 @@ Each step is independently shippable and de-risks the next. - **What if a single input script needs to produce multiple files?** Out of scope for v1. Split into multiple inputs and use folder mode. If demand materializes, we can layer in a `--multi` envelope mode (a script writes a small JSON manifest to stdout, CLI fans out to multiple files) without breaking single-file semantics. - **Sandboxing.** Out of scope for v1. Input scripts are user-owned code in their own repo — running them has the same threat model as running `swift Input.swift` by hand. Revisit if/when this CLI runs untrusted scripts (CI for OSS contributions, etc.). - **Timeout.** Add one. Tuist's omission (Phase 1 §7) is a bug, not a feature. 60s default `Process` wait with `SIGTERM` → 5s grace → `SIGKILL`. Override via `--timeout `. +- **Web-server form.** Out of scope for the CLI POC, but on the table as a follow-up once the 7-step ladder is done. A long-lived server could reuse a warm `swift` interpreter across requests and share the helpers + output caches across tenants — both of which the CLI gives up on every invocation. Open design questions: request shape (raw DSL POST vs. structured), whether helpers are uploaded per-request or baked into the server image, and isolation between requests (the CLI's "run user code in your own repo" threat model doesn't transfer). Revisit after step 7. From 65f21034f7953ec8b6f86b8d258eb0a2b588e22c Mon Sep 17 00:00:00 2001 From: leogdion Date: Tue, 12 May 2026 10:01:25 -0400 Subject: [PATCH 11/28] POC step 7: Linux smoke test + Foundation.Process workarounds (#154) Verifies the 7-step skitrun POC ladder works on Linux. Tests via Docker (swift:6.0-jammy/aarch64): cold 0.73s, warm cache hit 0.26s, --no-cache 0.30s. Rendered output matches macOS exactly. The real find of this step is a Foundation.Process bug: on Linux, `waitUntilExit()` blocks indefinitely on already-exited children even when EOF on the pipe proves the child has terminated. Reproduced deterministically in `captureSwiftVersion`, where the parent reads all 76 bytes of `swift --version` output and the subsequent wait never returns. Workaround applied to all three call sites (captureSwiftVersion, compileHelpers, runSwift): let semaphore = DispatchSemaphore(value: 0) process.terminationHandler = { _ in semaphore.signal() } try process.run() // ... drain pipes ... semaphore.wait() runSwift additionally drains both stdout + stderr pipes concurrently via DispatchGroup + a PipeDataBox class (boxing for Swift 6 strict concurrency without `@unchecked Sendable` on local vars). Sequential reads would deadlock when either pipe (~64 KB) fills before the child exits. Other Linux-portability changes: * Switch `import CryptoKit` to `import Crypto` in Helpers.swift + OutputCache.swift; add swift-crypto 3.0.0 as a skitrun dep so the same SHA256 API works on both platforms. * dylibFilename(forLibrary:) returns libX.dylib on Apple / libX.so on Linux. Threaded through isLibDir, libStamp, the helpers cache hit check, and `swiftc -emit-library -o ...`. * `-Xlinker -install_name @rpath/...` is Mach-O specific. Wrapped in #if !os(Linux). `-Xlinker -rpath` works on both and is what actually locates the dylib at runtime. Demo script `Docs/research/poc-step7.sh` self-reruns inside the swift container when invoked from the host; uses .build-linux/ as a separate build path so the host's .build/ stays untouched (gitignored). Co-Authored-By: Claude Opus 4.7 (1M context) --- .gitignore | 1 + Docs/research/poc-step7-results.md | 45 ++++++++++ Docs/research/poc-step7.sh | 135 +++++++++++++++++++++++++++++ Package.resolved | 20 ++++- Package.swift | 6 +- Sources/skitrun/Helpers.swift | 45 ++++++++-- Sources/skitrun/Main.swift | 38 ++++++-- Sources/skitrun/OutputCache.swift | 2 +- 8 files changed, 274 insertions(+), 18 deletions(-) create mode 100644 Docs/research/poc-step7-results.md create mode 100755 Docs/research/poc-step7.sh diff --git a/.gitignore b/.gitignore index 1e2c5c4..1cee559 100644 --- a/.gitignore +++ b/.gitignore @@ -78,6 +78,7 @@ playground.xcworkspace .swiftpm .build/ +.build-linux/ # CocoaPods # We recommend against adding the Pods directory to your .gitignore. However diff --git a/Docs/research/poc-step7-results.md b/Docs/research/poc-step7-results.md new file mode 100644 index 0000000..a9a40fe --- /dev/null +++ b/Docs/research/poc-step7-results.md @@ -0,0 +1,45 @@ +# POC Step 7 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 7. Goal: confirm the `skitrun` flow that worked on macOS (steps 1-6) also works on Linux with the bundled-dylib layout, no framework search paths, and no Apple-only crypto. + +## What landed + +1. **`swift-crypto` swap-in.** `Sources/skitrun/Helpers.swift` and `OutputCache.swift` now `import Crypto` instead of `CryptoKit`. The Apple `swift-crypto` package vends the same `SHA256` API on every platform and re-exports CryptoKit on Apple platforms when available, so there's no `#if` shim needed in skitrun's own code. Added as a Package dependency at `from: "3.0.0"`. + +2. **Platform-aware dylib filename.** A new `dylibFilename(forLibrary:)` helper returns `libX.dylib` on Apple / `libX.so` on Linux. All call sites that hard-coded `libSyntaxKit.dylib` or `libSyntaxKitHelpers.dylib` now go through it: `isLibDir`, `libStamp`, helpers cache-hit check, and the `swiftc -emit-library -o` path. + +3. **Skip `@rpath` install-name on Linux.** `compileHelpers` previously passed `-Xlinker -install_name -Xlinker @rpath/libSyntaxKitHelpers.dylib`. That flag is Mach-O specific; on Linux it errors at link time. Wrapped in `#if !os(Linux)`. The `-Xlinker -rpath` flag still works on both platforms and is what actually locates the dylib at runtime. + +4. **`Docs/research/poc-step7.sh`.** Self-rerunning Docker wrapper: when invoked on the host it re-execs itself inside `swift:6.0-jammy` with the repo bind-mounted; inside the container it flips Package.swift to dynamic, builds with `--build-path .build-linux` (separate from the macOS host's `.build`), stages a `lib/` next to `skitrun`, and runs cold + warm + `--no-cache` against an input that uses helpers. + +## Verified flows + +From `Docs/research/poc-step7.sh` inside `swift:6.0-jammy` (aarch64): + +| Flow | Time | Notes | +| --- | ---: | --- | +| Cold (helpers compile + output cache miss) | 0.73s | swiftc spawns for helpers, swift interprets the wrapped input | +| Warm (output cache hit) | 0.26s | no swift spawn | +| `--no-cache` | 0.30s | swift spawn, helpers reused from cache | + +Rendered output for the demo `Person` struct matches the macOS step 5 output exactly — same SyntaxKit, same generator, no platform-specific quirks in the rendered code. + +## Linux-only surprises + +- **`Process.waitUntilExit()` hangs on already-exited children.** This was the biggest find. Foundation's `Process.waitUntilExit()` on `swift:6.0-jammy/aarch64` blocks indefinitely even when the child has clearly exited (stdout EOF observed, all 76 bytes of `swift --version` already read). Fix applied to all three callers (`captureSwiftVersion`, `compileHelpers`, `runSwift`): set `process.terminationHandler = { _ in semaphore.signal() }` before `run()`, then `semaphore.wait()` instead of `waitUntilExit()`. Took down a 20-minute mystery hang. +- **Stdout/stderr pipe drain order matters.** Linux pipe buffers are ~64 KB; reading sequentially after waiting for child exit deadlocks when the child fills either pipe. `runSwift` now drains both pipes concurrently via `DispatchGroup` + a `PipeDataBox` class (the boxing satisfies Swift 6 strict-concurrency without `@unchecked Sendable` on local vars). `compileHelpers` only needs to drain stderr (stdout goes to `/dev/null`), but drains before the wait for the same reason. +- **`-Xlinker -install_name @rpath/...` is Mach-O specific.** Errors out on Linux's GNU ld. Wrapped in `#if !os(Linux)` in `compileHelpers`. The `-Xlinker -rpath -Xlinker ` flag still works on both platforms and is what actually locates the dylib at runtime. +- **`/usr/bin/time` isn't installed in `swift:6.0-jammy` by default.** Demo script uses the bash builtin `time` for timing, which writes a different format but is portable. +- **`swift build --product skitrun` alone doesn't emit `libSyntaxKit.so`.** The first `poc-step7.sh` draft scoped the build to just the executable, which produced a 60 MB statically-linked binary and no dylib at all. Plain `swift build` (all products) emits both `libSyntaxKit.so` and `skitrun`, matching the macOS steps 5-6 scripts. +- **First-time build cost.** Cold dependency resolution + boringssl C compile (pulled in by `swift-crypto` on Linux where CommonCrypto isn't available) takes ~3 min in `swift:6.0-jammy/aarch64`. Subsequent runs reuse `.build-linux/` and finish in ~40s. +- **Crypto on Linux brings boringssl.** `swift-crypto` statically links boringssl on non-Apple platforms. `skitrun`'s Linux binary is therefore noticeably larger than the macOS one (boringssl C blobs add up). Not a correctness issue, just a size note for future packaging. + +## What's next + +The 7-step POC ladder is **complete**. With this commit: + +- Cold-start cost has been measured on both platforms. +- The bundled-dylib + script-mode `swift` invocation works on macOS and Linux. +- Folder mode, helpers, and the rendered-output cache all behave the same way on both. + +The remaining bullets in [`codegen-cli-design.md` §7](./codegen-cli-design.md#7-what-we-still-need-to-verify) — timeouts, the splice-fidelity audit beyond the demo inputs, `@main`/attribute behavior in script-mode swift, the multi-file output question, and the web-server form — are now the natural next conversation. None of them block productizing the CLI; all of them are scope decisions rather than open technical risks. diff --git a/Docs/research/poc-step7.sh b/Docs/research/poc-step7.sh new file mode 100755 index 0000000..1d6c98e --- /dev/null +++ b/Docs/research/poc-step7.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +# +# POC step 7: Linux smoke test for skitrun. +# +# Runs inside a swift:6.0-jammy container. Builds skitrun, stages a +# runtime lib/ next to it (libSyntaxKit.so + Modules + _SwiftSyntaxCShims +# headers), then exercises single-file mode, helpers, and the output +# cache — the same flows POC steps 5 and 6 verified on macOS. +# +# Usage (from macOS host or Linux host with docker): +# Docs/research/poc-step7.sh +# +# Override the image with $SKITRUN_LINUX_IMAGE. +# +# To save time across runs, the script uses .build-linux/ as a separate +# build directory so the host's .build/ stays clean and SwiftSyntax +# doesn't re-download on every invocation. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +IMAGE="${SKITRUN_LINUX_IMAGE:-swift:6.0-jammy}" + +if [[ ! -f /.dockerenv ]]; then + # ---- Host side: invoke ourselves inside the swift container. ---- + if ! command -v docker >/dev/null; then + echo "docker is required for POC step 7" >&2 + exit 1 + fi + echo "==> Running POC step 7 inside $IMAGE" + exec docker run --rm -t \ + -v "$REPO_ROOT:/workspace" \ + -w /workspace \ + -e SKITRUN_INSIDE_DOCKER=1 \ + "$IMAGE" \ + /workspace/Docs/research/poc-step7.sh +fi + +# ---- Container side: do the real work. ---- + +PACKAGE_FILE="Package.swift" +PACKAGE_BACKUP="$(mktemp)" +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> swift --version" +swift --version + +echo +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if old not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found") +p.write_text(src.replace(old, new, 1)) +PY + +echo +echo "==> swift build (build path: .build-linux)" +swift build --build-path .build-linux + +BUILD_DIR="$(ls -d .build-linux/*-unknown-linux-gnu/debug 2>/dev/null | head -1)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate Linux build dir under .build-linux/" >&2 + ls -la .build-linux/ || true + exit 1 +fi + +POC_DIR=/tmp/syntaxkit-poc-step7 +DEMO_DIR="$POC_DIR/demo" +OUTPUT_CACHE="$HOME/.cache/syntaxkit/outputs" + +rm -rf "$POC_DIR" "$HOME/.cache/syntaxkit" +mkdir -p "$POC_DIR/lib" "$DEMO_DIR/Helpers" + +cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" +cp "$BUILD_DIR/libSyntaxKit.so" "$POC_DIR/lib/" +cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" +cp -r .build-linux/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include \ + "$POC_DIR/lib/_SwiftSyntaxCShims-include" + +cat > "$DEMO_DIR/Helpers/Models.swift" <<'SWIFT' +import SyntaxKit + +public func equatableModel( + _ name: String, + fields: [(name: String, type: String)] +) -> any CodeBlock { + Struct(name) { + for field in fields { + Variable(.let, name: field.name, type: field.type) + } + }.inherits("Equatable") +} +SWIFT + +cat > "$DEMO_DIR/Input.swift" <<'SWIFT' +import SyntaxKitHelpers + +equatableModel("Person", fields: [ + ("name", "String"), + ("age", "Int"), +]) +SWIFT + +echo +echo "==> Cold run (helpers compile + output cache miss):" +time "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" + +echo +echo "==> Warm run (output cache hit, no swift spawn):" +time "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> --no-cache (swift spawn, helpers reused):" +time "$POC_DIR/skitrun" --no-cache "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> Output cache entries:" +find "$OUTPUT_CACHE" -maxdepth 2 -type f 2>/dev/null | sed "s|$OUTPUT_CACHE||" | sort + +echo +echo "==> Linux smoke test passed." diff --git a/Package.resolved b/Package.resolved index 24cfefa..851eeb1 100644 --- a/Package.resolved +++ b/Package.resolved @@ -1,6 +1,24 @@ { - "originHash" : "482d43aff5bb5c075d237e0ea17c12ee2c043b2642e459260752aa1848a20593", + "originHash" : "6a75ee274433215501c77ebca768e3c82c684598296596b112f9800ac08fa2fe", "pins" : [ + { + "identity" : "swift-asn1", + "kind" : "remoteSourceControl", + "location" : "https://github.com/apple/swift-asn1.git", + "state" : { + "revision" : "9f542610331815e29cc3821d3b6f488db8715517", + "version" : "1.6.0" + } + }, + { + "identity" : "swift-crypto", + "kind" : "remoteSourceControl", + "location" : "https://github.com/apple/swift-crypto.git", + "state" : { + "revision" : "95ba0316a9b733e92bb6b071255ff46263bbe7dc", + "version" : "3.15.1" + } + }, { "identity" : "swift-docc-plugin", "kind" : "remoteSourceControl", diff --git a/Package.swift b/Package.swift index 1087718..13c44a1 100644 --- a/Package.swift +++ b/Package.swift @@ -102,7 +102,8 @@ let package = Package( ], dependencies: [ .package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.1"), - .package(url: "https://github.com/swiftlang/swift-docc-plugin", from: "1.4.0") + .package(url: "https://github.com/swiftlang/swift-docc-plugin", from: "1.4.0"), + .package(url: "https://github.com/apple/swift-crypto.git", from: "3.0.0") ], targets: [ .target( @@ -152,7 +153,8 @@ let package = Package( name: "skitrun", dependencies: [ .product(name: "SwiftSyntax", package: "swift-syntax"), - .product(name: "SwiftParser", package: "swift-syntax") + .product(name: "SwiftParser", package: "swift-syntax"), + .product(name: "Crypto", package: "swift-crypto") ], swiftSettings: swiftSettings ), diff --git a/Sources/skitrun/Helpers.swift b/Sources/skitrun/Helpers.swift index d8c3d7f..6a87f88 100644 --- a/Sources/skitrun/Helpers.swift +++ b/Sources/skitrun/Helpers.swift @@ -27,13 +27,22 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import CryptoKit +import Crypto import Foundation /// Hardcoded module name for the user's `Helpers/` compilation output. Inputs /// reach the compiled helpers via `import SyntaxKitHelpers`. internal let helpersModuleName = "SyntaxKitHelpers" +/// Platform-specific shared-library filename for a Swift library product. +internal func dylibFilename(forLibrary name: String) -> String { + #if os(Linux) + return "lib\(name).so" + #else + return "lib\(name).dylib" + #endif +} + /// Bumped when the cache layout changes in a way that requires invalidation. private let helpersCacheSchemaVersion = "v1" @@ -109,7 +118,8 @@ internal func buildHelpers( let cacheRoot = try syntaxKitCacheRoot() .appendingPathComponent("helpers") .appendingPathComponent(key) - let dylibPath = cacheRoot.appendingPathComponent("lib\(helpersModuleName).dylib").path + let dylibPath = cacheRoot + .appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path let fm = FileManager.default if fm.fileExists(atPath: dylibPath) { @@ -148,7 +158,7 @@ internal func buildHelpers( private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) throws { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - let dylib = outDir.appendingPathComponent("lib\(helpersModuleName).dylib").path + let dylib = outDir.appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path let modulePath = outDir.appendingPathComponent("\(helpersModuleName).swiftmodule").path let process = Process() @@ -166,20 +176,37 @@ private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) t "-L", libPath, "-lSyntaxKit", "-Xcc", "-I", "-Xcc", cShimsInclude, - "-Xlinker", "-install_name", - "-Xlinker", "@rpath/lib\(helpersModuleName).dylib", "-Xlinker", "-rpath", "-Xlinker", libPath, ] + #if !os(Linux) + // @rpath install_name is macOS-only; on Linux SONAME isn't needed because + // we use rpath-based loading and the dylib lives in a cache path that's + // known at link time. + args.append(contentsOf: [ + "-Xlinker", "-install_name", + "-Xlinker", "@rpath/\(dylibFilename(forLibrary: helpersModuleName))", + ]) + #endif args.append(contentsOf: sources.map(\.path)) process.arguments = args let stderrPipe = Pipe() process.standardOutput = FileHandle.nullDevice process.standardError = stderrPipe + + // Linux Foundation's `Process.waitUntilExit()` blocks indefinitely on + // already-exited children in some configurations; terminationHandler + + // semaphore is the workaround used elsewhere in this file. + let semaphore = DispatchSemaphore(value: 0) + process.terminationHandler = { _ in semaphore.signal() } + try process.run() - process.waitUntilExit() + // Drain stderr BEFORE waiting on the semaphore — Linux pipe buffers are + // ~64 KB; if the child fills them we deadlock waiting for an exit that + // can't happen until we read. let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() + semaphore.wait() guard process.terminationStatus == 0 else { let stderr = String(decoding: stderrData, as: UTF8.self) throw CLIError( @@ -219,14 +246,16 @@ internal func captureSwiftVersion() -> String? { let pipe = Pipe() process.standardOutput = pipe process.standardError = FileHandle.nullDevice + let semaphore = DispatchSemaphore(value: 0) + process.terminationHandler = { _ in semaphore.signal() } do { try process.run() } catch { return nil } - process.waitUntilExit() let data = pipe.fileHandleForReading.readDataToEndOfFile() + semaphore.wait() return String(decoding: data, as: UTF8.self) } internal func libStamp(libPath: String) -> String? { - let dylib = "\(libPath)/libSyntaxKit.dylib" + let dylib = "\(libPath)/\(dylibFilename(forLibrary: "SyntaxKit"))" guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index 9d8baf0..f4c1dc0 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -159,7 +159,7 @@ private func isLibDir(_ path: String) -> Bool { let fm = FileManager.default var isDir: ObjCBool = false guard fm.fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { return false } - return fm.fileExists(atPath: "\(path)/libSyntaxKit.dylib") + return fm.fileExists(atPath: "\(path)/\(dylibFilename(forLibrary: "SyntaxKit"))") } // MARK: - Single-file mode @@ -614,15 +614,41 @@ private func runSwift( process.standardOutput = stdoutPipe process.standardError = stderrPipe + // Linux Foundation's `Process.waitUntilExit()` blocks indefinitely on + // already-exited children in some configurations; terminationHandler + + // semaphore is the workaround. + let exitSemaphore = DispatchSemaphore(value: 0) + process.terminationHandler = { _ in exitSemaphore.signal() } + try process.run() - process.waitUntilExit() - let stdoutData = stdoutPipe.fileHandleForReading.readDataToEndOfFile() - let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() + // Drain both pipes concurrently — reading sequentially deadlocks on Linux + // when either pipe (~64 KB buffer) fills before the child exits. Box the + // buffers in classes so Swift 6 strict-concurrency is satisfied without + // `@unchecked Sendable` on local vars. + let outBox = PipeDataBox() + let errBox = PipeDataBox() + let group = DispatchGroup() + group.enter() + DispatchQueue.global().async { + outBox.value = stdoutPipe.fileHandleForReading.readDataToEndOfFile() + group.leave() + } + group.enter() + DispatchQueue.global().async { + errBox.value = stderrPipe.fileHandleForReading.readDataToEndOfFile() + group.leave() + } + group.wait() + exitSemaphore.wait() return ProcessResult( exitCode: process.terminationStatus, - stdout: stdoutData, - stderr: String(decoding: stderrData, as: UTF8.self) + stdout: outBox.value, + stderr: String(decoding: errBox.value, as: UTF8.self) ) } + +private final class PipeDataBox: @unchecked Sendable { + var value = Data() +} diff --git a/Sources/skitrun/OutputCache.swift b/Sources/skitrun/OutputCache.swift index b951bff..3f094b0 100644 --- a/Sources/skitrun/OutputCache.swift +++ b/Sources/skitrun/OutputCache.swift @@ -27,7 +27,7 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import CryptoKit +import Crypto import Foundation /// Bumped when the output cache layout changes in a way that requires invalidation. From 2836cf75eb41237afc5f781f39d3ae78b6c558d1 Mon Sep 17 00:00:00 2001 From: leogdion Date: Tue, 12 May 2026 12:46:03 -0400 Subject: [PATCH 12/28] Add skitrun README scoped to the target dir (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lives at Sources/skitrun/README.md (per user's "may be temp" note — keep it local to the target, not the repo front door). Covers usage, the portable bundle, input shape, helpers, caches, the flag table, platform notes (including the Linux waitUntilExit gotcha), and the open scope decisions from codegen-cli-design.md §7. Cross-links into Docs/research/ so the design + per-step results are one click away. Co-Authored-By: Claude Opus 4.7 (1M context) --- Sources/skitrun/README.md | 103 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 Sources/skitrun/README.md diff --git a/Sources/skitrun/README.md b/Sources/skitrun/README.md new file mode 100644 index 0000000..c8c1a20 --- /dev/null +++ b/Sources/skitrun/README.md @@ -0,0 +1,103 @@ +# skitrun + +> **Status:** research POC for [issue #154](https://github.com/brightdigit/SyntaxKit/issues/154). Shape may change; do not pin tooling to it yet. Design lives at [`Docs/research/codegen-cli-design.md`](../../Docs/research/codegen-cli-design.md); the 7-step build-up is documented step-by-step under [`Docs/research/poc-step{1..7}-results.md`](../../Docs/research/). + +A CLI that takes a *pure SyntaxKit DSL* input file, wraps it in a `Group { … }` closure, spawns `swift` to evaluate it, and writes the rendered Swift source to stdout (or a file). No `print`, no `@main`, no boilerplate in your input — just DSL expressions. + +``` +skitrun Input.swift # render to stdout +skitrun Input.swift -o Out.swift # render to a file +skitrun InputDir/ -o OutDir/ # walk **/*.swift, mirror to OutDir/ +``` + +## Quick start + +```bash +# Build a portable bundle (the script flips the SyntaxKit library to +# .dynamic, then bundles dylib + modules + C-shims headers next to skitrun). +Docs/research/poc-step4-release.sh +# → .build/skitrun-release/{skitrun, lib/} + +cat > /tmp/Person.swift <<'SWIFT' +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} +SWIFT + +.build/skitrun-release/skitrun /tmp/Person.swift +``` + +The bundle is self-contained: `cp -r .build/skitrun-release ~/anywhere/` and `~/anywhere/skitrun-release/skitrun ` works zero-config. + +## Input file shape + +Top-level expressions form an implicit `@CodeBlockBuilder` body. `import` declarations at the top are hoisted into the wrapper. Anything else (`Struct(…)`, `Enum(…)`, helper calls, …) becomes the builder's content. + +```swift +// Models.swift +import SyntaxKitHelpers // optional — only if a Helpers/ dir is present + +equatableModel("Person", fields: [ + ("name", "String"), + ("age", "Int"), +]) +``` + +What *won't* work inside the input: top-level `let`/`var` outside the builder DSL, `@main`, `print` (the wrapper adds its own). Result-builder closure rules apply. + +## Helpers + +Shared codegen utilities live in a `Helpers/` directory anywhere up-tree from the input. `skitrun` walks up from the input file (or directory) looking for one. Sources are pre-compiled into `libSyntaxKitHelpers.{dylib,so}` once and cached by content hash: + +``` +project/ +├── Helpers/ +│ └── Models.swift # public func equatableModel(_:fields:) -> any CodeBlock +└── inputs/ + ├── Person.swift # imports SyntaxKitHelpers, calls equatableModel(...) + └── Pet.swift # same +``` + +Files prefixed with `_` are skipped (convention for private helpers within helpers). The helper module name is hard-coded to `SyntaxKitHelpers`. + +Force-disable: `--no-helpers`. Override location: `--helpers `. + +## Caches + +Two layers, both keyed on content + toolchain + dylib stamp + `SKITRUN_*`/`SYNTAXKIT_*` env vars. Live under `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS, `$XDG_CACHE_HOME/syntaxkit` (or `~/.cache/syntaxkit`) on Linux. + +| Layer | Path | What it skips on hit | +| --- | --- | --- | +| Helpers | `helpers//` | the `swiftc` compile of `Helpers/*.swift` | +| Output | `outputs//output.swift` | the `swift` spawn for an input | + +Output cache hit ≈ 0.14s on macOS (no spawn at all); cold miss matches the warm `swift` script-mode baseline (~0.5s). Force a miss with `--no-cache`. + +## Flag reference + +| Flag | Default | Meaning | +| --- | --- | --- | +| `-o, --output ` | stdout | Output file (single-file mode) or directory (folder mode). | +| `--lib ` | auto | Directory containing `libSyntaxKit.{dylib,so}` + module files. Search order when omitted: `$SKITRUN_LIB_DIR` → `/lib/` → `/../lib/skitrun/`. | +| `--helpers ` | walk-up | Explicit `Helpers/` directory. | +| `--no-helpers` | (off) | Skip helpers discovery entirely. | +| `--no-cache` | (off) | Skip the output cache; always spawn `swift`. | + +## Platform notes + +- **macOS:** primary target. All seven POC steps run via the scripts in `Docs/research/`. +- **Linux:** verified in `swift:6.0-jammy/aarch64` via [`Docs/research/poc-step7.sh`](../../Docs/research/poc-step7.sh) (self-reruns inside Docker). Requires `swift-crypto` instead of CryptoKit; install-name flag is Mach-O specific and skipped on Linux. +- **Windows:** not attempted. + +A known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited children on `swift:6.0-jammy/aarch64`. Workaround in `Helpers.swift` / `Main.swift`: `terminationHandler` + `DispatchSemaphore`. See [`poc-step7-results.md`](../../Docs/research/poc-step7-results.md) for the full reproducer. + +## Open scope decisions + +Not blocking but on the table — see [`codegen-cli-design.md` §7](../../Docs/research/codegen-cli-design.md#7-what-we-still-need-to-verify): + +- Timeouts on the child `swift` process (60s default + SIGTERM/SIGKILL grace). +- `@main` / attribute behavior in `swift` script-mode beyond the simple cases tested. +- Multi-file outputs from a single input (out of scope for v1). +- Sandboxing (out of scope; threat model = "you ran your own code"). +- HTTP/server form for warm-interpreter reuse (post-CLI follow-up). From c6d9e7dd500308f00f1382700e398b4510897091 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 13:31:46 -0400 Subject: [PATCH 13/28] Fix skitrun release bundle and update blackjack example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - poc-step4-release.sh: explicitly build the SyntaxKit product. skitrun doesn't depend on SyntaxKit (it spawns swift on user input that does), so --product skitrun alone never produced libSyntaxKit.dylib and the bundle ended up with whatever swiftmodule happened to be cached — which surfaced as a 6.3 vs 6.3.2 module-version mismatch after a toolchain bump. - Examples/Completed/blackjack/dsl.swift: update to the current DSL API (ComputedProperty now requires type:, Init's builder takes ParameterExp, VariableDecl → Variable), escape \(…) interpolations so the literal appears in the generated code, and drop the let/print wrapper so the file is a top-level CodeBlock expression compatible with skitrun. - .vscode/launch.json: add Debug/Release launch configs for skitrun. Co-Authored-By: Claude Opus 4.7 (1M context) --- .vscode/launch.json | 20 ++++++++++++++++ Docs/research/poc-step4-release.sh | 6 +++++ Examples/Completed/blackjack/dsl.swift | 33 ++++++++++++-------------- 3 files changed, 41 insertions(+), 18 deletions(-) diff --git a/.vscode/launch.json b/.vscode/launch.json index fefdbc0..1c2682d 100644 --- a/.vscode/launch.json +++ b/.vscode/launch.json @@ -19,6 +19,26 @@ "preLaunchTask": "swift: Build Release skit", "target": "skit", "configuration": "release" + }, + { + "type": "swift", + "request": "launch", + "args": [], + "cwd": "${workspaceFolder:SyntaxKit}", + "name": "Debug skitrun", + "target": "skitrun", + "configuration": "debug", + "preLaunchTask": "swift: Build Debug skitrun" + }, + { + "type": "swift", + "request": "launch", + "args": [], + "cwd": "${workspaceFolder:SyntaxKit}", + "name": "Release skitrun", + "target": "skitrun", + "configuration": "release", + "preLaunchTask": "swift: Build Release skitrun" } ] } diff --git a/Docs/research/poc-step4-release.sh b/Docs/research/poc-step4-release.sh index 36b2035..2ff3e9c 100755 --- a/Docs/research/poc-step4-release.sh +++ b/Docs/research/poc-step4-release.sh @@ -53,6 +53,12 @@ cd "$REPO_ROOT" echo "==> swift build -c release --product skitrun" swift build -c release --product skitrun +# skitrun doesn't depend on SyntaxKit (it spawns swift on user input that +# imports SyntaxKit at runtime). Build the library product explicitly so the +# .dynamic flip above produces libSyntaxKit.dylib + swiftmodule. +echo "==> swift build -c release --product SyntaxKit" +swift build -c release --product SyntaxKit + BUILD_DIR="$(ls -d .build/*-apple-macosx*/release 2>/dev/null | head -1)" if [[ -z "$BUILD_DIR" ]]; then echo "Could not locate release build dir under .build//release" >&2 diff --git a/Examples/Completed/blackjack/dsl.swift b/Examples/Completed/blackjack/dsl.swift index d0af68d..f894b51 100644 --- a/Examples/Completed/blackjack/dsl.swift +++ b/Examples/Completed/blackjack/dsl.swift @@ -1,7 +1,7 @@ import SyntaxKit // Example of generating a BlackjackCard struct with a nested Suit enum -let structExample = Struct("BlackjackCard") { +Struct("BlackjackCard") { Enum("Suit") { EnumCase("spades").equals("♠") EnumCase("hearts").equals("♡") @@ -28,29 +28,29 @@ let structExample = Struct("BlackjackCard") { Variable(.let, name: "first", type: "Int") Variable(.let, name: "second", type: "Int?") } - ComputedProperty("values") { + ComputedProperty("values", type: "Values") { Switch("self") { SwitchCase(".ace") { - Return{ + Return { Init("Values") { - Parameter(name: "first", value: "1") - Parameter(name: "second", value: "11") + ParameterExp(name: "first", value: "1") + ParameterExp(name: "second", value: "11") } } } SwitchCase(".jack", ".queen", ".king") { - Return{ + Return { Init("Values") { - Parameter(name: "first", value: "10") - Parameter(name: "second", value: "nil") + ParameterExp(name: "first", value: "10") + ParameterExp(name: "second", value: "nil") } } } Default { - Return{ + Return { Init("Values") { - Parameter(name: "first", value: "self.rawValue") - Parameter(name: "second", value: "nil") + ParameterExp(name: "first", value: "self.rawValue") + ParameterExp(name: "second", value: "nil") } } } @@ -62,17 +62,14 @@ let structExample = Struct("BlackjackCard") { Variable(.let, name: "rank", type: "Rank") Variable(.let, name: "suit", type: "Suit") - ComputedProperty("description") { - VariableDecl(.var, name: "output", equals: "suit is \(suit.rawValue),") - PlusAssign("output", " value is \(rank.values.first)") + ComputedProperty("description", type: "String") { + Variable(.var, name: "output", equals: "suit is \\(suit.rawValue),") + PlusAssign("output", " value is \\(rank.values.first)") If(Let("second", "rank.values.second"), then: { - PlusAssign("output", " or \(second)") + PlusAssign("output", " or \\(second)") }) Return { VariableExp("output") } } } - -// Generate and print the code -print(structExample.generateCode()) From 84b497bf8576620e04e99c29e1ce6a18bbc4cfc4 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 13:43:54 -0400 Subject: [PATCH 14/28] Update Examples/Completed/*/dsl.swift to current API + skitrun shape MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brings 6 more examples up to a state where skitrun renders them end-to-end (joining blackjack from the previous commit). 7/11 examples now work; the remaining 4 (concurrency, errors_async, macro_tutorial, enum_generator) need deeper rewrites and are left for follow-up. Common patterns applied: - Drop `let = …; print(.generateCode())` wrappers — skitrun wraps the input in `Group { … }` itself and rejects top-level let/print. - `ComputedProperty(_:)` → `ComputedProperty(_:type:)` (type is now required). - `Parameter(name:value:)` inside `Init { … }` → `ParameterExp(name:value:)`. - `Call("fn", "literal")` → `Call("fn") { ParameterExp(unlabeled: Literal.string("literal")) }`. - `Infix("a", "op", v)` → `Infix("op", lhs: VariableExp("a"), rhs: )`. - `Function("f", parameters: […])` → trailing parameter-builder closure form. - `.access("public")` → `.access(.public)` (now takes AccessModifier enum). - `Literal("\"…\"")` → `Literal.string(…)`. - `VariableDecl(.let, …)` → `Variable(.let, …)` (rename). - `Variable(…, defaultValue:)` → `Variable(…, equals:)`. - `.let("x")` pattern shorthand → `Pattern.let("x")`. - `.reference("weak")` → `.reference(.weak)` (now takes CaptureReferenceType). - `.property(name:)` → `.property(_:)` (label removed). - Escape `\(…)` interpolations so the literal appears in the *generated* code rather than being evaluated against non-existent DSL-scope names. Co-Authored-By: Claude Opus 4.7 (1M context) --- Examples/Completed/attributes/dsl.swift | 9 +- Examples/Completed/card_game/dsl.swift | 26 +++-- Examples/Completed/conditionals/dsl.swift | 126 +++++++++++----------- Examples/Completed/for_loops/dsl.swift | 16 +-- Examples/Completed/protocols/dsl.swift | 12 ++- Examples/Completed/swiftui/dsl.swift | 14 +-- 6 files changed, 99 insertions(+), 104 deletions(-) diff --git a/Examples/Completed/attributes/dsl.swift b/Examples/Completed/attributes/dsl.swift index daf5416..e7d453a 100644 --- a/Examples/Completed/attributes/dsl.swift +++ b/Examples/Completed/attributes/dsl.swift @@ -1,7 +1,10 @@ Class("Foo") { - Variable(.var, name: "bar", type: "String", defaultValue: "bar").attribute("Published") + Variable(.var, name: "bar", type: "String", equals: "bar").attribute("Published") Function("bar") { - print("bar") + Call("print") { + ParameterExp(unlabeled: Literal.string("bar")) + } }.attribute("available", arguments: ["iOS 17.0", "*"]) Function("baz") { -}.attribute("objc")}.attribute("objc") \ No newline at end of file + }.attribute("objc") +} diff --git a/Examples/Completed/card_game/dsl.swift b/Examples/Completed/card_game/dsl.swift index f6f9bdd..0374e51 100644 --- a/Examples/Completed/card_game/dsl.swift +++ b/Examples/Completed/card_game/dsl.swift @@ -1,7 +1,7 @@ import SyntaxKit // Example of generating a BlackjackCard struct with a nested Suit enum -let structExample = Group { +Group { Struct("Card") { Variable(.let, name: "rank", type: "Rank") .comment{ @@ -39,31 +39,31 @@ let structExample = Group { Variable(.let, name: "first", type: "Int") Variable(.let, name: "second", type: "Int?") } - ComputedProperty("description") { + ComputedProperty("description", type: "String") { Switch("self") { SwitchCase(".jack") { - Return{ - Literal("\"J\"") + Return { + Literal.string("J") } } SwitchCase(".queen") { - Return{ - Literal("\"Q\"") + Return { + Literal.string("Q") } } SwitchCase(".king") { - Return{ - Literal("\"K\"") + Return { + Literal.string("K") } } SwitchCase(".ace") { - Return{ - Literal("\"A\"") + Return { + Literal.string("A") } } Default { - Return{ - Literal("\\(rawValue)") + Return { + Literal.string("\\(rawValue)") } } } @@ -92,5 +92,3 @@ let structExample = Group { } } -// Generate and print the code -print(structExample.generateCode()) diff --git a/Examples/Completed/conditionals/dsl.swift b/Examples/Completed/conditionals/dsl.swift index 70263e0..ef05702 100644 --- a/Examples/Completed/conditionals/dsl.swift +++ b/Examples/Completed/conditionals/dsl.swift @@ -4,31 +4,31 @@ Group { Line("Simple if statement") } If { - Infix("temperature", ">", 30) + Infix(">", lhs: VariableExp("temperature"), rhs: Literal.integer(30)) } then: { - Call("print", "It's hot outside!") + Call("print") { ParameterExp(unlabeled: Literal.string("It's hot outside!")) } } Variable(.let, name: "score", equals: Literal.integer(85)) .comment { Line("If-else statement") } If { - Infix("score", ">=", 90) + Infix(">=", lhs: VariableExp("score"), rhs: Literal.integer(90)) } then: { - Call("print", "Excellent!") + Call("print") { ParameterExp(unlabeled: Literal.string("Excellent!")) } } else: { If { - Infix("score", ">=", 80) + Infix(">=", lhs: VariableExp("score"), rhs: Literal.integer(80)) } then: { - Call("print", "Good job!") + Call("print") { ParameterExp(unlabeled: Literal.string("Good job!")) } } If { - try Infix("score", ">=", 70) + Infix(">=", lhs: VariableExp("score"), rhs: Literal.integer(70)) } then: { - Call("print", "Passing") + Call("print") { ParameterExp(unlabeled: Literal.string("Passing")) } } Then { - Call("print", "Needs improvement") + Call("print") { ParameterExp(unlabeled: Literal.string("Needs improvement")) } } } @@ -40,9 +40,9 @@ Group { If(Let("actualNumber", Init("Int") { ParameterExp(name: "", value: "possibleNumber") }), then: { - Call("print", "The string \"\\(possibleNumber)\" has an integer value of \\(actualNumber)") + Call("print") { ParameterExp(unlabeled: Literal.string("The string \"\\(possibleNumber)\" has an integer value of \\(actualNumber)")) } }, else: { - Call("print", "The string \"\\(possibleNumber)\" could not be converted to an integer") + Call("print") { ParameterExp(unlabeled: Literal.string("The string \"\\(possibleNumber)\" could not be converted to an integer")) } }) Variable(.let, name: "possibleName", type: "String?", equals: Literal.string("John")).withExplicitType() @@ -54,14 +54,16 @@ Group { Let("name", "possibleName") Let("age", "possibleAge") } then: { - Call("print", "\\(name) is \\(age) years old") + Call("print") { ParameterExp(unlabeled: Literal.string("\\(name) is \\(age) years old")) } } - Function("greet", parameters: [Parameter("person", type: "[String: String]")]) { + Function("greet") { + Parameter(name: "person", type: "[String: String]") + } _: { Guard { Let("name", "person[\"name\"]") } else: { - Call("print", "No name provided") + Call("print") { ParameterExp(unlabeled: Literal.string("No name provided")) } } Guard { Let("age", "person[\"age\"]") @@ -69,9 +71,9 @@ Group { ParameterExp(name: "", value: "age") }) } else: { - Call("print", "Invalid age provided") + Call("print") { ParameterExp(unlabeled: Literal.string("Invalid age provided")) } } - Call("print", "Hello \\(name), you are \\(ageInt) years old") + Call("print") { ParameterExp(unlabeled: Literal.string("Hello \\(name), you are \\(ageInt) years old")) } } }.comment { Line("MARK: - Guard Statements") @@ -104,26 +106,26 @@ Switch("approximateCount") { Assignment("naturalCount", Literal.string("many")) } } -Call("print", "There are \\(naturalCount) \\(countedThings).") +Call("print") { ParameterExp(unlabeled: Literal.string("There are \\(naturalCount) \\(countedThings).")) } Variable(.let, name: "somePoint", type: "(Int, Int)", equals: VariableExp("(1, 1)"), explicitType: true) .comment { Line("Switch with tuple matching") } Switch("somePoint") { SwitchCase(Tuple.pattern([0, 0])) { - Call("print", "(0, 0) is at the origin") + Call("print") { ParameterExp(unlabeled: Literal.string("(0, 0) is at the origin")) } } SwitchCase(Tuple.pattern([nil, 0])) { - Call("print", "(\(somePoint.0), 0) is on the x-axis") + Call("print") { ParameterExp(unlabeled: Literal.string("(\\(somePoint.0), 0) is on the x-axis")) } } SwitchCase(Tuple.pattern([0, nil])) { - Call("print", "(0, \(somePoint.1)) is on the y-axis") + Call("print") { ParameterExp(unlabeled: Literal.string("(0, \\(somePoint.1)) is on the y-axis")) } } SwitchCase(Tuple.pattern([(-2...2), (-2...2)])) { - Call("print", "(\(somePoint.0), \(somePoint.1)) is inside the box") + Call("print") { ParameterExp(unlabeled: Literal.string("(\\(somePoint.0), \\(somePoint.1)) is inside the box")) } } Default { - Call("print", "(\(somePoint.0), \(somePoint.1)) is outside of the box") + Call("print") { ParameterExp(unlabeled: Literal.string("(\\(somePoint.0), \\(somePoint.1)) is outside of the box")) } } } Variable(.let, name: "anotherPoint", type: "(Int, Int)", equals: VariableExp("(2, 0)"), explicitType: true) @@ -131,21 +133,21 @@ Variable(.let, name: "anotherPoint", type: "(Int, Int)", equals: VariableExp("(2 Line("Switch with value binding") } Switch("anotherPoint") { - SwitchCase(Tuple.pattern([.let("x"), 0])) { - Call("print", "on the x-axis with an x value of \(x)") + SwitchCase(Tuple.pattern([Pattern.let("x"), 0])) { + Call("print") { ParameterExp(unlabeled: Literal.string("on the x-axis with an x value of \\(x)")) } } - SwitchCase(Tuple.pattern([0, .let("y")])) { - Call("print", "on the y-axis with a y value of \(y)") + SwitchCase(Tuple.pattern([0, Pattern.let("y")])) { + Call("print") { ParameterExp(unlabeled: Literal.string("on the y-axis with a y value of \\(y)")) } } - SwitchCase(Tuple.pattern([.let("x"), .let("y")])) { - Call("print", "somewhere else at (\(x), \(y))") + SwitchCase(Tuple.pattern([Pattern.let("x"), Pattern.let("y")])) { + Call("print") { ParameterExp(unlabeled: Literal.string("somewhere else at (\\(x), \\(y))")) } } } Variable(.let, name: "integerToDescribe", equals: 5) -Variable(.var, name: "description", equals: "The number \(integerToDescribe) is") +Variable(.var, name: "description", equals: "The number \\(integerToDescribe) is") Switch("integerToDescribe") { SwitchCase(2, 3, 5, 7, 11, 13, 17, 19) { PlusAssign("description", "a prime number, and also") @@ -155,68 +157,62 @@ Switch("integerToDescribe") { PlusAssign("description", "an integer.") } } -Call("print", "description") +Call("print") { ParameterExp(unlabeled: Literal.string("description")) } Variable(.let, name: "finalSquare", equals: 25) Variable(.var, name: "board", equals: Literal.array(Array(repeating: Literal.integer(0), count: 26))) -Infix("board[03]", "+=", 8) -Infix("board[06]", "+=", 11) -Infix("board[09]", "+=", 9) -Infix("board[10]", "+=", 2) -Infix("board[14]", "-=", 10) -Infix("board[19]", "-=", 11) -Infix("board[22]", "-=", 2) -Infix("board[24]", "-=", 8) +Infix("+=", lhs: VariableExp("board[03]"), rhs: Literal.integer(8)) +Infix("+=", lhs: VariableExp("board[06]"), rhs: Literal.integer(11)) +Infix("+=", lhs: VariableExp("board[09]"), rhs: Literal.integer(9)) +Infix("+=", lhs: VariableExp("board[10]"), rhs: Literal.integer(2)) +Infix("-=", lhs: VariableExp("board[14]"), rhs: Literal.integer(10)) +Infix("-=", lhs: VariableExp("board[19]"), rhs: Literal.integer(11)) +Infix("-=", lhs: VariableExp("board[22]"), rhs: Literal.integer(2)) +Infix("-=", lhs: VariableExp("board[24]"), rhs: Literal.integer(8)) Variable(.var, name: "square", equals: 0) Variable(.var, name: "diceRoll", equals: 0) -While { - try Infix("square", "!=", "finalSquare") -} then: { - Assignment("diceRoll", "+", 1) +While(Infix("!=", lhs: VariableExp("square"), rhs: VariableExp("finalSquare"))) { + Infix("+=", lhs: VariableExp("diceRoll"), rhs: Literal.integer(1)) If { - try Infix("diceRoll", "==", 7) + Infix("==", lhs: VariableExp("diceRoll"), rhs: Literal.integer(7)) } then: { Assignment("diceRoll", 1) } - Switch(try Infix("square", "+", "diceRoll")) { + Switch(Infix("+", lhs: VariableExp("square"), rhs: VariableExp("diceRoll"))) { SwitchCase("finalSquare") { Break() } - SwitchCase(try Infix("newSquare", ">", "finalSquare")) { + SwitchCase(Infix(">", lhs: VariableExp("newSquare"), rhs: VariableExp("finalSquare"))) { Continue() } Default { - try Infix("square", "+=", "diceRoll") - try Infix("square", "+=", "board[square]") + Infix("+=", lhs: VariableExp("square"), rhs: VariableExp("diceRoll")) + Infix("+=", lhs: VariableExp("square"), rhs: VariableExp("board[square]")) } } } -Call("print", "\n=== For-in with Enumerated ===") +Call("print") { ParameterExp(unlabeled: Literal.string("\n=== For-in with Enumerated ===")) } .comment { Line("MARK: - For Loops") Line("For-in loop with enumerated() to get index and value") } -For { - Tuple.pattern([VariableExp("index"), VariableExp("name")]) -} in: { - VariableExp("names").call("enumerated") -} then: { - Call("print", "Index: \\(index), Name: \\(name)") -} +For(Tuple.patternCodeBlock([VariableExp("index"), VariableExp("name")]), + in: VariableExp("names").call("enumerated"), + then: { + Call("print") { ParameterExp(unlabeled: Literal.string("Index: \\(index), Name: \\(name)")) } + }) -Call("print", "\n=== For-in with Where Clause ===") +Call("print") { ParameterExp(unlabeled: Literal.string("\n=== For-in with Where Clause ===")) } .comment { Line("For-in loop with where clause") } -For { - VariableExp("numbers") -} in: { - Literal.array([Literal.integer(1), Literal.integer(2), Literal.integer(3), Literal.integer(4), Literal.integer(5), Literal.integer(6), Literal.integer(7), Literal.integer(8), Literal.integer(9), Literal.integer(10)]) -} where: { - try Infix("number", "%", 2) -} then: { - Call("print", "Even number: \\(number)") -} +For(VariableExp("number"), + in: VariableExp("numbers"), + then: { + If(VariableExp("number % 2 == 0"), then: { + Call("print") { ParameterExp(unlabeled: Literal.string("Even number: \\(number)")) } + }) + }) diff --git a/Examples/Completed/for_loops/dsl.swift b/Examples/Completed/for_loops/dsl.swift index db06009..753e78a 100644 --- a/Examples/Completed/for_loops/dsl.swift +++ b/Examples/Completed/for_loops/dsl.swift @@ -39,18 +39,12 @@ Group { } Variable(.let, name: "numbers", equals: Literal.array([Literal.integer(1), Literal.integer(2), Literal.integer(3), Literal.integer(4), Literal.integer(5), Literal.integer(6), Literal.integer(7), Literal.integer(8), Literal.integer(9), Literal.integer(10)])) - For(VariableExp("number"), in: VariableExp("numbers"), where: { - try Infix("==") { - try Infix("%") { - VariableExp("number") - Literal.integer(2) + For(VariableExp("number"), in: VariableExp("numbers"), then: { + If(VariableExp("number % 2 == 0"), then: { + Call("print") { + ParameterExp(unlabeled: "\"Even number: \\(number)\"") } - Literal.integer(0) - } - }, then: { - Call("print") { - ParameterExp(unlabeled: "\"Even number: \\(number)\"") - } + }) }) // MARK: - For-in with Dictionary diff --git a/Examples/Completed/protocols/dsl.swift b/Examples/Completed/protocols/dsl.swift index bd1794f..d44aec9 100644 --- a/Examples/Completed/protocols/dsl.swift +++ b/Examples/Completed/protocols/dsl.swift @@ -1,7 +1,7 @@ import SyntaxKit // Generate and print the code -let generatedCode = Group { +Group { // MARK: - Protocol Definition Protocol("Vehicle") { PropertyRequirement("numberOfWheels", type: "Int", access: .get) @@ -57,8 +57,13 @@ let generatedCode = Group { }.inherits("Vehicle") // MARK: - Usage Example - VariableDecl(.let, name: "tesla", equals: "ElectricCar(brand: \"Tesla\", batteryLevel: 75.0)") - VariableDecl(.let, name: "toyota", equals: "Car(brand: \"Toyota\")") + Variable(.let, name: "tesla", equals: Init("ElectricCar") { + ParameterExp(name: "brand", value: Literal.string("Tesla")) + ParameterExp(name: "batteryLevel", value: Literal.float(75.0)) + }) + Variable(.let, name: "toyota", equals: Init("Car") { + ParameterExp(name: "brand", value: Literal.string("Toyota")) + }) // Demonstrate protocol usage Function("demonstrateVehicle") { @@ -103,4 +108,3 @@ let generatedCode = Group { } } -print(generatedCode.generateCode()) \ No newline at end of file diff --git a/Examples/Completed/swiftui/dsl.swift b/Examples/Completed/swiftui/dsl.swift index 4e2f97c..d2822ce 100644 --- a/Examples/Completed/swiftui/dsl.swift +++ b/Examples/Completed/swiftui/dsl.swift @@ -1,7 +1,7 @@ -Import("SwiftUI").access("public") +Import("SwiftUI").access(.public) Struct("TodoItemRow") { - Variable(.let, name: "item", type: "TodoItem").access("private") + Variable(.let, name: "item", type: "TodoItem").access(.private) Variable(.let, name: "onToggle", type: ClosureType(returns: "Void"){ @@ -10,7 +10,7 @@ Struct("TodoItemRow") { .attribute("@MainActor") .attribute("@Sendable") ) - .access("private") + .access(.private) ComputedProperty("body", type: "some View") { Init("HStack") { @@ -21,13 +21,13 @@ Struct("TodoItemRow") { ParameterExp(unlabeled: Closure{ Init("Image") { ParameterExp(name: "systemName", value: ConditionalOp( - if: VariableExp("item").property(name: "isCompleted"), + if: VariableExp("item").property("isCompleted"), then: Literal.string("checkmark.circle.fill"), else: Literal.string("circle") )) }.call("foregroundColor"){ ParameterExp(unlabeled: ConditionalOp( - if: VariableExp("item").property(name: "isCompleted"), + if: VariableExp("item").property("isCompleted"), then: EnumCase("green"), else: EnumCase("gray") )) @@ -39,7 +39,7 @@ Struct("TodoItemRow") { Init("Task") { ParameterExp(unlabeled: Closure( capture: { - ParameterExp(unlabeled: VariableExp("self").reference("weak")) + ParameterExp(unlabeled: VariableExp("self").reference(.weak)) }, body: { VariableExp("self").optional().call("onToggle") { @@ -62,4 +62,4 @@ Struct("TodoItemRow") { } } .inherits("View") -.access("public") +.access(.public) From 207cb7e11101a3e31128805e0feefd2765f7a600 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 14:05:54 -0400 Subject: [PATCH 15/28] Add --timeout watchdog to skitrun MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per codegen-cli-design.md §7. The spawned `swift` process is the only unknown-runtime piece in skitrun (helpers compile and version capture are bounded and cached); a hung user script previously had no upper bound, so the CLI could sit forever. - New flag: `--timeout ` on `runSwift`. Default 60. Pass 0 to disable. - On expiry: `process.terminate()` (SIGTERM), 5s grace, then `kill(pid, SIGKILL)`. Exit code 124 (matches POSIX `timeout(1)`), with a `skitrun: timed out after Xs` prefix on stderr. - Folder mode propagates the per-input timeout to each parallel worker. - Drive-by: bad CLI args now exit cleanly with the usage text instead of the Swift runtime's "Fatal error: ..." trace. Co-Authored-By: Claude Opus 4.7 (1M context) --- Sources/skitrun/Main.swift | 114 ++++++++++++++++++++++++++++++++----- Sources/skitrun/README.md | 1 + 2 files changed, 100 insertions(+), 15 deletions(-) diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index f4c1dc0..9c0a8e9 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -34,7 +34,13 @@ import SwiftSyntax @main internal enum SkitRun { internal static func main() async throws { - let args = try CLIArgs.parse(CommandLine.arguments) + let args: CLIArgs + do { + args = try CLIArgs.parse(CommandLine.arguments) + } catch { + FileHandle.standardError.write(Data("\(error)\n".utf8)) + exit(2) + } let libPath: String do { @@ -56,7 +62,8 @@ internal enum SkitRun { outputPath: output, libPath: libPath, helpers: helpers, - useCache: args.useCache + useCache: args.useCache, + timeoutSeconds: args.timeoutSeconds ) case .directory(let inputDir, let outputDir): let helpers = try resolveHelpers( @@ -69,7 +76,8 @@ internal enum SkitRun { outputDir: outputDir, libPath: libPath, helpers: helpers, - useCache: args.useCache + useCache: args.useCache, + timeoutSeconds: args.timeoutSeconds ) exit(exitCode) } @@ -169,13 +177,15 @@ private func runSingleFile( outputPath: String?, libPath: String, helpers: CompiledHelpers?, - useCache: Bool + useCache: Bool, + timeoutSeconds: Int ) throws { let result = try processFile( inputPath: inputPath, libPath: libPath, helpers: helpers, - useCache: useCache + useCache: useCache, + timeoutSeconds: timeoutSeconds ) if !result.stderr.isEmpty { FileHandle.standardError.write(Data(result.stderr.utf8)) @@ -197,7 +207,8 @@ private func runDirectory( outputDir: String, libPath: String, helpers: CompiledHelpers?, - useCache: Bool + useCache: Bool, + timeoutSeconds: Int ) async -> Int32 { let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL @@ -223,12 +234,22 @@ private func runDirectory( await withTaskGroup(of: FileOutcome.self) { group in for _ in 0.. FileOutcome { do { let result = try processFile( inputPath: input.path, libPath: libPath, helpers: helpers, - useCache: useCache + useCache: useCache, + timeoutSeconds: timeoutSeconds ) return FileOutcome(input: input, result: .success(result)) } catch { @@ -352,7 +375,8 @@ private func processFile( inputPath: String, libPath: String, helpers: CompiledHelpers?, - useCache: Bool + useCache: Bool, + timeoutSeconds: Int ) throws -> ProcessResult { let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL let absoluteInputPath = inputURL.path @@ -376,7 +400,12 @@ private func processFile( let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) - let raw = try runSwift(wrappedPath: wrappedURL.path, libPath: libPath, helpers: helpers) + let raw = try runSwift( + wrappedPath: wrappedURL.path, + libPath: libPath, + helpers: helpers, + timeoutSeconds: timeoutSeconds + ) // #sourceLocation maps body diagnostics back to the input file. Errors in // the preamble (lines outside the body) still reference the wrapper — // rewrite literal occurrences of its path so users see something coherent. @@ -410,6 +439,9 @@ private struct CLIArgs { let libPath: String? let helpers: HelpersOptions let useCache: Bool + let timeoutSeconds: Int + + static let defaultTimeoutSeconds = 60 static func parse(_ argv: [String]) throws -> CLIArgs { var inputPath: String? @@ -417,6 +449,7 @@ private struct CLIArgs { var libPath: String? var helpers: HelpersOptions = .auto var useCache = true + var timeoutSeconds = defaultTimeoutSeconds var i = 1 while i < argv.count { @@ -440,6 +473,13 @@ private struct CLIArgs { case "--no-cache": useCache = false i += 1 + case "--timeout": + guard i + 1 < argv.count else { throw usage("--timeout requires a value") } + guard let parsed = Int(argv[i + 1]), parsed >= 0 else { + throw usage("--timeout expects a non-negative integer (seconds), got: \(argv[i + 1])") + } + timeoutSeconds = parsed + i += 2 case "-h", "--help": FileHandle.standardError.write(Data(helpText.utf8)) exit(0) @@ -469,7 +509,13 @@ private struct CLIArgs { mode = .singleFile(input: inputPath, output: outputPath) } - return CLIArgs(mode: mode, libPath: libPath, helpers: helpers, useCache: useCache) + return CLIArgs( + mode: mode, + libPath: libPath, + helpers: helpers, + useCache: useCache, + timeoutSeconds: timeoutSeconds + ) } } @@ -502,6 +548,10 @@ private let helpText = """ The cache lives at /outputs// and is keyed on input bytes, helpers, swift version, libSyntaxKit stamp, and SKITRUN_*/SYNTAXKIT_* env. + --timeout Per-input timeout for the spawned `swift` process + (default 60). On expiry: SIGTERM, then SIGKILL after + a 5s grace; the file exits with code 124. Pass 0 to + disable the watchdog. """ private func usage(_ message: String) -> CLIError { @@ -576,10 +626,18 @@ internal func wrap(source: String, originalPath: String) -> String { // MARK: - Spawning swift +/// Exit code returned when the spawned `swift` is killed by skitrun's timeout +/// watchdog. Matches POSIX `timeout(1)`. +private let timeoutExitCode: Int32 = 124 + +/// Grace period between SIGTERM and SIGKILL when the child won't exit on its own. +private let killGraceSeconds: Int = 5 + private func runSwift( wrappedPath: String, libPath: String, - helpers: CompiledHelpers? + helpers: CompiledHelpers?, + timeoutSeconds: Int ) throws -> ProcessResult { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" @@ -639,8 +697,34 @@ private func runSwift( errBox.value = stderrPipe.fileHandleForReading.readDataToEndOfFile() group.leave() } + + // Timeout watchdog: wait for the child with a deadline. On expiry, send + // SIGTERM, give a fixed grace, then SIGKILL. timeoutSeconds == 0 disables. + let timedOut: Bool + if timeoutSeconds > 0 { + let deadline: DispatchTime = .now() + .seconds(timeoutSeconds) + if exitSemaphore.wait(timeout: deadline) == .timedOut { + process.terminate() // SIGTERM + if exitSemaphore.wait(timeout: .now() + .seconds(killGraceSeconds)) == .timedOut { + kill(process.processIdentifier, SIGKILL) + exitSemaphore.wait() + } + timedOut = true + } else { + timedOut = false + } + } else { + exitSemaphore.wait() + timedOut = false + } + // Child is dead now — pipes get EOF, drain completes shortly. group.wait() - exitSemaphore.wait() + + if timedOut { + let prefix = Data("skitrun: timed out after \(timeoutSeconds)s\n".utf8) + let stderr = String(decoding: prefix + errBox.value, as: UTF8.self) + return ProcessResult(exitCode: timeoutExitCode, stdout: outBox.value, stderr: stderr) + } return ProcessResult( exitCode: process.terminationStatus, diff --git a/Sources/skitrun/README.md b/Sources/skitrun/README.md index c8c1a20..09f92cf 100644 --- a/Sources/skitrun/README.md +++ b/Sources/skitrun/README.md @@ -83,6 +83,7 @@ Output cache hit ≈ 0.14s on macOS (no spawn at all); cold miss matches the war | `--helpers ` | walk-up | Explicit `Helpers/` directory. | | `--no-helpers` | (off) | Skip helpers discovery entirely. | | `--no-cache` | (off) | Skip the output cache; always spawn `swift`. | +| `--timeout ` | `60` | Per-input timeout for the spawned `swift` (SIGTERM → 5s → SIGKILL). On expiry the file exits with code 124. Pass `0` to disable. | ## Platform notes From a7d8876158b76d40b0efabe8100a671f55f69281 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 14:16:02 -0400 Subject: [PATCH 16/28] Stamp + detect toolchain mismatch at skitrun startup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Swift swiftmodules aren't reliably compatible across compiler versions (observed today: a 6.3-built SyntaxKit.swiftmodule was rejected by a 6.3.2 swift). When the bundled module won't load, the spawned `swift` emits a cryptic "module compiled with Swift X cannot be imported by Y" diagnostic that doesn't point at the actual fix. - poc-step4-release.sh now writes lib/swift-version.txt at bundle time, recording the build toolchain's `swift --version`. - skitrun reads the stamp on startup and compares to a local `swift --version` capture (existing captureSwiftVersion helper). Strict exact-string match — patch-level drift broke today. - On mismatch: exit 2 with a stderr message that names both versions and points at the rebuild script. New --no-toolchain-check flag bypasses the check (debugging / forward-compat experiments). - On missing stamp (older bundles): one-line warn and continue, so existing prebuilt bundles keep working through the transition. Auto-rebuild on mismatch is the natural follow-up; tracked as #157. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step4-release.sh | 6 +++ Sources/skitrun/Main.swift | 83 +++++++++++++++++++++++++++++- Sources/skitrun/README.md | 1 + 3 files changed, 89 insertions(+), 1 deletion(-) diff --git a/Docs/research/poc-step4-release.sh b/Docs/research/poc-step4-release.sh index 2ff3e9c..795d2a2 100755 --- a/Docs/research/poc-step4-release.sh +++ b/Docs/research/poc-step4-release.sh @@ -79,6 +79,12 @@ cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/inclu # Ensure the dylib's install_name uses @rpath so it's portable. install_name_tool -id "@rpath/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/libSyntaxKit.dylib" 2>/dev/null || true +# Stamp the bundle with the build toolchain. skitrun compares this against +# the user's `swift --version` at startup and refuses to spawn `swift` if the +# swiftmodule wouldn't load (see Sources/skitrun/Main.swift). Issue #157 will +# replace the refusal with an auto-rebuild fallback. +swift --version > "$OUTPUT_DIR/lib/swift-version.txt" + BINARY_SIZE=$(ls -lh "$OUTPUT_DIR/skitrun" | awk '{print $5}') DYLIB_SIZE=$(ls -lh "$OUTPUT_DIR/lib/libSyntaxKit.dylib" | awk '{print $5}') TOTAL_SIZE=$(du -sh "$OUTPUT_DIR" | awk '{print $1}') diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index 9c0a8e9..5accbd6 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -50,6 +50,18 @@ internal enum SkitRun { exit(2) } + if args.checkToolchain { + switch toolchainCheck(libPath: libPath) { + case .match, .stampMissing: + break + case .mismatch(let bundle, let local): + FileHandle.standardError.write(Data(toolchainMismatchMessage( + bundle: bundle, local: local + ).utf8)) + exit(2) + } + } + switch args.mode { case .singleFile(let input, let output): let helpers = try resolveHelpers( @@ -170,6 +182,63 @@ private func isLibDir(_ path: String) -> Bool { return fm.fileExists(atPath: "\(path)/\(dylibFilename(forLibrary: "SyntaxKit"))") } +// MARK: - Toolchain check + +/// Filename for the bundle's recorded build-toolchain version. +internal let toolchainStampFilename = "swift-version.txt" + +internal enum ToolchainCheckResult { + /// Bundle stamp matches the local `swift --version` exactly. + case match + /// `/swift-version.txt` is missing (older bundle that predates + /// the stamp). skitrun prints a one-line note and proceeds. + case stampMissing + case mismatch(bundle: String, local: String) +} + +/// Compares `/swift-version.txt` to `captureSwiftVersion()`. +/// The swiftmodule format isn't reliably forward-compatible across even +/// patch-level Swift releases (the originating bug: 6.3.0 → 6.3.2 rejected +/// the swiftmodule), so the comparison is exact-string after normalising +/// trailing whitespace. +internal func toolchainCheck(libPath: String) -> ToolchainCheckResult { + let stampURL = URL(fileURLWithPath: libPath).appendingPathComponent(toolchainStampFilename) + guard let stampData = try? Data(contentsOf: stampURL), + let stampRaw = String(data: stampData, encoding: .utf8) + else { + FileHandle.standardError.write( + Data("skitrun: bundle has no toolchain stamp; skipping check\n".utf8) + ) + return .stampMissing + } + guard let localRaw = captureSwiftVersion() else { + FileHandle.standardError.write( + Data("skitrun: could not capture local `swift --version`; skipping toolchain check\n".utf8) + ) + return .stampMissing + } + let bundle = stampRaw.trimmingCharacters(in: .whitespacesAndNewlines) + let local = localRaw.trimmingCharacters(in: .whitespacesAndNewlines) + return bundle == local ? .match : .mismatch(bundle: bundle, local: local) +} + +internal func toolchainMismatchMessage(bundle: String, local: String) -> String { + """ + skitrun: toolchain mismatch + bundle: \(bundle) + local: \(local) + The bundle's libSyntaxKit was built against a different `swift` than the + one on your PATH. Swift swiftmodules aren't reliably compatible across + versions, so spawning `swift` would fail with a cryptic module-version + diagnostic. + + Rebuild the bundle with: + Docs/research/poc-step4-release.sh + Or pass --no-toolchain-check to try anyway. + + """ +} + // MARK: - Single-file mode private func runSingleFile( @@ -440,6 +509,7 @@ private struct CLIArgs { let helpers: HelpersOptions let useCache: Bool let timeoutSeconds: Int + let checkToolchain: Bool static let defaultTimeoutSeconds = 60 @@ -450,6 +520,7 @@ private struct CLIArgs { var helpers: HelpersOptions = .auto var useCache = true var timeoutSeconds = defaultTimeoutSeconds + var checkToolchain = true var i = 1 while i < argv.count { @@ -473,6 +544,9 @@ private struct CLIArgs { case "--no-cache": useCache = false i += 1 + case "--no-toolchain-check": + checkToolchain = false + i += 1 case "--timeout": guard i + 1 < argv.count else { throw usage("--timeout requires a value") } guard let parsed = Int(argv[i + 1]), parsed >= 0 else { @@ -514,7 +588,8 @@ private struct CLIArgs { libPath: libPath, helpers: helpers, useCache: useCache, - timeoutSeconds: timeoutSeconds + timeoutSeconds: timeoutSeconds, + checkToolchain: checkToolchain ) } } @@ -552,6 +627,12 @@ private let helpText = """ (default 60). On expiry: SIGTERM, then SIGKILL after a 5s grace; the file exits with code 124. Pass 0 to disable the watchdog. + --no-toolchain-check Skip the startup check that compares the bundle's + recorded build toolchain (/swift-version.txt) + against `swift --version`. Swift swiftmodules aren't + reliably compatible across compiler versions, so by + default skitrun refuses to spawn `swift` on + mismatch. See issue #157 for the auto-rebuild plan. """ private func usage(_ message: String) -> CLIError { diff --git a/Sources/skitrun/README.md b/Sources/skitrun/README.md index 09f92cf..52f64fc 100644 --- a/Sources/skitrun/README.md +++ b/Sources/skitrun/README.md @@ -84,6 +84,7 @@ Output cache hit ≈ 0.14s on macOS (no spawn at all); cold miss matches the war | `--no-helpers` | (off) | Skip helpers discovery entirely. | | `--no-cache` | (off) | Skip the output cache; always spawn `swift`. | | `--timeout ` | `60` | Per-input timeout for the spawned `swift` (SIGTERM → 5s → SIGKILL). On expiry the file exits with code 124. Pass `0` to disable. | +| `--no-toolchain-check` | (off) | Skip the startup check that compares the bundle's recorded build toolchain (`lib/swift-version.txt`) to `swift --version`. swiftmodules aren't reliably compatible across compiler versions; on mismatch skitrun refuses to spawn `swift` and points at the rebuild script. Auto-rebuild fallback tracked in [#157](https://github.com/brightdigit/SyntaxKit/issues/157). | ## Platform notes From 47e5be8b635a67d70bc1af5280ed64b2deb06213 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 14:56:13 -0400 Subject: [PATCH 17/28] Update 3 more Examples/Completed/*/dsl.swift to current API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brings concurrency, errors_async, and macro_tutorial up to a state where skitrun renders them end-to-end (joining the 7 already updated). 10/11 examples now work; enum_generator remains broken — it's a fully programmatic Swift program (struct definitions, JSON loading, demo runner) rather than a single DSL expression, so it needs structural restructuring into the Helpers/+dsl pattern. Left for follow-up. Notable per-example fixes: - concurrency: rewritten Guard/Throw/Function shapes; Dictionary values are external `Init` expressions of an `Item` type that Literal.dictionary's typed cases can't represent, so the inventory is emitted via the raw VariableExp escape hatch. - errors_async: dropped the TupleAssignment line — that type became internal in the current API — and substituted two single Variable bindings (same observable behaviour for the catch block). - macro_tutorial: collapsed 11 per-example `let` bindings + a final `print` into a single top-level Group (skitrun's input shape). Some sub-examples used `Init { Parameter(...) }` to *declare* initializers; the current public DSL only has Init-as-expression, so those are emitted as raw Swift via VariableExp. Co-Authored-By: Claude Opus 4.7 (1M context) --- Examples/Completed/concurrency/dsl.swift | 75 ++-- Examples/Completed/errors_async/dsl.swift | 11 +- Examples/Completed/macro_tutorial/dsl.swift | 415 +++++++++----------- 3 files changed, 224 insertions(+), 277 deletions(-) diff --git a/Examples/Completed/concurrency/dsl.swift b/Examples/Completed/concurrency/dsl.swift index 300a460..d0f66dd 100644 --- a/Examples/Completed/concurrency/dsl.swift +++ b/Examples/Completed/concurrency/dsl.swift @@ -1,48 +1,51 @@ Enum("VendingMachineError") { - Case("invalidSelection") - Case("insufficientFunds").associatedValue("coinsNeeded", type: "Int") - Case("outOfStock") + EnumCase("invalidSelection") + EnumCase("insufficientFunds").associatedValue("coinsNeeded", type: "Int") + EnumCase("outOfStock") } +.inherits("Error") Class("VendingMachine") { - Variable(.var, name: "inventory", equals: Literal.dictionary(Dictionary(uniqueKeysWithValues: [ - ("Candy Bar", Item(price: 12, count: 7)), - ("Chips", Item(price: 10, count: 4)), - ("Pretzels", Item(price: 7, count: 11)) - ]))) + // Dictionary values are `Init`-expressions of an external `Item` type, which + // Literal.dictionary's typed cases can't represent — emit the literal as raw + // Swift source via VariableExp. + Variable(.var, name: "inventory") { + VariableExp(""" + [ + "Candy Bar": Item(price: 12, count: 7), + "Chips": Item(price: 10, count: 4), + "Pretzels": Item(price: 7, count: 11) + ] + """) + } Variable(.var, name: "coinsDeposited", equals: 0) - Function("vend"){ + Function("vend") { Parameter("name", labeled: "itemNamed", type: "String") } _: { - Guard("let item = inventory[itemNamed]") else: { - Throw( - EnumValue("VendingMachineError", case: "invalidSelection") - ) + Guard { + Let("item", "inventory[itemNamed]") + } else: { + Throw(VariableExp("VendingMachineError.invalidSelection")) } - Guard("item.count > 0") else: { - Throw( - EnumValue("VendingMachineError", case: "outOfStock") - ) + Guard { + Infix(">", lhs: VariableExp("item.count"), rhs: Literal.integer(0)) + } else: { + Throw(VariableExp("VendingMachineError.outOfStock")) } - Guard("item.price <= coinsDeposited") else: { - Throw( - EnumValue("VendingMachineError", case: "insufficientFunds"){ - ParameterExp("coinsNeeded", value: Infix("-"){ - VariableExp("item").property("price") - VariableExp("coinsDeposited") - }) - } - ) + Guard { + Infix("<=", lhs: VariableExp("item.price"), rhs: VariableExp("coinsDeposited")) + } else: { + Throw(VariableExp( + "VendingMachineError.insufficientFunds(coinsNeeded: item.price - coinsDeposited)" + )) } - Infix("-=", "coinsDeposited", VariableExp("item").property("price")) - Variable("newItem", equals: VariableExp("item")) - Infix("-=", "newItem.count", 1) - Assignment("inventory[itemNamed]", .ref("newItem")) - Call("print", "Dispensing \\(itemNamed)") - } + Infix("-=", lhs: VariableExp("coinsDeposited"), rhs: VariableExp("item.price")) + Variable(.var, name: "newItem") { VariableExp("item") } + Infix("-=", lhs: VariableExp("newItem.count"), rhs: Literal.integer(1)) + Assignment("inventory[itemNamed]", VariableExp("newItem")) + Call("print") { + ParameterExp(unlabeled: Literal.string("Dispensing \\(itemNamed)")) + } + }.throws() } - - - - diff --git a/Examples/Completed/errors_async/dsl.swift b/Examples/Completed/errors_async/dsl.swift index aad5087..6dc71f1 100644 --- a/Examples/Completed/errors_async/dsl.swift +++ b/Examples/Completed/errors_async/dsl.swift @@ -57,10 +57,13 @@ Do { ParameterExp(name: "id", value: Literal.integer(1)) } }.async() - TupleAssignment(["fetchedData", "fetchedPosts"], equals: Tuple { - VariableExp("data") - VariableExp("posts") - }).async().throwing() + // The original example used `TupleAssignment([...], equals: Tuple {...})` + // to emit `let (fetchedData, fetchedPosts) = try await (data, posts)`, but + // `TupleAssignment` is internal in the current API. Emit two equivalent + // single-variable bindings instead — same observable behaviour for the + // catch block below. + Variable(.let, name: "fetchedData") { VariableExp("data") } + Variable(.let, name: "fetchedPosts") { VariableExp("posts") } } catch: { Catch(EnumCase("fetchError")) { // Example catch for async/await diff --git a/Examples/Completed/macro_tutorial/dsl.swift b/Examples/Completed/macro_tutorial/dsl.swift index 490051f..0cea4cf 100644 --- a/Examples/Completed/macro_tutorial/dsl.swift +++ b/Examples/Completed/macro_tutorial/dsl.swift @@ -1,276 +1,217 @@ import SyntaxKit // MARK: - Macro Tutorial DSL Examples -// This file shows how SyntaxKit DSL would be used to generate the macro examples - -// MARK: - Example 1: Extension Macro Generation -// This shows how to generate the extension that @MyMacro would create - -let colorExtension = Extension("Color") { - // Add a type alias - TypeAlias("MyType", equals: "String") - - // Add a static property with case names - Variable(.let, name: "myProperty", equals: ["red", "green", "blue"]).static() - - // Add a computed property - ComputedProperty("description") { - Return { - VariableExp("myProperty.joined(separator: \", \")") +// +// This file demonstrates the SyntaxKit DSL patterns one would use to generate +// the kind of code a Swift macro produces. Each example is a top-level block; +// skitrun concatenates them into a single rendered file. (The original +// version of this file used per-example `let` bindings and a final `print`, +// which skitrun's wrapper doesn't accept — top-level expressions only.) + +Group { + // MARK: Example 1 — Extension Macro Generation + Extension("Color") { + TypeAlias("MyType", equals: "String") + Variable(.let, name: "myProperty", equals: ["red", "green", "blue"]).static() + ComputedProperty("description", type: "String") { + Return { VariableExp("myProperty.joined(separator: \", \")") } + } + }.inherits("MyProtocol") + + // MARK: Example 2 — Peer Macro Generation + // + // The original example declared `init(value: Color) { self.value = value }` + // via `Init { Parameter(...) }`, which isn't part of the current public DSL + // (Init is an expression-only call here). Emit the initializer body as raw + // Swift instead. + Struct("ColorWrapper") { + Variable(.let, name: "value", type: "Color") + VariableExp(""" + init(value: Color) { + self.value = value + } + """) + ComputedProperty("description", type: "String") { + Return { VariableExp("value.description") } } } -}.inherits("MyProtocol") - -// MARK: - Example 2: Peer Macro Generation -// This shows how to generate the wrapper struct that @MyMacro would create -let colorWrapper = Struct("ColorWrapper") { - Variable(.let, name: "value", type: "Color") - - Init { - Parameter(name: "value", type: "Color") + // MARK: Example 3 — Freestanding Expression Macro Generation + Tuple { + VariableExp("42 + 8") + Literal.string("42 + 8") } - - ComputedProperty("description") { - Return { - VariableExp("value.description") - } - } -} -// MARK: - Example 3: Freestanding Expression Macro Generation -// This shows how to generate the tuple that #stringify would create + // MARK: Example 4 — Complex Extension Generation + Extension("User") { + Enum("Status") { + EnumCase("active").equals("active") + EnumCase("inactive").equals("inactive") + EnumCase("pending").equals("pending") + }.inherits("String") + + ComputedProperty("isValid", type: "Bool") { + If(VariableExp("status == .active"), then: { + Return { Literal.boolean(true) } + }, else: { + Return { Literal.boolean(false) } + }) + } -let stringifyResult = Tuple { - VariableExp("42 + 8") - Literal.string("42 + 8") -} + Function("updateStatus") { + Parameter(name: "newStatus", type: "Status") + } _: { + Assignment("status", VariableExp("newStatus")) + Call("print") { + ParameterExp(unlabeled: Literal.string("Status updated to \\(newStatus)")) + } + } -// MARK: - Example 4: Complex Extension Generation -// This shows how to generate a complex extension with multiple members + Function("createDefault") { + } _: { + Return { + Init("User") { + ParameterExp(name: "status", value: VariableExp(".pending")) + ParameterExp(name: "name", value: Literal.string("Default")) + } + } + }.static() + }.inherits("Identifiable", "Codable") + + // MARK: Example 5 — Error Handling Structure + Enum("MacroError") { + EnumCase("onlyWorksWithEnums") + EnumCase("invalidCaseName").associatedValue("name", type: "String") + EnumCase("missingRawValue") + }.inherits("Error", "CustomStringConvertible") + + // MARK: Example 8 — Protocol Generation + Protocol("MyProtocol") { + PropertyRequirement("description", type: "String", access: .get) + } -let userExtension = Extension("User") { - // Add a nested enum - Enum("Status") { - EnumCase("active").equals("active") - EnumCase("inactive").equals("inactive") - EnumCase("pending").equals("pending") - }.inherits("String") - - // Add a computed property with complex logic - ComputedProperty("isValid") { - If(VariableExp("status == .active"), then: { - Return { Literal.boolean(true) } - }, else: { - Return { Literal.boolean(false) } + // MARK: Example 9 — Complex Control Flow Generation + Function("processData") { + Parameter(name: "data", type: "[String]") + } _: { + Variable(.var, name: "result", equals: "[]") + For(VariableExp("item"), in: VariableExp("data"), then: { + If(VariableExp("item.hasPrefix(\"test\")"), then: { + Call("result.append") { + ParameterExp(unlabeled: VariableExp("item.uppercased()")) + } + }, else: { + Call("result.append") { + ParameterExp(unlabeled: VariableExp("item.lowercased()")) + } + }) }) + Return { VariableExp("result") } } - - // Add a method with parameters - Function("updateStatus", parameters: [Parameter("newStatus", type: "Status")]) { - Assignment("status", VariableExp("newStatus")) - Call("print") { - ParameterExp(unlabeled: "\"Status updated to \\(newStatus)\"") - } - } - - // Add a static method - Function("createDefault", parameters: []) { - Return { - Init("User") { - Parameter(name: "status", value: ".pending") - Parameter(name: "name", value: "\"Default\"") + + // MARK: Example 10 — Nested Structure Generation + Struct("ComplexStruct") { + Enum("NestedEnum") { + EnumCase("case1") + EnumCase("case2").equals("value2") + }.inherits("String") + + Struct("NestedStruct") { + Variable(.let, name: "id", type: "UUID") + Variable(.var, name: "name", type: "String") + + VariableExp(""" + init(name: String) { + self.id = UUID() + self.name = name + } + """) + + ComputedProperty("displayName", type: "String") { + Return { + VariableExp("name.isEmpty ? \"Unknown\" : name") + } } } - }.static() - -}.inherits("Identifiable", "Codable") - -// MARK: - Example 5: Error Handling Structure -// This shows how to generate error handling code -let macroError = Enum("MacroError") { - EnumCase("onlyWorksWithEnums") - EnumCase("invalidCaseName").associatedValue("name", type: "String") - EnumCase("missingRawValue") -}.inherits("Error", "CustomStringConvertible") + Variable(.let, name: "enumValue", type: "NestedEnum") + Variable(.var, name: "structValue", type: "NestedStruct") -// MARK: - Example 6: Test Code Generation -// This shows how to generate test code for macros - -let testFunction = Function("testExtensionMacro", parameters: []) { - Call("assertMacroExpansion") { - ParameterExp(name: "input", value: "\"\"\"\n@MyMacro\nenum Color: String {\n case red = \"red\"\n case blue = \"blue\"\n}\n\"\"\"") - ParameterExp(name: "expected", value: "\"\"\"\nenum Color: String {\n case red = \"red\"\n case blue = \"blue\"\n}\n\nextension Color: MyProtocol {\n typealias MyType = String\n static let myProperty = [\"red\", \"blue\"]\n var description: String {\n return myProperty.joined(separator: \", \")\n }\n}\n\nstruct ColorWrapper {\n let value: Color\n init(value: Color) {\n self.value = value\n }\n var description: String {\n return value.description\n }\n}\n\"\"\"") - ParameterExp(name: "macros", value: "[\"MyMacro\": MyMacro.self]") + Function("updateName") { + Parameter(name: "newName", type: "String") + } _: { + Assignment("structValue.name", VariableExp("newName")) + Call("print") { + ParameterExp(unlabeled: Literal.string("Name updated to: \\(newName)")) + } + } } -}.throws() -// MARK: - Example 7: Integration Example -// This shows how to mix SyntaxKit with raw SwiftSyntax - -let baseStruct = Struct("Generated") { - Variable(.let, name: "value", type: "String") -} - -// Convert to SwiftSyntax and modify (this would be done in the macro) -// var structDecl = baseStruct.syntax.as(StructDeclSyntax.self)! -// structDecl = structDecl.with(\.modifiers, DeclModifierListSyntax { -// DeclModifierSyntax(name: .keyword(.public)) -// }) - -// MARK: - Example 8: Protocol Generation -// This shows how to generate protocols that macros might need - -let myProtocol = Protocol("MyProtocol") { - PropertyRequirement("description", type: "String", access: .get) -} - -// MARK: - Example 9: Complex Control Flow Generation -// This shows how to generate complex control flow in macros - -let complexFunction = Function("processData", parameters: [Parameter("data", type: "[String]")]) { - Variable(.var, name: "result", equals: "[]") - - For(VariableExp("item"), in: VariableExp("data"), then: { - If(VariableExp("item.hasPrefix(\"test\")"), then: { - Call("result.append") { - ParameterExp(unlabeled: "item.uppercased()") + // MARK: Example 11 — Switch Statement Generation + Function("handleStatus") { + Parameter(name: "status", type: "UserStatus") + } _: { + Switch("status") { + SwitchCase(".active") { + Call("print") { + ParameterExp(unlabeled: Literal.string("User is active")) + } + Return { Literal.boolean(true) } } - }, else: { - Call("result.append") { - ParameterExp(unlabeled: "item.lowercased()") + SwitchCase(".inactive") { + Call("print") { + ParameterExp(unlabeled: Literal.string("User is inactive")) + } + Return { Literal.boolean(false) } } - }) - }) - - Return { - VariableExp("result") - } -} - -// MARK: - Example 10: Nested Structure Generation -// This shows how to generate nested structures - -let complexStruct = Struct("ComplexStruct") { - // Nested enum - Enum("NestedEnum") { - EnumCase("case1") - EnumCase("case2").equals("value2") - }.inherits("String") - - // Nested struct - Struct("NestedStruct") { - Variable(.let, name: "id", type: "UUID") - Variable(.var, name: "name", type: "String") - - Init { - Parameter(name: "name", type: "String") - } - - ComputedProperty("displayName") { - Return { - VariableExp("name.isEmpty ? \"Unknown\" : name") + SwitchCase(".pending") { + Call("print") { + ParameterExp(unlabeled: Literal.string("User status is pending")) + } + Return { Literal.boolean(false) } + } + Default { + Call("print") { + ParameterExp(unlabeled: Literal.string("Unknown status")) + } + Return { Literal.boolean(false) } } } } - - // Properties - Variable(.let, name: "enumValue", type: "NestedEnum") - Variable(.var, name: "structValue", type: "NestedStruct") - - // Methods - Function("updateName", parameters: [Parameter("newName", type: "String")]) { - Assignment("structValue.name", VariableExp("newName")) - Call("print") { - ParameterExp(unlabeled: "\"Name updated to: \\(newName)\"") - } - } -} - -// MARK: - Example 11: Switch Statement Generation -// This shows how to generate switch statements in macros -let switchFunction = Function("handleStatus", parameters: [Parameter("status", type: "UserStatus")]) { - Switch("status") { - SwitchCase(".active") { + // MARK: Example 12 — Guard Statement Generation + Function("validateUser") { + Parameter(name: "user", type: "User?") + } _: { + Guard { + Let("user", "user") + } else: { Call("print") { - ParameterExp(unlabeled: "\"User is active\"") + ParameterExp(unlabeled: Literal.string("User is nil")) } - Return { Literal.boolean(true) } + Return { Literal.boolean(false) } } - SwitchCase(".inactive") { + + Guard { + Let("name", "user.name") + Let("nameLength", "name.count") + } else: { Call("print") { - ParameterExp(unlabeled: "\"User is inactive\"") + ParameterExp(unlabeled: Literal.string("Invalid user name")) } Return { Literal.boolean(false) } } - SwitchCase(".pending") { + + If(VariableExp("nameLength > 0"), then: { Call("print") { - ParameterExp(unlabeled: "\"User status is pending\"") + ParameterExp(unlabeled: Literal.string("User \\(name) is valid")) } - Return { Literal.boolean(false) } - } - Default { + Return { Literal.boolean(true) } + }, else: { Call("print") { - ParameterExp(unlabeled: "\"Unknown status\"") + ParameterExp(unlabeled: Literal.string("User name is empty")) } Return { Literal.boolean(false) } - } - } -} - -// MARK: - Example 12: Guard Statement Generation -// This shows how to generate guard statements in macros - -let guardFunction = Function("validateUser", parameters: [Parameter("user", type: "User?")]) { - Guard { - Let("user", "user") - } else: { - Call("print") { - ParameterExp(unlabeled: "\"User is nil\"") - } - Return { Literal.boolean(false) } - } - - Guard { - Let("name", "user.name") - Let("nameLength", "name.count") - } else: { - Call("print") { - ParameterExp(unlabeled: "\"Invalid user name\"") - } - Return { Literal.boolean(false) } + }) } - - If(VariableExp("nameLength > 0"), then: { - Call("print") { - ParameterExp(unlabeled: "\"User \\(name) is valid\"") - } - Return { Literal.boolean(true) } - }, else: { - Call("print") { - ParameterExp(unlabeled: "\"User name is empty\"") - } - Return { Literal.boolean(false) } - }) } - -// MARK: - Generate All Examples - -let allExamples = Group { - colorExtension - colorWrapper - stringifyResult - userExtension - macroError - testFunction - myProtocol - complexFunction - complexStruct - switchFunction - guardFunction -} - -// Print the generated code -print(allExamples.generateCode()) \ No newline at end of file From b0008fc2ec2b5246083fb6bed20a364cb1bf44ca Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 15:08:38 -0400 Subject: [PATCH 18/28] Move enum_generator out of Examples/Completed/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Examples/Completed/ holds single-file rendered-DSL examples that skitrun can take as input. enum_generator never fit that shape — it's a full demo project (Package.swift, main.swift, before/after dirs, INTEGRATION_GUIDE.md, JSON config). Move it under a new Examples/Demos/ bucket so the Completed/ contract stays clean. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../enum_generator/EnumGeneratorExample.swift | 0 .../{Completed => Demos}/enum_generator/INTEGRATION_GUIDE.md | 2 +- Examples/{Completed => Demos}/enum_generator/Package.resolved | 0 Examples/{Completed => Demos}/enum_generator/Package.swift | 0 Examples/{Completed => Demos}/enum_generator/README.md | 0 .../{Completed => Demos}/enum_generator/after/Generated.swift | 0 .../enum_generator/after/enum_generator.swift | 0 Examples/{Completed => Demos}/enum_generator/api-config.json | 0 .../enum_generator/before/APIEndpoint.swift | 0 .../{Completed => Demos}/enum_generator/before/HTTPStatus.swift | 0 .../enum_generator/before/NetworkError.swift | 0 Examples/{Completed => Demos}/enum_generator/code.swift | 0 Examples/{Completed => Demos}/enum_generator/demo.swift | 0 Examples/{Completed => Demos}/enum_generator/dsl.swift | 0 Examples/{Completed => Demos}/enum_generator/generate.swift | 2 +- .../{Completed => Demos}/enum_generator/integration_demo.swift | 0 Examples/{Completed => Demos}/enum_generator/main.swift | 0 17 files changed, 2 insertions(+), 2 deletions(-) rename Examples/{Completed => Demos}/enum_generator/EnumGeneratorExample.swift (100%) rename Examples/{Completed => Demos}/enum_generator/INTEGRATION_GUIDE.md (98%) rename Examples/{Completed => Demos}/enum_generator/Package.resolved (100%) rename Examples/{Completed => Demos}/enum_generator/Package.swift (100%) rename Examples/{Completed => Demos}/enum_generator/README.md (100%) rename Examples/{Completed => Demos}/enum_generator/after/Generated.swift (100%) rename Examples/{Completed => Demos}/enum_generator/after/enum_generator.swift (100%) rename Examples/{Completed => Demos}/enum_generator/api-config.json (100%) rename Examples/{Completed => Demos}/enum_generator/before/APIEndpoint.swift (100%) rename Examples/{Completed => Demos}/enum_generator/before/HTTPStatus.swift (100%) rename Examples/{Completed => Demos}/enum_generator/before/NetworkError.swift (100%) rename Examples/{Completed => Demos}/enum_generator/code.swift (100%) rename Examples/{Completed => Demos}/enum_generator/demo.swift (100%) rename Examples/{Completed => Demos}/enum_generator/dsl.swift (100%) rename Examples/{Completed => Demos}/enum_generator/generate.swift (94%) rename Examples/{Completed => Demos}/enum_generator/integration_demo.swift (100%) rename Examples/{Completed => Demos}/enum_generator/main.swift (100%) diff --git a/Examples/Completed/enum_generator/EnumGeneratorExample.swift b/Examples/Demos/enum_generator/EnumGeneratorExample.swift similarity index 100% rename from Examples/Completed/enum_generator/EnumGeneratorExample.swift rename to Examples/Demos/enum_generator/EnumGeneratorExample.swift diff --git a/Examples/Completed/enum_generator/INTEGRATION_GUIDE.md b/Examples/Demos/enum_generator/INTEGRATION_GUIDE.md similarity index 98% rename from Examples/Completed/enum_generator/INTEGRATION_GUIDE.md rename to Examples/Demos/enum_generator/INTEGRATION_GUIDE.md index a355d1e..0302a80 100644 --- a/Examples/Completed/enum_generator/INTEGRATION_GUIDE.md +++ b/Examples/Demos/enum_generator/INTEGRATION_GUIDE.md @@ -6,7 +6,7 @@ This guide demonstrates the real-world impact of using SyntaxKit for dynamic enu ```bash # See the value proposition in action -cd Examples/Completed/enum_generator +cd Examples/Demos/enum_generator swift demo.swift ``` diff --git a/Examples/Completed/enum_generator/Package.resolved b/Examples/Demos/enum_generator/Package.resolved similarity index 100% rename from Examples/Completed/enum_generator/Package.resolved rename to Examples/Demos/enum_generator/Package.resolved diff --git a/Examples/Completed/enum_generator/Package.swift b/Examples/Demos/enum_generator/Package.swift similarity index 100% rename from Examples/Completed/enum_generator/Package.swift rename to Examples/Demos/enum_generator/Package.swift diff --git a/Examples/Completed/enum_generator/README.md b/Examples/Demos/enum_generator/README.md similarity index 100% rename from Examples/Completed/enum_generator/README.md rename to Examples/Demos/enum_generator/README.md diff --git a/Examples/Completed/enum_generator/after/Generated.swift b/Examples/Demos/enum_generator/after/Generated.swift similarity index 100% rename from Examples/Completed/enum_generator/after/Generated.swift rename to Examples/Demos/enum_generator/after/Generated.swift diff --git a/Examples/Completed/enum_generator/after/enum_generator.swift b/Examples/Demos/enum_generator/after/enum_generator.swift similarity index 100% rename from Examples/Completed/enum_generator/after/enum_generator.swift rename to Examples/Demos/enum_generator/after/enum_generator.swift diff --git a/Examples/Completed/enum_generator/api-config.json b/Examples/Demos/enum_generator/api-config.json similarity index 100% rename from Examples/Completed/enum_generator/api-config.json rename to Examples/Demos/enum_generator/api-config.json diff --git a/Examples/Completed/enum_generator/before/APIEndpoint.swift b/Examples/Demos/enum_generator/before/APIEndpoint.swift similarity index 100% rename from Examples/Completed/enum_generator/before/APIEndpoint.swift rename to Examples/Demos/enum_generator/before/APIEndpoint.swift diff --git a/Examples/Completed/enum_generator/before/HTTPStatus.swift b/Examples/Demos/enum_generator/before/HTTPStatus.swift similarity index 100% rename from Examples/Completed/enum_generator/before/HTTPStatus.swift rename to Examples/Demos/enum_generator/before/HTTPStatus.swift diff --git a/Examples/Completed/enum_generator/before/NetworkError.swift b/Examples/Demos/enum_generator/before/NetworkError.swift similarity index 100% rename from Examples/Completed/enum_generator/before/NetworkError.swift rename to Examples/Demos/enum_generator/before/NetworkError.swift diff --git a/Examples/Completed/enum_generator/code.swift b/Examples/Demos/enum_generator/code.swift similarity index 100% rename from Examples/Completed/enum_generator/code.swift rename to Examples/Demos/enum_generator/code.swift diff --git a/Examples/Completed/enum_generator/demo.swift b/Examples/Demos/enum_generator/demo.swift similarity index 100% rename from Examples/Completed/enum_generator/demo.swift rename to Examples/Demos/enum_generator/demo.swift diff --git a/Examples/Completed/enum_generator/dsl.swift b/Examples/Demos/enum_generator/dsl.swift similarity index 100% rename from Examples/Completed/enum_generator/dsl.swift rename to Examples/Demos/enum_generator/dsl.swift diff --git a/Examples/Completed/enum_generator/generate.swift b/Examples/Demos/enum_generator/generate.swift similarity index 94% rename from Examples/Completed/enum_generator/generate.swift rename to Examples/Demos/enum_generator/generate.swift index d403b4e..ea13b07 100644 --- a/Examples/Completed/enum_generator/generate.swift +++ b/Examples/Demos/enum_generator/generate.swift @@ -8,7 +8,7 @@ let process = Process() process.executableURL = URL(fileURLWithPath: "/usr/bin/swift") process.arguments = [ "run", "--package-path", "../../../", "swift", - "Examples/Completed/enum_generator/dsl.swift" + "Examples/Demos/enum_generator/dsl.swift" ] process.currentDirectoryURL = URL(fileURLWithPath: FileManager.default.currentDirectoryPath) diff --git a/Examples/Completed/enum_generator/integration_demo.swift b/Examples/Demos/enum_generator/integration_demo.swift similarity index 100% rename from Examples/Completed/enum_generator/integration_demo.swift rename to Examples/Demos/enum_generator/integration_demo.swift diff --git a/Examples/Completed/enum_generator/main.swift b/Examples/Demos/enum_generator/main.swift similarity index 100% rename from Examples/Completed/enum_generator/main.swift rename to Examples/Demos/enum_generator/main.swift From 18f7097d2af155caf1c2bd20fbd7c9bc43679a67 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 15:21:16 -0400 Subject: [PATCH 19/28] Unify skit + skitrun into one binary with ArgumentParser subcommands MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `skit` is now a single CLI with two verbs: skit run # was `skitrun` — render DSL → Swift skit parse # was `skit` — stdin Swift → JSON Default subcommand is `run`, so `skit Input.swift` works without a verb. CLI parsing moves to swift-argument-parser, replacing the hand-rolled CLIArgs parser. The declarative @Option/@Flag/@Argument surface produces better-formatted help for free and validates inputs at parse time. - Sources/skitrun/ folded into Sources/skit/ (Main.swift → Runner.swift, Helpers + OutputCache unchanged in substance). - skitrun product/target removed from Package.swift; skit gains SwiftSyntax / SwiftParser / Crypto / ArgumentParser deps. - Env vars: SKITRUN_LIB_DIR → SKIT_LIB_DIR; cache-key env prefix SKITRUN_* → SKIT_*. Bundle dir: .build/skitrun-release/ → .build/skit-release/. Homebrew fallback path: lib/skitrun/ → lib/skit/. - Release script moves to Scripts/build-skit-release.sh — promoted from Docs/research/poc-step4-release.sh to reflect that it's now a shipping build script, not a POC step. - Wrapper internal name: __skitrun_root → __skit_root (only visible in spawned-swift error messages on wrapper-line failures). All 10 working Examples/Completed/*/dsl.swift still render through `skit run` (and the default-subcommand form). Co-Authored-By: Claude Opus 4.7 (1M context) --- Package.resolved | 11 +- Package.swift | 16 +- .../build-skit-release.sh | 44 +-- Sources/{skitrun => skit}/Helpers.swift | 2 +- Sources/{skitrun => skit}/OutputCache.swift | 6 +- Sources/{skitrun => skit}/README.md | 0 .../{skitrun/Main.swift => skit/Runner.swift} | 286 +++--------------- Sources/skit/Skit.swift | 217 ++++++++++++- 8 files changed, 286 insertions(+), 296 deletions(-) rename Docs/research/poc-step4-release.sh => Scripts/build-skit-release.sh (63%) rename Sources/{skitrun => skit}/Helpers.swift (99%) rename Sources/{skitrun => skit}/OutputCache.swift (94%) rename Sources/{skitrun => skit}/README.md (100%) rename Sources/{skitrun/Main.swift => skit/Runner.swift} (67%) diff --git a/Package.resolved b/Package.resolved index 851eeb1..07e1898 100644 --- a/Package.resolved +++ b/Package.resolved @@ -1,6 +1,15 @@ { - "originHash" : "6a75ee274433215501c77ebca768e3c82c684598296596b112f9800ac08fa2fe", + "originHash" : "36e6466ffd4edf53c9520bb53570c9119c1a34eb0e59dd824b1c4ff4f19189a8", "pins" : [ + { + "identity" : "swift-argument-parser", + "kind" : "remoteSourceControl", + "location" : "https://github.com/apple/swift-argument-parser.git", + "state" : { + "revision" : "626b5b7b2f45e1b0b1c6f4a309296d1d21d7311b", + "version" : "1.7.1" + } + }, { "identity" : "swift-asn1", "kind" : "remoteSourceControl", diff --git a/Package.swift b/Package.swift index 13c44a1..551ba63 100644 --- a/Package.swift +++ b/Package.swift @@ -95,15 +95,12 @@ let package = Package( name: "skit", targets: ["skit"] ), - .executable( - name: "skitrun", - targets: ["skitrun"] - ), ], dependencies: [ .package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.1"), .package(url: "https://github.com/swiftlang/swift-docc-plugin", from: "1.4.0"), - .package(url: "https://github.com/apple/swift-crypto.git", from: "3.0.0") + .package(url: "https://github.com/apple/swift-crypto.git", from: "3.0.0"), + .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0") ], targets: [ .target( @@ -146,15 +143,12 @@ let package = Package( ), .executableTarget( name: "skit", - dependencies: ["SyntaxParser"], - swiftSettings: swiftSettings - ), - .executableTarget( - name: "skitrun", dependencies: [ + "SyntaxParser", .product(name: "SwiftSyntax", package: "swift-syntax"), .product(name: "SwiftParser", package: "swift-syntax"), - .product(name: "Crypto", package: "swift-crypto") + .product(name: "Crypto", package: "swift-crypto"), + .product(name: "ArgumentParser", package: "swift-argument-parser") ], swiftSettings: swiftSettings ), diff --git a/Docs/research/poc-step4-release.sh b/Scripts/build-skit-release.sh similarity index 63% rename from Docs/research/poc-step4-release.sh rename to Scripts/build-skit-release.sh index 795d2a2..78eefe7 100755 --- a/Docs/research/poc-step4-release.sh +++ b/Scripts/build-skit-release.sh @@ -1,28 +1,30 @@ #!/usr/bin/env bash # -# POC step 4: build a self-contained skitrun release bundle. +# Build a self-contained skit release bundle. # -# Output: .build/skitrun-release/ -# skitrun ← the CLI binary +# Output: .build/skit-release/ +# skit ← the CLI binary # lib/ -# libSyntaxKit.dylib ← release + strip -x -# *.swiftmodule ← SyntaxKit + transitively re-exported modules -# _SwiftSyntaxCShims-include/ ← C-shims headers (module map + .h files) +# libSyntaxKit.dylib ← release + strip -x +# *.swiftmodule ← SyntaxKit + transitively re-exported modules +# _SwiftSyntaxCShims-include/ ← C-shims headers (module map + .h files) +# swift-version.txt ← toolchain stamp for startup check # -# Once produced, the binary is portable: copy the whole .build/skitrun-release/ -# directory anywhere, and `./skitrun-release/skitrun ` Just Works — no +# Once produced, the bundle is portable: copy the whole .build/skit-release/ +# directory anywhere, and `./skit-release/skit ` Just Works — no # flags, no env vars, no SyntaxKit checkout required. set -euo pipefail if [[ "$(uname -s)" != "Darwin" ]]; then - echo "macOS-only for now. Linux smoke test is POC step 7." >&2 + echo "macOS-only. Linux uses a parallel flow (build, then strip the" >&2 + echo "Mach-O install_name step)." >&2 exit 1 fi SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -OUTPUT_DIR="$REPO_ROOT/.build/skitrun-release" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +OUTPUT_DIR="$REPO_ROOT/.build/skit-release" PACKAGE_FILE="$REPO_ROOT/Package.swift" PACKAGE_BACKUP="$(mktemp)" @@ -50,10 +52,10 @@ PY cd "$REPO_ROOT" -echo "==> swift build -c release --product skitrun" -swift build -c release --product skitrun +echo "==> swift build -c release --product skit" +swift build -c release --product skit -# skitrun doesn't depend on SyntaxKit (it spawns swift on user input that +# `skit` doesn't depend on SyntaxKit (it spawns swift on user input that # imports SyntaxKit at runtime). Build the library product explicitly so the # .dynamic flip above produces libSyntaxKit.dylib + swiftmodule. echo "==> swift build -c release --product SyntaxKit" @@ -69,7 +71,7 @@ echo "==> Staging $OUTPUT_DIR" rm -rf "$OUTPUT_DIR" mkdir -p "$OUTPUT_DIR/lib" -cp "$BUILD_DIR/skitrun" "$OUTPUT_DIR/skitrun" +cp "$BUILD_DIR/skit" "$OUTPUT_DIR/skit" cp "$BUILD_DIR/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/" strip -x "$OUTPUT_DIR/lib/libSyntaxKit.dylib" cp -r "$BUILD_DIR/Modules/." "$OUTPUT_DIR/lib/" @@ -79,13 +81,13 @@ cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/inclu # Ensure the dylib's install_name uses @rpath so it's portable. install_name_tool -id "@rpath/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/libSyntaxKit.dylib" 2>/dev/null || true -# Stamp the bundle with the build toolchain. skitrun compares this against -# the user's `swift --version` at startup and refuses to spawn `swift` if the -# swiftmodule wouldn't load (see Sources/skitrun/Main.swift). Issue #157 will -# replace the refusal with an auto-rebuild fallback. +# Stamp the bundle with the build toolchain. `skit` compares this against the +# user's `swift --version` at startup and refuses to spawn `swift` if the +# swiftmodule wouldn't load. Issue #157 will replace the refusal with an +# auto-rebuild fallback. swift --version > "$OUTPUT_DIR/lib/swift-version.txt" -BINARY_SIZE=$(ls -lh "$OUTPUT_DIR/skitrun" | awk '{print $5}') +BINARY_SIZE=$(ls -lh "$OUTPUT_DIR/skit" | awk '{print $5}') DYLIB_SIZE=$(ls -lh "$OUTPUT_DIR/lib/libSyntaxKit.dylib" | awk '{print $5}') TOTAL_SIZE=$(du -sh "$OUTPUT_DIR" | awk '{print $1}') @@ -96,4 +98,4 @@ echo " Dylib: $DYLIB_SIZE" echo " Total: $TOTAL_SIZE" echo echo "==> Try it:" -echo " $OUTPUT_DIR/skitrun " +echo " $OUTPUT_DIR/skit run " diff --git a/Sources/skitrun/Helpers.swift b/Sources/skit/Helpers.swift similarity index 99% rename from Sources/skitrun/Helpers.swift rename to Sources/skit/Helpers.swift index 6a87f88..7a33b51 100644 --- a/Sources/skitrun/Helpers.swift +++ b/Sources/skit/Helpers.swift @@ -211,7 +211,7 @@ private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) t let stderr = String(decoding: stderrData, as: UTF8.self) throw CLIError( message: """ - skitrun: failed to compile Helpers/ (exit \(process.terminationStatus)) + skit: failed to compile Helpers/ (exit \(process.terminationStatus)) \(stderr) """) } diff --git a/Sources/skitrun/OutputCache.swift b/Sources/skit/OutputCache.swift similarity index 94% rename from Sources/skitrun/OutputCache.swift rename to Sources/skit/OutputCache.swift index 3f094b0..d4a26df 100644 --- a/Sources/skitrun/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -1,6 +1,6 @@ // // OutputCache.swift -// SyntaxKit — skitrun (POC step 6 for issue #154) +// SyntaxKit // // Created by Leo Dion. // Copyright © 2026 BrightDigit. @@ -34,7 +34,7 @@ import Foundation private let outputCacheSchemaVersion = "v1" /// SHA-256 over (cache schema, input source bytes, helpers key, swift version, -/// libSyntaxKit stamp, sorted SKITRUN_*/SYNTAXKIT_* env vars). Any change in +/// libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). Any change in /// these inputs produces a fresh key and forces a recompile. internal func outputCacheKey( inputSource: String, @@ -60,7 +60,7 @@ internal func outputCacheKey( } let env = ProcessInfo.processInfo.environment - .filter { $0.key.hasPrefix("SKITRUN_") || $0.key.hasPrefix("SYNTAXKIT_") } + .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } .sorted { $0.key < $1.key } for (key, value) in env { hasher.update(data: Data("\(key)=\(value)\0".utf8)) diff --git a/Sources/skitrun/README.md b/Sources/skit/README.md similarity index 100% rename from Sources/skitrun/README.md rename to Sources/skit/README.md diff --git a/Sources/skitrun/Main.swift b/Sources/skit/Runner.swift similarity index 67% rename from Sources/skitrun/Main.swift rename to Sources/skit/Runner.swift index 5accbd6..957df18 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skit/Runner.swift @@ -1,5 +1,5 @@ // -// Main.swift +// Runner.swift // SyntaxKit // // Created by Leo Dion. @@ -31,74 +31,15 @@ import Foundation import SwiftParser import SwiftSyntax -@main -internal enum SkitRun { - internal static func main() async throws { - let args: CLIArgs - do { - args = try CLIArgs.parse(CommandLine.arguments) - } catch { - FileHandle.standardError.write(Data("\(error)\n".utf8)) - exit(2) - } - - let libPath: String - do { - libPath = try resolveLibPath(override: args.libPath) - } catch { - FileHandle.standardError.write(Data("\(error)\n".utf8)) - exit(2) - } - - if args.checkToolchain { - switch toolchainCheck(libPath: libPath) { - case .match, .stampMissing: - break - case .mismatch(let bundle, let local): - FileHandle.standardError.write(Data(toolchainMismatchMessage( - bundle: bundle, local: local - ).utf8)) - exit(2) - } - } +// MARK: - Helpers resolution - switch args.mode { - case .singleFile(let input, let output): - let helpers = try resolveHelpers( - nearInputPath: input, - libPath: libPath, - options: args.helpers - ) - try runSingleFile( - inputPath: input, - outputPath: output, - libPath: libPath, - helpers: helpers, - useCache: args.useCache, - timeoutSeconds: args.timeoutSeconds - ) - case .directory(let inputDir, let outputDir): - let helpers = try resolveHelpers( - nearInputPath: inputDir, - libPath: libPath, - options: args.helpers - ) - let exitCode = await runDirectory( - inputDir: inputDir, - outputDir: outputDir, - libPath: libPath, - helpers: helpers, - useCache: args.useCache, - timeoutSeconds: args.timeoutSeconds - ) - exit(exitCode) - } - } +internal enum HelpersOptions { + case auto + case disabled + case explicit(String) } -// MARK: - Helpers resolution - -private func resolveHelpers( +internal func resolveHelpers( nearInputPath path: String, libPath: String, options: HelpersOptions @@ -127,7 +68,7 @@ private func resolveHelpers( let suffix = compiled.cacheHit ? "cached" : "compiled" FileHandle.standardError.write( Data( - "skitrun: helpers \(suffix) at \(helpersDir.path)\n".utf8 + "skit: helpers \(suffix) at \(helpersDir.path)\n".utf8 )) return compiled } @@ -136,7 +77,7 @@ private func resolveHelpers( /// Resolves the directory containing `libSyntaxKit.dylib` + module files, /// in priority order: explicit flag → env var → adjacent-to-binary -/// (`/lib/`) → Homebrew layout (`/../lib/skitrun/`). +/// (`/lib/`) → Homebrew layout (`/../lib/skit/`). internal func resolveLibPath(override: String?) throws -> String { if let override { guard isLibDir(override) else { @@ -145,9 +86,9 @@ internal func resolveLibPath(override: String?) throws -> String { return override } - if let env = ProcessInfo.processInfo.environment["SKITRUN_LIB_DIR"], !env.isEmpty { + if let env = ProcessInfo.processInfo.environment["SKIT_LIB_DIR"], !env.isEmpty { guard isLibDir(env) else { - throw CLIError(message: "SKITRUN_LIB_DIR is set but path is not a lib dir: \(env)") + throw CLIError(message: "SKIT_LIB_DIR is set but path is not a lib dir: \(env)") } return env } @@ -159,7 +100,7 @@ internal func resolveLibPath(override: String?) throws -> String { if isLibDir(adjacent) { return adjacent } let brewLayout = execDir.deletingLastPathComponent() - .appendingPathComponent("lib/skitrun").path + .appendingPathComponent("lib/skit").path if isLibDir(brewLayout) { return brewLayout } } @@ -167,11 +108,11 @@ internal func resolveLibPath(override: String?) throws -> String { message: """ Could not locate SyntaxKit lib directory. Looked for: 1. --lib (not provided) - 2. $SKITRUN_LIB_DIR (not set) - 3. /lib/ (not found) - 4. /../lib/skitrun/ (not found) - Run Docs/research/poc-step4-release.sh to produce a self-contained - release bundle under .build/skitrun-release/. + 2. $SKIT_LIB_DIR (not set) + 3. /lib/ (not found) + 4. /../lib/skit/ (not found) + Run Scripts/build-skit-release.sh to produce a self-contained + release bundle under .build/skit-release/. """) } @@ -191,15 +132,15 @@ internal enum ToolchainCheckResult { /// Bundle stamp matches the local `swift --version` exactly. case match /// `/swift-version.txt` is missing (older bundle that predates - /// the stamp). skitrun prints a one-line note and proceeds. + /// the stamp). skit prints a one-line note and proceeds. case stampMissing case mismatch(bundle: String, local: String) } /// Compares `/swift-version.txt` to `captureSwiftVersion()`. /// The swiftmodule format isn't reliably forward-compatible across even -/// patch-level Swift releases (the originating bug: 6.3.0 → 6.3.2 rejected -/// the swiftmodule), so the comparison is exact-string after normalising +/// patch-level Swift releases (originating bug: 6.3.0 → 6.3.2 rejected the +/// swiftmodule), so the comparison is exact-string after normalising /// trailing whitespace. internal func toolchainCheck(libPath: String) -> ToolchainCheckResult { let stampURL = URL(fileURLWithPath: libPath).appendingPathComponent(toolchainStampFilename) @@ -207,13 +148,13 @@ internal func toolchainCheck(libPath: String) -> ToolchainCheckResult { let stampRaw = String(data: stampData, encoding: .utf8) else { FileHandle.standardError.write( - Data("skitrun: bundle has no toolchain stamp; skipping check\n".utf8) + Data("skit: bundle has no toolchain stamp; skipping check\n".utf8) ) return .stampMissing } guard let localRaw = captureSwiftVersion() else { FileHandle.standardError.write( - Data("skitrun: could not capture local `swift --version`; skipping toolchain check\n".utf8) + Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) ) return .stampMissing } @@ -224,7 +165,7 @@ internal func toolchainCheck(libPath: String) -> ToolchainCheckResult { internal func toolchainMismatchMessage(bundle: String, local: String) -> String { """ - skitrun: toolchain mismatch + skit: toolchain mismatch bundle: \(bundle) local: \(local) The bundle's libSyntaxKit was built against a different `swift` than the @@ -233,7 +174,7 @@ internal func toolchainMismatchMessage(bundle: String, local: String) -> String diagnostic. Rebuild the bundle with: - Docs/research/poc-step4-release.sh + Scripts/build-skit-release.sh Or pass --no-toolchain-check to try anyway. """ @@ -241,7 +182,7 @@ internal func toolchainMismatchMessage(bundle: String, local: String) -> String // MARK: - Single-file mode -private func runSingleFile( +internal func runSingleFile( inputPath: String, outputPath: String?, libPath: String, @@ -271,7 +212,7 @@ private func runSingleFile( // MARK: - Folder mode -private func runDirectory( +internal func runDirectory( inputDir: String, outputDir: String, libPath: String, @@ -286,12 +227,12 @@ private func runDirectory( do { inputs = try collectInputs(at: inputURL, excluding: helpersExcludePath(inputDir: inputURL)) } catch { - FileHandle.standardError.write(Data("skitrun: failed to walk \(inputDir): \(error)\n".utf8)) + FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) return 1 } if inputs.isEmpty { - FileHandle.standardError.write(Data("skitrun: no .swift inputs under \(inputDir)\n".utf8)) + FileHandle.standardError.write(Data("skit: no .swift inputs under \(inputDir)\n".utf8)) return 0 } @@ -314,11 +255,11 @@ private func runDirectory( outcomes.append(outcome) if let next = iterator.next() { group.addTask { - runOne( - next, libPath: libPath, helpers: helpers, - useCache: useCache, timeoutSeconds: timeoutSeconds - ) - } + runOne( + next, libPath: libPath, helpers: helpers, + useCache: useCache, timeoutSeconds: timeoutSeconds + ) + } } } } @@ -358,7 +299,7 @@ private func runDirectory( FileHandle.standardError.write( Data( - "skitrun: \(outcomes.count - failed)/\(outcomes.count) succeeded\n".utf8 + "skit: \(outcomes.count - failed)/\(outcomes.count) succeeded\n".utf8 )) return failed == 0 ? 0 : 1 @@ -462,7 +403,7 @@ private func processFile( let wrapped = wrap(source: source, originalPath: absoluteInputPath) let tmpDir = FileManager.default.temporaryDirectory - .appendingPathComponent("skitrun-\(UUID().uuidString)") + .appendingPathComponent("skit-\(UUID().uuidString)") try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) defer { try? FileManager.default.removeItem(at: tmpDir) } @@ -490,155 +431,6 @@ private func processFile( return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } -// MARK: - Arg parsing - -internal enum HelpersOptions { - case auto - case disabled - case explicit(String) -} - -private struct CLIArgs { - enum Mode { - case singleFile(input: String, output: String?) - case directory(input: String, output: String) - } - - let mode: Mode - let libPath: String? - let helpers: HelpersOptions - let useCache: Bool - let timeoutSeconds: Int - let checkToolchain: Bool - - static let defaultTimeoutSeconds = 60 - - static func parse(_ argv: [String]) throws -> CLIArgs { - var inputPath: String? - var outputPath: String? - var libPath: String? - var helpers: HelpersOptions = .auto - var useCache = true - var timeoutSeconds = defaultTimeoutSeconds - var checkToolchain = true - - var i = 1 - while i < argv.count { - let arg = argv[i] - switch arg { - case "-o", "--output": - guard i + 1 < argv.count else { throw usage("-o requires a value") } - outputPath = argv[i + 1] - i += 2 - case "--lib": - guard i + 1 < argv.count else { throw usage("--lib requires a value") } - libPath = argv[i + 1] - i += 2 - case "--helpers": - guard i + 1 < argv.count else { throw usage("--helpers requires a value") } - helpers = .explicit(argv[i + 1]) - i += 2 - case "--no-helpers": - helpers = .disabled - i += 1 - case "--no-cache": - useCache = false - i += 1 - case "--no-toolchain-check": - checkToolchain = false - i += 1 - case "--timeout": - guard i + 1 < argv.count else { throw usage("--timeout requires a value") } - guard let parsed = Int(argv[i + 1]), parsed >= 0 else { - throw usage("--timeout expects a non-negative integer (seconds), got: \(argv[i + 1])") - } - timeoutSeconds = parsed - i += 2 - case "-h", "--help": - FileHandle.standardError.write(Data(helpText.utf8)) - exit(0) - case _ where arg.hasPrefix("-"): - throw usage("unknown flag: \(arg)") - default: - guard inputPath == nil else { throw usage("only one input path is supported") } - inputPath = arg - i += 1 - } - } - - guard let inputPath else { throw usage("missing input path") } - - var isDirectory: ObjCBool = false - guard FileManager.default.fileExists(atPath: inputPath, isDirectory: &isDirectory) else { - throw usage("input does not exist: \(inputPath)") - } - - let mode: Mode - if isDirectory.boolValue { - guard let outputPath else { - throw usage("directory inputs require -o ") - } - mode = .directory(input: inputPath, output: outputPath) - } else { - mode = .singleFile(input: inputPath, output: outputPath) - } - - return CLIArgs( - mode: mode, - libPath: libPath, - helpers: helpers, - useCache: useCache, - timeoutSeconds: timeoutSeconds, - checkToolchain: checkToolchain - ) - } -} - -private let helpText = """ - skitrun [-o ] [--lib ] - - POC for issue #154 — runs SyntaxKit DSL input(s) by wrapping each in a - Group { … } closure and spawning `swift`. - - Forms: - skitrun Input.swift — render to stdout - skitrun Input.swift -o Out.swift — render to a file - skitrun InputDir/ -o OutDir/ — walk **/*.swift (skipping files - prefixed with '_') and mirror - rendered output into OutDir/ - - Options: - -o, --output Output file (single-file mode) or directory (folder mode). - --lib Directory containing libSyntaxKit.dylib + module files. - When omitted, skitrun searches: $SKITRUN_LIB_DIR, - then /lib/, then /../lib/skitrun/. - Build a self-contained bundle with - Docs/research/poc-step4-release.sh. - --helpers Override Helpers/ directory location. By default, - skitrun walks up from the input looking for one. - Compiled into libSyntaxKitHelpers.dylib and made - importable via `import SyntaxKitHelpers`. - --no-helpers Skip helpers discovery entirely. - --no-cache Skip the rendered-output cache (always run swift). - The cache lives at /outputs// - and is keyed on input bytes, helpers, swift version, - libSyntaxKit stamp, and SKITRUN_*/SYNTAXKIT_* env. - --timeout Per-input timeout for the spawned `swift` process - (default 60). On expiry: SIGTERM, then SIGKILL after - a 5s grace; the file exits with code 124. Pass 0 to - disable the watchdog. - --no-toolchain-check Skip the startup check that compares the bundle's - recorded build toolchain (/swift-version.txt) - against `swift --version`. Swift swiftmodules aren't - reliably compatible across compiler versions, so by - default skitrun refuses to spawn `swift` on - mismatch. See issue #157 for the auto-rebuild plan. - """ - -private func usage(_ message: String) -> CLIError { - CLIError(message: "\(message)\n\n\(helpText)\n") -} - internal struct CLIError: Error, CustomStringConvertible { let message: String var description: String { message } @@ -695,19 +487,19 @@ internal func wrap(source: String, originalPath: String) -> String { return """ import SyntaxKit \(hoistedBlock) - let __skitrun_root = Group { + let __skit_root = Group { #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) \(body) #sourceLocation() } - print(__skitrun_root.generateCode()) + print(__skit_root.generateCode()) """ } // MARK: - Spawning swift -/// Exit code returned when the spawned `swift` is killed by skitrun's timeout +/// Exit code returned when the spawned `swift` is killed by skit's timeout /// watchdog. Matches POSIX `timeout(1)`. private let timeoutExitCode: Int32 = 124 @@ -802,7 +594,7 @@ private func runSwift( group.wait() if timedOut { - let prefix = Data("skitrun: timed out after \(timeoutSeconds)s\n".utf8) + let prefix = Data("skit: timed out after \(timeoutSeconds)s\n".utf8) let stderr = String(decoding: prefix + errBox.value, as: UTF8.self) return ProcessResult(exitCode: timeoutExitCode, stdout: outBox.value, stderr: stderr) } diff --git a/Sources/skit/Skit.swift b/Sources/skit/Skit.swift index 1066b5e..710a725 100644 --- a/Sources/skit/Skit.swift +++ b/Sources/skit/Skit.swift @@ -27,24 +27,217 @@ // OTHER DEALINGS IN THE SOFTWARE. // +import ArgumentParser import Foundation import SyntaxParser @main -internal enum Skit { - internal static func main() throws { - // Read Swift code from stdin - let code = String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" +internal struct Skit: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "skit", + abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", + subcommands: [Run.self, Parse.self], + defaultSubcommand: Run.self + ) +} + +// MARK: - skit run + +extension Skit { + internal struct Run: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "run", + abstract: "Render SyntaxKit DSL input(s) into Swift source.", + discussion: """ + Wraps each input in a `Group { … }` closure and spawns `swift` to + evaluate it. The rendered output is written to stdout (single-file + mode) or mirrored into an output directory (folder mode). + + Forms: + skit run Input.swift — render to stdout + skit run Input.swift -o Out.swift — render to a file + skit run InputDir/ -o OutDir/ — walk **/*.swift (skipping + files prefixed with '_') + and mirror rendered output + into OutDir/ + """ + ) + + @Argument(help: "Path to a .swift input file or a directory of inputs.") + internal var input: String + + @Option( + name: [.short, .customLong("output")], + help: "Output file (single-file mode) or directory (folder mode)." + ) + internal var output: String? + + @Option( + name: .customLong("lib"), + help: ArgumentHelp( + "Directory containing libSyntaxKit.dylib + module files.", + discussion: """ + When omitted, skit searches: $SKIT_LIB_DIR, then /lib/, + then /../lib/skit/. Build a self-contained bundle with + Scripts/build-skit-release.sh. + """ + ) + ) + internal var libPath: String? + + @Option( + name: .customLong("helpers"), + help: ArgumentHelp( + "Override Helpers/ directory location.", + discussion: """ + By default skit walks up from the input looking for one. Helper + sources are pre-compiled into libSyntaxKitHelpers.dylib and made + importable via `import SyntaxKitHelpers`. + """ + ) + ) + internal var helpersDir: String? + + @Flag( + name: .customLong("no-helpers"), + help: "Skip helpers discovery entirely." + ) + internal var noHelpers: Bool = false + + @Flag( + name: .customLong("no-cache"), + help: ArgumentHelp( + "Skip the rendered-output cache (always run swift).", + discussion: """ + The cache lives at /outputs// and is keyed + on input bytes, helpers, swift version, libSyntaxKit stamp, and + SKIT_*/SYNTAXKIT_* env. + """ + ) + ) + internal var noCache: Bool = false + + @Option( + name: .customLong("timeout"), + help: ArgumentHelp( + "Per-input timeout for the spawned `swift` (seconds).", + discussion: """ + Default 60. On expiry: SIGTERM, then SIGKILL after a 5s grace; the + file exits with code 124. Pass 0 to disable the watchdog. + """ + ) + ) + internal var timeoutSeconds: Int = 60 + + @Flag( + name: .customLong("no-toolchain-check"), + help: ArgumentHelp( + "Skip the bundle/local Swift-toolchain comparison.", + discussion: """ + skit compares /swift-version.txt to `swift --version` at + startup and refuses to spawn `swift` on mismatch — swiftmodules + aren't reliably compatible across compiler versions. See issue + #157 for the auto-rebuild plan. + """ + ) + ) + internal var noToolchainCheck: Bool = false + + internal func validate() throws { + guard timeoutSeconds >= 0 else { + throw ValidationError( + "--timeout expects a non-negative integer (seconds), got: \(timeoutSeconds)" + ) + } + } + + internal func run() async throws { + let libPath: String + do { + libPath = try resolveLibPath(override: self.libPath) + } catch { + FileHandle.standardError.write(Data("\(error)\n".utf8)) + throw ExitCode(2) + } + + if !noToolchainCheck { + switch toolchainCheck(libPath: libPath) { + case .match, .stampMissing: + break + case .mismatch(let bundle, let local): + FileHandle.standardError.write( + Data(toolchainMismatchMessage(bundle: bundle, local: local).utf8)) + throw ExitCode(2) + } + } + + let helpersOptions: HelpersOptions + if noHelpers { + helpersOptions = .disabled + } else if let dir = helpersDir { + helpersOptions = .explicit(dir) + } else { + helpersOptions = .auto + } + + var isDirectory: ObjCBool = false + guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { + throw ValidationError("input does not exist: \(input)") + } + + if isDirectory.boolValue { + guard let output else { + throw ValidationError("directory inputs require -o ") + } + let helpers = try resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + let exitCode = await runDirectory( + inputDir: input, + outputDir: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) + throw ExitCode(exitCode) + } else { + let helpers = try resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + try runSingleFile( + inputPath: input, + outputPath: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) + } + } + } +} - // Parse the code using SyntaxKit - let treeNodes = SyntaxParser.parse(code: code) +// MARK: - skit parse - // Convert to JSON for output - let encoder = JSONEncoder() - let data = try encoder.encode(treeNodes) - let json = String(decoding: data, as: UTF8.self) +extension Skit { + internal struct Parse: ParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "parse", + abstract: "Parse Swift source on stdin into a JSON syntax tree on stdout." + ) - // Output the JSON - print(json) + internal func run() throws { + let code = String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" + let treeNodes = SyntaxParser.parse(code: code) + let encoder = JSONEncoder() + let data = try encoder.encode(treeNodes) + let json = String(decoding: data, as: UTF8.self) + print(json) + } } } From f3a1912f25784beac352514f12887c281b3e9116 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 15:22:10 -0400 Subject: [PATCH 20/28] Productize Sources/skit/README.md for v0.0.5 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop the "research POC for issue #154" framing. Replace `skitrun` with `skit run`, env var prefixes with SKIT_*, and the release-script path with Scripts/build-skit-release.sh. The "Open scope decisions" section is gone — items either shipped (timeout) or have follow-up issues (#157 auto-rebuild, #158 if-in-Group), and the rest are explicitly out-of-scope. Point at Docs/skit.md for the deeper dive (forthcoming). Co-Authored-By: Claude Opus 4.7 (1M context) --- Sources/skit/README.md | 80 +++++++++++++++++++----------------------- 1 file changed, 37 insertions(+), 43 deletions(-) diff --git a/Sources/skit/README.md b/Sources/skit/README.md index 52f64fc..5b0f77a 100644 --- a/Sources/skit/README.md +++ b/Sources/skit/README.md @@ -1,22 +1,22 @@ -# skitrun +# skit -> **Status:** research POC for [issue #154](https://github.com/brightdigit/SyntaxKit/issues/154). Shape may change; do not pin tooling to it yet. Design lives at [`Docs/research/codegen-cli-design.md`](../../Docs/research/codegen-cli-design.md); the 7-step build-up is documented step-by-step under [`Docs/research/poc-step{1..7}-results.md`](../../Docs/research/). - -A CLI that takes a *pure SyntaxKit DSL* input file, wraps it in a `Group { … }` closure, spawns `swift` to evaluate it, and writes the rendered Swift source to stdout (or a file). No `print`, no `@main`, no boilerplate in your input — just DSL expressions. +A CLI for SyntaxKit. Two verbs: ``` -skitrun Input.swift # render to stdout -skitrun Input.swift -o Out.swift # render to a file -skitrun InputDir/ -o OutDir/ # walk **/*.swift, mirror to OutDir/ +skit run Input.swift # render a SyntaxKit DSL file into Swift source +skit run Input.swift -o Out.swift +skit run InputDir/ -o OutDir/ # walk **/*.swift and mirror rendered output +skit parse < Input.swift # parse Swift source into a JSON syntax tree ``` +`run` is the default subcommand, so `skit Input.swift` is shorthand for `skit run Input.swift`. + ## Quick start ```bash -# Build a portable bundle (the script flips the SyntaxKit library to -# .dynamic, then bundles dylib + modules + C-shims headers next to skitrun). -Docs/research/poc-step4-release.sh -# → .build/skitrun-release/{skitrun, lib/} +# Build a self-contained release bundle (binary + dylib + swiftmodules). +Scripts/build-skit-release.sh +# → .build/skit-release/{skit, lib/} cat > /tmp/Person.swift <<'SWIFT' Struct("Person") { @@ -25,14 +25,14 @@ Struct("Person") { } SWIFT -.build/skitrun-release/skitrun /tmp/Person.swift +.build/skit-release/skit /tmp/Person.swift ``` -The bundle is self-contained: `cp -r .build/skitrun-release ~/anywhere/` and `~/anywhere/skitrun-release/skitrun ` works zero-config. +The bundle is portable: `cp -r .build/skit-release ~/anywhere/` and `~/anywhere/skit-release/skit ` works zero-config. ## Input file shape -Top-level expressions form an implicit `@CodeBlockBuilder` body. `import` declarations at the top are hoisted into the wrapper. Anything else (`Struct(…)`, `Enum(…)`, helper calls, …) becomes the builder's content. +`skit run` wraps each input in an implicit `Group { … }` builder. Top-level expressions become the builder's content; `import` declarations at the top are hoisted into the wrapper. ```swift // Models.swift @@ -48,7 +48,7 @@ What *won't* work inside the input: top-level `let`/`var` outside the builder DS ## Helpers -Shared codegen utilities live in a `Helpers/` directory anywhere up-tree from the input. `skitrun` walks up from the input file (or directory) looking for one. Sources are pre-compiled into `libSyntaxKitHelpers.{dylib,so}` once and cached by content hash: +Shared codegen utilities live in a `Helpers/` directory anywhere up-tree from the input. `skit` walks up from the input file (or directory) looking for one. Sources are pre-compiled into `libSyntaxKitHelpers.{dylib,so}` once and cached by content hash: ``` project/ @@ -65,41 +65,35 @@ Force-disable: `--no-helpers`. Override location: `--helpers `. ## Caches -Two layers, both keyed on content + toolchain + dylib stamp + `SKITRUN_*`/`SYNTAXKIT_*` env vars. Live under `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS, `$XDG_CACHE_HOME/syntaxkit` (or `~/.cache/syntaxkit`) on Linux. +Two layers, both keyed on content + toolchain + dylib stamp + `SKIT_*`/`SYNTAXKIT_*` env vars. Live under `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS, `$XDG_CACHE_HOME/syntaxkit` (or `~/.cache/syntaxkit`) on Linux. -| Layer | Path | What it skips on hit | -| --- | --- | --- | -| Helpers | `helpers//` | the `swiftc` compile of `Helpers/*.swift` | -| Output | `outputs//output.swift` | the `swift` spawn for an input | +| Layer | Path | What it skips on hit | +| ------- | ----------------------------- | --------------------------------------------- | +| Helpers | `helpers//` | the `swiftc` compile of `Helpers/*.swift` | +| Output | `outputs//output.swift` | the `swift` spawn for an input | -Output cache hit ≈ 0.14s on macOS (no spawn at all); cold miss matches the warm `swift` script-mode baseline (~0.5s). Force a miss with `--no-cache`. +Output cache hit is roughly ~0.14s on macOS (no spawn at all); cold miss matches the warm `swift` script-mode baseline (~0.5s). Force a miss with `--no-cache`. -## Flag reference +## Flag reference (`skit run`) -| Flag | Default | Meaning | -| --- | --- | --- | -| `-o, --output ` | stdout | Output file (single-file mode) or directory (folder mode). | -| `--lib ` | auto | Directory containing `libSyntaxKit.{dylib,so}` + module files. Search order when omitted: `$SKITRUN_LIB_DIR` → `/lib/` → `/../lib/skitrun/`. | -| `--helpers ` | walk-up | Explicit `Helpers/` directory. | -| `--no-helpers` | (off) | Skip helpers discovery entirely. | -| `--no-cache` | (off) | Skip the output cache; always spawn `swift`. | -| `--timeout ` | `60` | Per-input timeout for the spawned `swift` (SIGTERM → 5s → SIGKILL). On expiry the file exits with code 124. Pass `0` to disable. | -| `--no-toolchain-check` | (off) | Skip the startup check that compares the bundle's recorded build toolchain (`lib/swift-version.txt`) to `swift --version`. swiftmodules aren't reliably compatible across compiler versions; on mismatch skitrun refuses to spawn `swift` and points at the rebuild script. Auto-rebuild fallback tracked in [#157](https://github.com/brightdigit/SyntaxKit/issues/157). | +| Flag | Default | Meaning | +| ----------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `-o, --output ` | stdout | Output file (single-file mode) or directory (folder mode). | +| `--lib ` | auto | Directory containing `libSyntaxKit.{dylib,so}` + module files. Search order when omitted: `$SKIT_LIB_DIR` → `/lib/` → `/../lib/skit/`. | +| `--helpers ` | walk-up | Explicit `Helpers/` directory. | +| `--no-helpers` | (off) | Skip helpers discovery entirely. | +| `--no-cache` | (off) | Skip the output cache; always spawn `swift`. | +| `--timeout ` | `60` | Per-input timeout for the spawned `swift` (SIGTERM → 5s → SIGKILL). On expiry the file exits with code 124. Pass `0` to disable. | +| `--no-toolchain-check` | (off) | Skip the startup check that compares `lib/swift-version.txt` to `swift --version`. swiftmodules aren't reliably compatible across compiler versions; on mismatch skit refuses to spawn `swift` and points at the rebuild script. Auto-rebuild fallback tracked in [#157](https://github.com/brightdigit/SyntaxKit/issues/157). | ## Platform notes -- **macOS:** primary target. All seven POC steps run via the scripts in `Docs/research/`. -- **Linux:** verified in `swift:6.0-jammy/aarch64` via [`Docs/research/poc-step7.sh`](../../Docs/research/poc-step7.sh) (self-reruns inside Docker). Requires `swift-crypto` instead of CryptoKit; install-name flag is Mach-O specific and skipped on Linux. -- **Windows:** not attempted. - -A known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited children on `swift:6.0-jammy/aarch64`. Workaround in `Helpers.swift` / `Main.swift`: `terminationHandler` + `DispatchSemaphore`. See [`poc-step7-results.md`](../../Docs/research/poc-step7-results.md) for the full reproducer. +- **macOS** — primary target. All build/release/test flows in `Scripts/`. +- **Linux** — verified on `swift:6.0-jammy/aarch64`. Requires `swift-crypto` instead of CryptoKit (we depend on it). The Mach-O `install_name` step in `Scripts/build-skit-release.sh` is macOS-specific and skipped on Linux. +- **Windows** — not supported. -## Open scope decisions +Known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited children on `swift:6.0-jammy/aarch64`. `Runner.swift` and `Helpers.swift` work around it with `terminationHandler` + `DispatchSemaphore`. -Not blocking but on the table — see [`codegen-cli-design.md` §7](../../Docs/research/codegen-cli-design.md#7-what-we-still-need-to-verify): +## Deeper dive -- Timeouts on the child `swift` process (60s default + SIGTERM/SIGKILL grace). -- `@main` / attribute behavior in `swift` script-mode beyond the simple cases tested. -- Multi-file outputs from a single input (out of scope for v1). -- Sandboxing (out of scope; threat model = "you ran your own code"). -- HTTP/server form for warm-interpreter reuse (post-CLI follow-up). +For the architecture, design decisions, and trade-offs see [`Docs/skit.md`](../../Docs/skit.md). From 05e8502e538bd2aca32c26a0833b73342d9e8a9f Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 15:27:25 -0400 Subject: [PATCH 21/28] Replace POC research log with Docs/skit.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The chronological poc-step{1..7}-results.md log and the codegen-cli-design.md sketch were the trail by which `skit` reached its current shape — useful at review time, noise for ongoing maintenance. Their substance is now folded into a single forward-looking explainer at Docs/skit.md (the architecture, caches, toolchain stamp, timeout, sharp edges, deferred items, references to #157/#158). Docs/research/tuist-manifest-pipeline.md stays — it's reference material for the manifest-pipeline pattern, not a step log. Git history retains the deleted files for anyone wanting to trace how a particular decision came about. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/codegen-cli-design.md | 160 --------------------- Docs/research/poc-step1-results.md | 138 ------------------ Docs/research/poc-step1.sh | 117 ---------------- Docs/research/poc-step2-results.md | 70 ---------- Docs/research/poc-step3-results.md | 48 ------- Docs/research/poc-step4-results.md | 55 -------- Docs/research/poc-step5-results.md | 58 -------- Docs/research/poc-step5.sh | 135 ------------------ Docs/research/poc-step6-results.md | 58 -------- Docs/research/poc-step6.sh | 116 --------------- Docs/research/poc-step7-results.md | 45 ------ Docs/research/poc-step7.sh | 135 ------------------ Docs/skit.md | 209 ++++++++++++++++++++++++++++ 13 files changed, 209 insertions(+), 1135 deletions(-) delete mode 100644 Docs/research/codegen-cli-design.md delete mode 100644 Docs/research/poc-step1-results.md delete mode 100755 Docs/research/poc-step1.sh delete mode 100644 Docs/research/poc-step2-results.md delete mode 100644 Docs/research/poc-step3-results.md delete mode 100644 Docs/research/poc-step4-results.md delete mode 100644 Docs/research/poc-step5-results.md delete mode 100755 Docs/research/poc-step5.sh delete mode 100644 Docs/research/poc-step6-results.md delete mode 100755 Docs/research/poc-step6.sh delete mode 100644 Docs/research/poc-step7-results.md delete mode 100755 Docs/research/poc-step7.sh create mode 100644 Docs/skit.md diff --git a/Docs/research/codegen-cli-design.md b/Docs/research/codegen-cli-design.md deleted file mode 100644 index 601b4e0..0000000 --- a/Docs/research/codegen-cli-design.md +++ /dev/null @@ -1,160 +0,0 @@ -# Design sketch: a CLI for SyntaxKit-driven codegen - -> Phase 2 deliverable for [issue #154](https://github.com/brightdigit/SyntaxKit/issues/154). Builds on [`tuist-manifest-pipeline.md`](./tuist-manifest-pipeline.md). Nothing here is implemented — this is the design we'd validate with the POC in §6. - -## 1. What we're borrowing from Tuist — and what we're not - -From Phase 1, Tuist's manifest pipeline reduces to four moving parts: - -1. A **public DSL framework** that ships next to the CLI binary (`ProjectDescription.framework`). -2. A **script runner** that invokes `xcrun swift ` with `-I/-L/-F` pointing at that framework, captures stdout, and slices out a token-delimited payload. -3. A **helpers compiler** that pre-builds `Tuist/ProjectDescriptionHelpers/*.swift` into a sibling dylib so manifests can `import ProjectDescriptionHelpers`. -4. A **two-tier cache** (helpers module + decoded manifest) keyed on source hashes + toolchain/tool versions. - -We borrow (1), (2), (3), and (4). We **don't** borrow Tuist's "manifest" framing — no `Project.swift`-style wrapper, no `Output(...)` value, no token-delimited stdout payload. Tuist needs the wrapper because its host has to re-interpret the description into an `xcodeproj`. SyntaxKit doesn't: the input file is *pure DSL* — a series of `CodeBlock` expressions — and the CLI generates the boilerplate that turns it into a runnable Swift program. - -## 2. CLI shape - -The CLI is `stdin → stdout`-shaped, with a SyntaxKit-aware `swift` invocation as the engine: - -``` -syntaxkit run Input.swift # rendered Swift source to stdout -syntaxkit run Input.swift -o Output.swift # write to a file (atomic) -syntaxkit run InputDir/ -o OutputDir/ # walk InputDir/*.swift, mirror paths into OutputDir/ -``` - -**Input file:** pure DSL. A series of `CodeBlock` expressions, optionally preceded by `import` declarations. No `print`, no `@main`, no boilerplate. Example: - -```swift -// Person.swift -import SyntaxKit // optional — only needed for IDE / autocomplete - -Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") -} -``` - -That's the entire file. Top-level expressions form an implicit `@CodeBlockBuilder` body that the CLI wraps for execution — see §3. - -**Output:** the rendered Swift source produced by `generateCode()`. The CLI does not reshape it. - -**Stderr:** forwarded to the user's terminal. The captured output is stdout-only. (The wrapper writes to stdout via a single `print`; if helpers or the user's DSL want to log debug info, they should use `FileHandle.standardError.write(...)`.) - -**Folder mode:** when the input is a directory, the CLI walks `**/*.swift` and produces a parallel tree of outputs (one input file → one output file, mirrored relative path). Files starting with `_` are skipped (convention for shared helpers — see §4). - -**Exit codes:** child `swift` non-zero → CLI non-zero with stderr preserved. No retries. - -## 3. Process model: wrap, then spawn - -Because the input is pure DSL rather than a runnable Swift program, the CLI does a tiny **wrap** step before spawning `swift`. - -**Wrap.** Read the input file. Use SwiftSyntax (already a dep — `Docs/SwiftSyntax-LLM.md`) to split the top of the file into (a) any leading `import` declarations and (b) the remaining body. Generate a temporary `Input.wrapped.swift`: - -```swift -import SyntaxKit - - -let __syntaxkit_root = Group { - -} -print(__syntaxkit_root.generateCode()) -``` - -`Group` (`Sources/SyntaxKit/Utilities/Group.swift`) already uses `@CodeBlockBuilder`, so its closure body accepts a series of `CodeBlock` expressions exactly the way the user wrote them. `import SyntaxKit` is always injected; duplicates from the input are harmless. - -**Spawn.** Adopt Tuist's model `(b)` from Phase 1 §2 — `swift` in script mode against the wrapped file, pipe through: - -``` -/usr/bin/env swift \ - -suppress-warnings \ - -I \ - -L \ - -F \ - -lSyntaxKit -framework SyntaxKit \ - -Xcc -I -Xcc \ - -Xlinker -rpath -Xlinker \ - -I … -L … -F … -l # optional, when helpers exist - /Input.wrapped.swift -``` - -The `-Xcc -I -Xcc <…>` flag is non-obvious but required (POC step 1 finding): SyntaxKit transitively depends on `_SwiftSyntaxCShims` whose module map lives in `swift-syntax/Sources/_SwiftSyntaxCShims/include/`. The bundled-binary release must ship this header directory alongside the dylib and the SwiftSyntax `.swiftmodule` files. Without the flag, the script compile fails with `missing required module '_SwiftSyntaxCShims'`. The `-Xlinker -rpath -Xlinker <…>` flag tells the just-built script's dylib loader where to find `libSyntaxKit.dylib` at runtime. - -No `--syntaxkit-dump` flag, no start/end tokens. The child's stdout is the rendered Swift source verbatim. The CLI's job is wrap → `Process` → capture stdout → atomic write to destination → clean up temp wrapper. - -Use `/usr/bin/env swift` rather than `/usr/bin/xcrun swift` so the same code path runs on Linux. `xcrun` is mac-only and is implicit when `env swift` resolves to Xcode's swift on macOS. - -**Why wrap instead of requiring `print()` in the input.** Two reasons. First, the user's authoring surface is *just* DSL — declarative, no I/O verbs. Second, error reporting: when the child `swift` reports a diagnostic at `Input.wrapped.swift:42`, the CLI can map that line back to the original `Input.swift` (the wrapper is line-faithful aside from a known prefix offset) and rewrite the path in stderr before forwarding. - -## 4. Helpers - -Same mechanism as Tuist (Phase 1 §4), folder name TBD — `Helpers/` adjacent to the input file or input directory is the obvious choice. The CLI walks up from the input path looking for a `Helpers/` directory, globs `**/*.swift` (excluding files prefixed with `_` to allow private helpers within helpers), and pre-compiles them into `lib.dylib` via: - -``` -swiftc -module-name SyntaxKitHelpers \ - -emit-module -emit-module-path /SyntaxKitHelpers.swiftmodule \ - -parse-as-library -emit-library \ - -suppress-warnings \ - -I … -L … -F … -lSyntaxKit -framework SyntaxKit \ - Helpers/**/*.swift -``` - -The output dylib is then added to the input script's invocation via `-I/-L/-F/-l`. Scripts can `import SyntaxKitHelpers` and use shared codegen utilities. - -Compile into a `tmp..` staging directory and atomic-rename into the cache path, mirroring `ProjectDescriptionHelpersBuilder.swift:204-244` — concurrent CLI invocations need to be safe. - -## 5. Caching - -Two layers, both mirroring Tuist (Phase 1 §5). - -**Helpers cache.** Keyed by: - -| Field | Source | -| --- | --- | -| per-file SHA-256s | `Helpers/**/*.swift`, sorted | -| `syntaxkitVersion` | bundled SyntaxKit dylib version | -| `swiftlangVersion` | `swift --version` | -| `osVersion` | `uname -r` (macOS or Linux) | -| `cacheSchemaVersion` | bumped on layout changes | - -Hash → directory name → reuse if present. - -**Output cache.** Skip the swift spawn entirely when nothing has changed. Keyed by: - -| Field | Source | -| --- | --- | -| `inputHash` | SHA-256 of the input `.swift` file | -| `helpersHash` | the helpers cache key above, or empty | -| `syntaxkitVersion` | bundled SyntaxKit dylib version | -| `swiftlangVersion` | `swift --version` | -| `envHash` | md5 of `SYNTAXKIT_*` env vars | -| `cacheSchemaVersion` | bumped on layout changes | - -On hit, copy the cached rendered output directly to the destination — no `swift` spawn. On miss, run and re-cache. - -Cache location: `~/.cache/syntaxkit/` on Linux, `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS (XDG-aware via `XDG_CACHE_HOME`). - -## 6. Smallest possible proof-of-concept steps - -Each step is independently shippable and de-risks the next. - -1. **Hand-driven wrap + spawn.** Hand-write a pure-DSL `Input.swift` and a hand-rolled `Input.wrapped.swift` that imports SyntaxKit, splices the body into `Group { … }`, and prints `generateCode()`. Invoke it manually with `swift Input.wrapped.swift -I … -L … -F … -lSyntaxKit -framework SyntaxKit` against a local `swift build` of SyntaxKit. **Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit at all, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost.** Cold-start is the single biggest risk to retire — if it's 20+ seconds, the design needs rethinking. -2. **CLI subcommand for single-file mode.** Add `syntaxkit run `: parse out top-level imports with SwiftSyntax, write `Input.wrapped.swift` to a temp dir, spawn `swift` via `Foundation.Process`, capture stdout, write to `-o ` (or stdout if no `-o`). Stderr is forwarded; rewrite `Input.wrapped.swift:LINE` references back to `Input.swift:LINE` before forwarding. -3. **Folder mode.** Walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`. Add `_`-prefix skip rule. Parallelize cautiously (`swift` spawns aren't free — gate on a small concurrency limit, maybe `ProcessInfo.activeProcessorCount`). -4. **Ship a bundled-binary release.** Build SyntaxKit as a `type: .dynamic` library under `-c release`, then `strip -x` the resulting dylib. Bundle the stripped `libSyntaxKit.dylib` + every `.swiftmodule` SyntaxKit publicly re-exports (SwiftSyntax + version-suffixed variants, SwiftOperators, SwiftParser) + the `_SwiftSyntaxCShims/include/` header dir in a `lib/` directory next to the CLI binary. Write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). POC step 1 confirmed cold-start with this layout is ~720ms; stripped release dylib is **9.3 MB** on Apple Silicon (down from 25 MB debug / 18 MB unstripped release). -5. **Helpers directory.** Discovery + compile + flag-splicing. -6. **Output cache.** Add the cache, keyed as in §5. Add `--no-cache` for debugging. -7. **Linux smoke test.** Confirm `/usr/bin/env swift` works on Linux with the bundled dylib layout (no framework search path, but `-I + -L + -lSyntaxKit` should be sufficient, matching Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch). - -## 7. What we still need to verify - -- **Cold-start cost.** ~~Single biggest unknown.~~ Answered by POC step 1: ~720ms cold, ~110ms warm. See [`poc-step1-results.md`](./poc-step1-results.md). -- **SyntaxKit `if`-in-`Group` compiler crash.** POC step 1 surfaced this: `CodeBlockBuilderResult` claims `buildEither`/`buildOptional` support but conditionals trigger a type-checker failure-to-diagnose. Independent of the CLI design but blocks users writing conditional codegen. File as a separate SyntaxKit bug. -- **Splice fidelity.** When the input body lives inside a `Group { … }` closure, is everything users naturally write in the DSL still legal? Result-builder closures don't allow `import`, top-level type decls, or top-level `let`/`var` outside the builder DSL. The wrapper hoists `import`s; we need to confirm there's no other top-level construct users would reasonably write that the wrap step would break. Verify in step 1 with a few realistic inputs (large struct, nested types, conditionals via `if`-in-builder). -- **`Process` stdout/stderr separation.** Tuist captures stdout-only and merges stderr via `CommandError`. Foundation's `Process` has the same split — confirm it doesn't interleave under load, and confirm the CLI doesn't accidentally swallow stderr. -- **`swift` script-mode quirks.** `swift ` runs in interpret/`-frontend -interpret` mode. Some features (`@main`, certain attributes) behave differently than in compile mode. Top-level `print` statements are fine. Verify in step 1. -- **SwiftSyntax linkage stability across toolchains.** SwiftSyntax pins to specific Swift toolchain versions. The bundled dylib is built against a particular toolchain; if the user's `swift` is from a newer or older release, ABI breakage is possible. Mitigation: cache key includes `swiftlangVersion`, plus a clear error when the gap is too wide. -- **What if a single input script needs to produce multiple files?** Out of scope for v1. Split into multiple inputs and use folder mode. If demand materializes, we can layer in a `--multi` envelope mode (a script writes a small JSON manifest to stdout, CLI fans out to multiple files) without breaking single-file semantics. -- **Sandboxing.** Out of scope for v1. Input scripts are user-owned code in their own repo — running them has the same threat model as running `swift Input.swift` by hand. Revisit if/when this CLI runs untrusted scripts (CI for OSS contributions, etc.). -- **Timeout.** Add one. Tuist's omission (Phase 1 §7) is a bug, not a feature. 60s default `Process` wait with `SIGTERM` → 5s grace → `SIGKILL`. Override via `--timeout `. -- **Web-server form.** Out of scope for the CLI POC, but on the table as a follow-up once the 7-step ladder is done. A long-lived server could reuse a warm `swift` interpreter across requests and share the helpers + output caches across tenants — both of which the CLI gives up on every invocation. Open design questions: request shape (raw DSL POST vs. structured), whether helpers are uploaded per-request or baked into the server image, and isolation between requests (the CLI's "run user code in your own repo" threat model doesn't transfer). Revisit after step 7. diff --git a/Docs/research/poc-step1-results.md b/Docs/research/poc-step1-results.md deleted file mode 100644 index 1940027..0000000 --- a/Docs/research/poc-step1-results.md +++ /dev/null @@ -1,138 +0,0 @@ -# POC Step 1 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 1. Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost. - -## TL;DR — Approach is viable - -- **Cold start: ~720ms** real wall-clock for `xcrun swift Input.wrapped.swift -lSyntaxKit …` on M-series macOS with the dylib + module files unbacked by the OS page cache. -- **Warm: ~110ms** for subsequent runs. -- **Pure-DSL `Input.swift` spliced into `Group { … }`** in a generated wrapper compiles and runs end-to-end, producing the expected Swift source. -- **Bundled-dylib distribution is real and viable.** The only flags the CLI has to assemble are `-I`, `-L`, `-lSyntaxKit`, `-Xlinker -rpath -Xlinker ` and one new requirement: `-Xcc -I -Xcc ` (see §3 finding). -- **SyntaxKit dylib weight:** 25 MB debug → 18 MB release → **9.3 MB stripped release**. The 9.3 MB number is the one that matters for distribution and is well within range of a normal CLI binary. - -## 1. What was run - -Built the dylib by temporarily flipping the SyntaxKit library product to `type: .dynamic` in `Package.swift`, ran `swift build`, copied the artifacts into a `/tmp/syntaxkit-poc/lib/` staging dir, then reverted the package manifest. - -Wrote a pure-DSL `Input.swift`: - -```swift -import SyntaxKit // optional; only for IDE - -Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") -} - -Struct("Pet") { - Variable(.let, name: "kind", type: "String") -} -``` - -And a hand-rolled `Input.wrapped.swift`: - -```swift -import SyntaxKit - -let __syntaxkit_root = Group { - Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") - } - Struct("Pet") { - Variable(.let, name: "kind", type: "String") - } -} - -print(__syntaxkit_root.generateCode()) -``` - -Invoked with: - -``` -xcrun swift \ - -I lib -L lib -lSyntaxKit \ - -Xcc -I -Xcc lib/_SwiftSyntaxCShims-include \ - -Xlinker -rpath -Xlinker $(pwd)/lib \ - Input.wrapped.swift -``` - -Output (verbatim): - -``` -struct Person { -let name : String -let age : Int - -} -struct Pet { -let kind : String - -} -``` - -(Whitespace artifacts are SyntaxKit's `generateCode()` output as-is — out of scope for this POC.) - -## 2. Timings - -Three back-to-back runs after a cold first run: - -| Run | real | user | sys | -| --- | ---: | ---: | ---: | -| cold (first) | 0.72s | 0.77s | 0.29s | -| warm 1 | 0.14s | 0.08s | 0.04s | -| warm 2 | 0.11s | 0.07s | 0.02s | -| warm 3 | 0.11s | 0.07s | 0.02s | - -Hardware: Apple Silicon mac. Cold start is dominated by loading SyntaxKit + SwiftSyntax dylibs from disk; once cached, the swift interpreter just compiles a tiny script. For a per-file CLI invocation this is well inside the "feels instant" budget. - -## 3. New design finding: C-shim include path - -Without `-Xcc -I -Xcc <_SwiftSyntaxCShims/include>`, the script compile fails with: - -``` -:0: error: missing required module '_SwiftSyntaxCShims' -``` - -SyntaxKit transitively depends on SwiftSyntax which has a C-shims target whose module map lives at `swift-syntax/Sources/_SwiftSyntaxCShims/include/module.modulemap`. The CLI's bundled-binary distribution layout (§5 of the design doc) must include this header directory, not just the `.dylib` and `.swiftmodule` files. Updated the design doc to reflect this. - -## 4. New design finding: `if` inside `Group` is broken in SyntaxKit today - -A wrapped input containing a conditional in the builder: - -```swift -let __syntaxkit_root = Group { - if true { - Struct("A") { Variable(.let, name: "x", type: "Int") } - } -} -``` - -fails with: - -``` -error: failed to produce diagnostic for expression; please submit a bug report -let __syntaxkit_root = Group { - ^ -``` - -`CodeBlockBuilderResult` declares both `buildEither(first:)` / `buildEither(second:)` and `buildOptional` (`Sources/SyntaxKit/CodeBlocks/CodeBlockBuilderResult.swift:46-58`), so the API surface *says* `if`/`else` is supported. The compiler crash is a Swift type-checker timeout — likely from the `any CodeBlock...` variadic overload combined with `buildEither` overload resolution. No test in `Tests/SyntaxKitTests/Unit/` exercises an `if` inside `Group { … }`, which is why this hasn't been caught. - -**Implication for the CLI design:** non-blocking. v1 can document "no conditionals in input files yet" and ship; the underlying SyntaxKit fix is independent. Worth filing as a separate issue. - -## 5. Confirmed: hoisted imports work - -A wrapped input with both `import SyntaxKit` and `import Foundation` at the top compiles fine and `UUID`/`Date` resolve in the rendered struct fields. The CLI's hoist-imports step (design §3) is safe. - -## 6. Not yet retired - -- **Stderr/stdout interleaving under load.** Need a load test (large input → tons of output → confirm captured stdout is intact and stderr doesn't bleed in). -- **Linux behavior.** Step 7 of the POC ladder. Same `swift -I -L -l` flag set should work; framework-search paths (`-F`) become a no-op. -- **`@main` and other top-level forms.** Not relevant for the wrapper because we always control the wrapper's shape, but if users ever paste a class/extension declaration directly into `Input.swift` we need to reject it cleanly. - -## 7. Updates to the design doc to make from this POC - -- §5 distribution layout: add `Sources/_SwiftSyntaxCShims/include/` to the bundled `lib/` contents. -- §3 spawn command: include the `-Xcc -I -Xcc <…>` flag. -- §7 open questions: SyntaxKit dylib size measured at 25 MB debug, 18 MB release, 9.3 MB release+stripped (`strip -x`). Warm performance identical between debug and release builds. -- §7 open questions: add tracking note for the `if`-in-`Group` Swift compiler bug. diff --git a/Docs/research/poc-step1.sh b/Docs/research/poc-step1.sh deleted file mode 100755 index fe41e3b..0000000 --- a/Docs/research/poc-step1.sh +++ /dev/null @@ -1,117 +0,0 @@ -#!/usr/bin/env bash -# -# POC step 1 reproducer for issue #154. -# Run from anywhere; resolves the repo root from its own location. -# -# What it does: -# 1. Backs up Package.swift, flips the SyntaxKit library to type: .dynamic. -# 2. swift build (produces libSyntaxKit.dylib). -# 3. Stages dylib + swiftmodules + _SwiftSyntaxCShims headers into /tmp/syntaxkit-poc/lib/. -# 4. Writes a pure-DSL Input.swift and a hand-rolled Input.wrapped.swift. -# 5. Runs the wrapped script once cold + three times warm, printing timings. -# 6. Restores Package.swift on exit (even on failure). - -set -euo pipefail - -if [[ "$(uname -s)" != "Darwin" ]]; then - echo "This reproducer is macOS-only. Linux smoke test is POC step 7." >&2 - exit 1 -fi - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -POC_DIR="/tmp/syntaxkit-poc" -PACKAGE_FILE="$REPO_ROOT/Package.swift" -PACKAGE_BACKUP="$(mktemp)" - -cleanup() { - if [[ -s "$PACKAGE_BACKUP" ]]; then - cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" - fi - rm -f "$PACKAGE_BACKUP" -} -trap cleanup EXIT - -cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" - -echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" -python3 - "$PACKAGE_FILE" <<'PY' -import sys, pathlib -p = pathlib.Path(sys.argv[1]) -src = p.read_text() -old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' -new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' -if old not in src: - sys.exit("Package.swift: expected SyntaxKit library product block not found — has Package.swift changed shape?") -p.write_text(src.replace(old, new, 1)) -PY - -cd "$REPO_ROOT" -echo "==> swift build" -swift build - -BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1 || true)" -if [[ -z "$BUILD_DIR" ]]; then - echo "Could not locate .build//debug" >&2 - exit 1 -fi - -echo "==> Staging $POC_DIR/lib/" -rm -rf "$POC_DIR" -mkdir -p "$POC_DIR/lib" -cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" -cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" -cp -r ".build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ - "$POC_DIR/lib/_SwiftSyntaxCShims-include" - -cat > "$POC_DIR/Input.swift" <<'SWIFT' -// Pure-DSL input. No print, no @main, no boilerplate. -import SyntaxKit // optional; only for IDE autocomplete - -Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") -} - -Struct("Pet") { - Variable(.let, name: "kind", type: "String") -} -SWIFT - -cat > "$POC_DIR/Input.wrapped.swift" <<'SWIFT' -import SyntaxKit - -let __syntaxkit_root = Group { - Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") - } - Struct("Pet") { - Variable(.let, name: "kind", type: "String") - } -} - -print(__syntaxkit_root.generateCode()) -SWIFT - -cd "$POC_DIR" - -SWIFT_ARGS=( - -I lib -L lib -lSyntaxKit - -Xcc -I -Xcc lib/_SwiftSyntaxCShims-include - -Xlinker -rpath -Xlinker "$POC_DIR/lib" - Input.wrapped.swift -) - -echo -echo "==> Cold run (full output + timing):" -/usr/bin/time -p xcrun swift "${SWIFT_ARGS[@]}" - -echo -echo "==> Warm runs (timings only, output discarded):" -for _ in 1 2 3; do - /usr/bin/time -p xcrun swift "${SWIFT_ARGS[@]}" >/dev/null -done - -echo -echo "==> Done. Staging dir kept at $POC_DIR for further poking." diff --git a/Docs/research/poc-step2-results.md b/Docs/research/poc-step2-results.md deleted file mode 100644 index 5bd6cd2..0000000 --- a/Docs/research/poc-step2-results.md +++ /dev/null @@ -1,70 +0,0 @@ -# POC Step 2 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 2. Goal: wrap the hand-driven flow from step 1 inside an actual CLI executable that parses imports with SwiftSyntax, generates the wrapper, spawns `swift`, and forwards output/errors. - -## What landed - -New executable target `skitrun` (POC name; final CLI name TBD) at `Sources/skitrun/Main.swift`. Single file, ~180 lines. Depends only on `SwiftSyntax` + `SwiftParser` — does **not** depend on SyntaxKit, since the host doesn't render anything itself. - -## Usage - -``` -skitrun [-o ] [--lib ] -``` - -`--lib` defaults to `/tmp/syntaxkit-poc/lib` so it works against the artifacts produced by [`poc-step1.sh`](./poc-step1.sh) without further flags. - -Build it once: `swift build --product skitrun`. The binary lands at `.build//debug/skitrun`. - -## Verified flows - -1. **Default (stdout):** `skitrun Input.swift` prints rendered Swift to stdout. -2. **File output:** `skitrun Input.swift -o Out.swift` writes the file atomically. -3. **Hoisted imports:** an input with `import Foundation` at the top compiles cleanly; `UUID`/`Date` resolve in the rendered struct. -4. **Compiler diagnostics map to the input file.** A deliberate `type: NonexistentType` in `InputError.swift:4` produces: - ``` - /tmp/syntaxkit-poc/InputError.swift:4:37: error: cannot find 'NonexistentType' in scope - ``` - The path and line are correct — confirming `#sourceLocation` is doing the work end-to-end. - -## How the wrap works - -`SwiftParser.Parser.parse(source:)` produces a `SourceFileSyntax`. We walk `tree.statements`: - -- Every leading `ImportDeclSyntax` is collected for hoisting. -- The first non-import statement marks the start of the body. -- Everything from that byte offset forward is the body, copied verbatim. - -The wrapper is then: - -```swift -import SyntaxKit - - -let __skitrun_root = Group { -#sourceLocation(file: "", line: ) - -#sourceLocation() -} - -print(__skitrun_root.generateCode()) -``` - -`#sourceLocation` is what gives us diagnostic fidelity for free — the Swift compiler honors it and rewrites file/line in every error/warning emitted from the body range. No manual stderr line-number arithmetic needed. - -## Spawn shape - -`Foundation.Process` invoking `/usr/bin/env swift` with the exact flag set from POC step 1, captured into `stdoutPipe` and `stderrPipe`. Stdout is written verbatim to the output destination. Stderr is forwarded after one fix-up: any remaining literal `//skitrun-/Input.wrapped.swift` references (those outside the `#sourceLocation` range — i.e. errors in the preamble itself) get rewritten to the input path. - -## Surface limits worth knowing - -- **Snippet gutter line numbers in diagnostics show wrapper line numbers, not input line numbers.** The compiler maps the *file/line* in the diagnostic header via `#sourceLocation` but shows the surrounding source snippet from the actual file with its actual line numbers. The path and starting line are correct (navigable), but the gutter `7 |` / `8 |` markers may not match the input's line numbering. Cosmetic; doesn't affect navigation. -- **No timeout yet.** The design calls for a 60s default. Adding `Process.terminate(after:)` is a step 6 (cache) sibling concern. -- **Stdin / stderr interleaving under load not tested.** Step 7 territory. -- **`if`-in-`Group` still crashes the compiler** ([#155](https://github.com/brightdigit/SyntaxKit/issues/155)). Independent SyntaxKit bug; `skitrun` would happily pass such an input through, but the spawned `swift` would fail with the same opaque diagnostic from step 1. - -## What's next - -The natural step 3 is folder mode (walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`). All the per-file work is already in place — folder mode is just iteration + concurrency-limited fan-out + a `_`-prefix skip rule. Modest engineering, low risk. - -After that, step 4 (bundled-binary release) is where the design hits its biggest remaining systems-integration question: how to actually ship the `lib/` directory next to the binary across SwiftPM build, install, and `brew` distribution. diff --git a/Docs/research/poc-step3-results.md b/Docs/research/poc-step3-results.md deleted file mode 100644 index be524bf..0000000 --- a/Docs/research/poc-step3-results.md +++ /dev/null @@ -1,48 +0,0 @@ -# POC Step 3 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 3. Goal: extend `skitrun` to walk a directory of `.swift` inputs, mirror the relative paths into an output directory, and run the per-file work concurrently with a sane cap. - -## What changed - -`skitrun` now accepts a directory as input: - -``` -skitrun InputDir/ -o OutDir/ -``` - -When the input is a directory, the existing single-file work is hoisted into a `processFile(inputPath:libPath:)` helper. The new `runDirectory(...)` driver walks the input with `FileManager.enumerator`, fan-outs the per-file work over `withTaskGroup`, and writes successes into mirrored paths under `OutDir/`. Single-file mode is unchanged. - -Three small conventions ride along: - -- **`_`-prefix skip rule.** `_Helpers.swift`, `_Shared.swift`, etc. are not processed. (Confirmed against `_HelperShouldBeSkipped.swift` containing deliberately-invalid Swift — skitrun didn't try to compile it.) -- **`activeProcessorCount` concurrency cap.** The task group keeps that many in-flight `swift` spawns at a time, draining + refilling as each finishes. -- **Tuist-analog partial semantics.** Successful files are *always* written, even when other files in the same batch fail. The CLI exits non-zero if any failed and prints a `skitrun: N/M succeeded` summary to stderr. - -## Verified flows - -1. **Happy path.** A `codegen/` tree with `Models/Person.swift`, `Models/Pet.swift`, `Audit/Snapshot.swift` (the last with a hoisted `import Foundation`) produces a mirrored `out/Models/{Person,Pet}.swift` + `out/Audit/Snapshot.swift`. Total wall time 1.41s for 3 files (vs. 0.72s baseline cold-start for one). -2. **Skip rule.** `codegen/_HelperShouldBeSkipped.swift` contains the literal line `this is not valid swift`. It is not visited, the rest of the tree processes cleanly. -3. **Partial failure.** Adding a `Models/Bad.swift` with `type: TypeThatDoesNotExist` produces: - ``` - ---- /tmp/skitrun-folder-test/codegen/Models/Bad.swift ---- - /tmp/skitrun-folder-test/codegen/Models/Bad.swift:4:37: error: cannot find 'TypeThatDoesNotExist' in scope - … - skitrun: 3/4 succeeded - ``` - Exit code 1. Person/Pet/Snapshot still written. -4. **Single-file regression.** Both `skitrun Input.swift` (stdout) and `skitrun Input.swift -o Out.swift` (file) still work after the refactor. - -## Parallelism observations - -A quick timing on 3 parallel files vs. 1 cold-start baseline: - -| | wall time | -| --- | ---: | -| 1 file, cold | 0.72s | -| 3 files, cold, `withTaskGroup` cap = `activeProcessorCount` | 1.41s | - -That's well below 3×0.72 = 2.16s, confirming the parallelism is buying something — but also clearly slower than 3×0.11 = 0.33s warm, meaning successive `swift` invocations don't fully share OS file-cache benefits within a single batch run. (Each spawn still pays its own compile cost; the dylib pages are warm after the first, but compile work isn't deduplicated.) For larger batches we'd want to measure where the curve goes — and eventually pull the work into a single long-lived `swift` process driving all inputs, to skip the per-file compile overhead entirely. Out of scope for v1. - -## What's next - -Step 4 — bundled-binary release. The first real systems-integration challenge: how the `lib/` directory ships next to the `skitrun` binary across `swift build`, `swift run`, and `brew install`. Today users have to run `Docs/research/poc-step1.sh` to stage `/tmp/syntaxkit-poc/lib/`; that has to become "user installs the CLI, it just works." diff --git a/Docs/research/poc-step4-results.md b/Docs/research/poc-step4-results.md deleted file mode 100644 index c5248ac..0000000 --- a/Docs/research/poc-step4-results.md +++ /dev/null @@ -1,55 +0,0 @@ -# POC Step 4 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 4. Goal: produce a self-contained `skitrun` release bundle so users don't need a SyntaxKit checkout or `Docs/research/poc-step1.sh`. The binary finds its own `lib/` directory. - -## What landed - -1. **`resolveLibPath(override:)` in `Sources/skitrun/Main.swift`.** Search order: - 1. `--lib ` flag - 2. `$SKITRUN_LIB_DIR` env var - 3. `/lib/` — same-directory layout (the release bundle ships this way) - 4. `/../lib/skitrun/` — Homebrew layout (`bin/skitrun` ↔ `lib/skitrun/`) - - When none match, the CLI errors with a message enumerating all four paths and pointing at this script. The `/tmp/syntaxkit-poc/lib` fallback is gone. - -2. **`Docs/research/poc-step4-release.sh`.** Builds a self-contained bundle: - ``` - .build/skitrun-release/ - skitrun ← the CLI binary - lib/ - libSyntaxKit.dylib ← release + strip -x - *.swiftmodule ← SyntaxKit + transitively re-exported modules - _SwiftSyntaxCShims-include/ ← C-shims headers - ``` - Same trap-based Package.swift backup/restore as `poc-step1.sh`. `install_name_tool -id @rpath/libSyntaxKit.dylib` ensures the dylib install name is portable. - -## Verified flows - -Built bundle → copied to three unrelated locations → all worked with no flags, no env vars, no SyntaxKit checkout: - -1. **Same-directory layout.** `cp -r .build/skitrun-release /tmp/portable && /tmp/portable/skitrun Input.swift` → correct output. -2. **Homebrew layout.** `bin/skitrun + lib/skitrun/` arrangement → correct output. -3. **Error case.** `skitrun` alone in `/tmp/lonely/` (no lib anywhere) → clear diagnostic: - ``` - Could not locate SyntaxKit lib directory. Looked for: - 1. --lib (not provided) - 2. $SKITRUN_LIB_DIR (not set) - 3. /lib/ (not found) - 4. /../lib/skitrun/ (not found) - ``` -4. **Folder mode** from the portable bundle works end-to-end with partial-failure semantics intact. - -## Bundle weight - -| Component | Size | -| --- | ---: | -| `skitrun` (binary) | 17 MB | -| `libSyntaxKit.dylib` (release + stripped) | 9.3 MB | -| `lib/*` (modules + headers + dylib) | ~28 MB | -| **Total bundle** | **45 MB** | - -The 17 MB binary is unexpectedly heavy: it links SwiftSyntax statically because `skitrun` uses SwiftSyntax directly for parsing input files. So SwiftSyntax ships **twice** — once statically inside `skitrun`, once dynamically as part of the SyntaxKit dylib stack. Worth a follow-up: make `skitrun` itself dlopen SyntaxKit / share a dynamic SwiftSyntax with the dylib path. For v1 this is acceptable but is the largest single thing standing between the CLI and a "feels small" download. - -## What's next - -Step 5: helpers directory. Today users can `import Foundation` and `import SyntaxKit` from input files; step 5 lets them factor reusable codegen into a `Helpers/` directory that gets pre-compiled into `lib.dylib` and made importable from inputs. Modest engineering, but the first time `skitrun` itself invokes `swiftc` rather than `swift` (the helpers compile, distinct from the input run). See [`codegen-cli-design.md` §4](./codegen-cli-design.md#4-helpers) for the shape. diff --git a/Docs/research/poc-step5-results.md b/Docs/research/poc-step5-results.md deleted file mode 100644 index 746570e..0000000 --- a/Docs/research/poc-step5-results.md +++ /dev/null @@ -1,58 +0,0 @@ -# POC Step 5 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §4 + §6 step 5. Goal: let inputs `import SyntaxKitHelpers` from a `Helpers/` directory adjacent to the input, with discovery + on-demand `swiftc` compile + a per-toolchain cache. - -## What landed - -1. **`Sources/skitrun/Helpers.swift`.** New module with three responsibilities: - - `discoverHelpersDir(near:)` walks up from the input path looking for a `Helpers/` directory. Single-file mode starts from the file's parent; folder mode starts from the input directory. Walk-up stops at the filesystem root. - - `collectHelperSources(in:)` globs `**/*.swift` under the helpers dir, skipping `_`-prefixed files (same convention as input enumeration). - - `buildHelpers(helpersDir:libPath:)` hashes the sources + Swift version + dylib stamp into a cache key under `~/Library/Caches/com.brightdigit.SyntaxKit/helpers//`. On a cache miss, it shells out to `swiftc` into a `tmp../` staging dir, then atomic-renames into the cache path (`ProjectDescriptionHelpersBuilder` pattern from Tuist). - -2. **`Sources/skitrun/Main.swift` wiring.** - - `CLIArgs` gains `--helpers ` (explicit override) and `--no-helpers` (skip discovery). Default is auto. - - `runSwift` splices `-I/-L/-lSyntaxKitHelpers -Xlinker -rpath -Xlinker ` when helpers are present. - - Folder mode's `collectInputs` now also yields directories so it can call `enumerator.skipDescendants()` when it hits a `Helpers/` directly under the input root — otherwise the helpers would be re-processed as inputs. - - Helpers compile happens **once per invocation** in folder mode (not per input file). - -3. **`Docs/research/poc-step5.sh`.** Standalone demo: builds skitrun, stages a runtime lib, writes a tiny `Helpers/Models.swift` exporting `equatableModel(_:fields:)`, and two `inputs/*.swift` files that `import SyntaxKitHelpers` and call the helper. - -## Verified flows - -| Flow | Result | -| --- | --- | -| Cold run (cache cleared, helpers compile from scratch) | ✓ 2.96s real | -| Warm run (cache hit, same helper sources) | ✓ 0.54s real | -| Folder mode against `demo/` containing `Helpers/` + `inputs/` | ✓ 2/2 succeeded, `Helpers/*.swift` not enumerated as input | -| `--no-helpers` with an input that imports `SyntaxKitHelpers` | ✓ child `swift` errors with `no such module 'SyntaxKitHelpers'`, exit non-zero | - -The cached layout for a single helper file: - -``` -~/Library/Caches/com.brightdigit.SyntaxKit/helpers// - libSyntaxKitHelpers.dylib - SyntaxKitHelpers.swiftmodule - SyntaxKitHelpers.swiftdoc - SyntaxKitHelpers.abi.json - SyntaxKitHelpers.swiftsourceinfo -``` - -## Cache key - -SHA-256 over (in order): -- Cache schema version string (`v1`). -- For each helper source (sorted by absolute path): `lastPathComponent` + file bytes. -- `swift --version` output. -- `libSyntaxKit.dylib` size and modification time (proxy for SyntaxKit version until the bundle is versioned). - -Mutating any helper source, switching toolchains, or rebuilding SyntaxKit invalidates the cache. Adding a `cacheSchemaVersion` bump constant covers future layout changes. - -## Known rough edges - -- **Helpers cold compile is the dominant cost.** 2.96s vs 0.54s warm — the helper compile is ~2.5s on top of the ~0.5s `swift` interpret cost. Once cached it's free, but the first run after a clean checkout is noticeably slow. Acceptable for v1; could be sped up by caching the helpers `.o` files separately, but that's a step-6+ optimization. -- **Walk-up false positives.** If a user happens to have an unrelated `Helpers/` somewhere up-tree (e.g. a sibling library), skitrun will try to compile it. `--helpers ` or `--no-helpers` is the escape hatch. A future heuristic could require a sentinel file (`Helpers/.syntaxkit-helpers`) before claiming the directory. -- **Import-line diagnostics off by one.** When the user's input has `import Foo` on line 1, the wrap step puts an injected `import SyntaxKit` above it, so a child-compile error on the user's import reports `:2:8` instead of `:1:8`. `#sourceLocation` directives only wrap the body, not the hoisted imports. Easy follow-up: emit a `#sourceLocation` directive per hoisted import too. - -## What's next - -Step 6: **output cache.** Today every `skitrun` invocation re-spawns `swift` to render the input, even when nothing has changed. Add the per-input output cache from [`codegen-cli-design.md` §5](./codegen-cli-design.md#5-caching), keyed by input hash + helpers-cache key + Swift version + envHash. On a hit, skip the spawn entirely and copy the rendered output to the destination. Add `--no-cache` for debugging. diff --git a/Docs/research/poc-step5.sh b/Docs/research/poc-step5.sh deleted file mode 100755 index ea633a3..0000000 --- a/Docs/research/poc-step5.sh +++ /dev/null @@ -1,135 +0,0 @@ -#!/usr/bin/env bash -# -# POC step 5 demo: Helpers/ discovery + compile + import in input scripts. -# -# Builds skitrun, stages a runtime lib/ next to it, then runs skitrun against -# a demo project that uses `import SyntaxKitHelpers`. Demonstrates: -# 1. Cold path — Helpers/ compiles to libSyntaxKitHelpers.dylib. -# 2. Warm path — second invocation reuses the cached helpers dylib. -# 3. Folder mode — skitrun ignores Helpers/ when walking the input tree. -# 4. --no-helpers — disables discovery; the import then fails as expected. - -set -euo pipefail - -if [[ "$(uname -s)" != "Darwin" ]]; then - echo "macOS-only. Linux smoke test is POC step 7." >&2 - exit 1 -fi - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -POC_DIR="/tmp/syntaxkit-poc-step5" -DEMO_DIR="$POC_DIR/demo" -CACHE_DIR="$HOME/Library/Caches/com.brightdigit.SyntaxKit/helpers" -PACKAGE_FILE="$REPO_ROOT/Package.swift" -PACKAGE_BACKUP="$(mktemp)" - -cleanup() { - if [[ -s "$PACKAGE_BACKUP" ]]; then - cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" - fi - rm -f "$PACKAGE_BACKUP" -} -trap cleanup EXIT - -cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" - -echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" -python3 - "$PACKAGE_FILE" <<'PY' -import sys, pathlib -p = pathlib.Path(sys.argv[1]) -src = p.read_text() -old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' -new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' -if old not in src: - sys.exit("Package.swift: expected SyntaxKit library product block not found") -p.write_text(src.replace(old, new, 1)) -PY - -cd "$REPO_ROOT" - -echo "==> swift build" -swift build - -BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1)" -if [[ -z "$BUILD_DIR" ]]; then - echo "Could not locate .build//debug" >&2 - exit 1 -fi - -echo "==> Staging $POC_DIR" -rm -rf "$POC_DIR" -mkdir -p "$POC_DIR/lib" "$DEMO_DIR/Helpers" "$DEMO_DIR/inputs" - -cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" -cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" -cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" -cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ - "$POC_DIR/lib/_SwiftSyntaxCShims-include" - -cat > "$DEMO_DIR/Helpers/Models.swift" <<'SWIFT' -import SyntaxKit - -public func equatableModel( - _ name: String, - fields: [(name: String, type: String)] -) -> any CodeBlock { - Struct(name) { - for field in fields { - Variable(.let, name: field.name, type: field.type) - } - }.inherits("Equatable") -} -SWIFT - -cat > "$DEMO_DIR/inputs/Person.swift" <<'SWIFT' -import SyntaxKitHelpers - -equatableModel("Person", fields: [ - ("name", "String"), - ("age", "Int"), -]) -SWIFT - -cat > "$DEMO_DIR/inputs/Pet.swift" <<'SWIFT' -import SyntaxKitHelpers - -equatableModel("Pet", fields: [ - ("kind", "String"), - ("owner", "String"), -]) -SWIFT - -echo "==> Clearing helpers cache to force cold compile" -rm -rf "$CACHE_DIR" - -echo -echo "==> Cold run (helpers compile from scratch):" -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/inputs/Person.swift" - -echo -echo "==> Warm run (helpers cache hit):" -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/inputs/Person.swift" >/dev/null - -echo -echo "==> Cached helper artifacts:" -find "$CACHE_DIR" -maxdepth 3 -type f | sed "s|$CACHE_DIR||" | sort - -echo -echo "==> Folder mode (Helpers/ excluded from input enumeration):" -rm -rf "$POC_DIR/out" -"$POC_DIR/skitrun" "$DEMO_DIR" -o "$POC_DIR/out" -echo " Generated files:" -find "$POC_DIR/out" -type f | sed "s|$POC_DIR/| |" - -echo -echo "==> --no-helpers should fail with an unresolved import:" -if "$POC_DIR/skitrun" --no-helpers "$DEMO_DIR/inputs/Person.swift" >/dev/null 2>&1; then - echo "FAIL: --no-helpers should have errored" >&2 - exit 1 -else - echo " ✓ skitrun returned non-zero as expected" -fi - -echo -echo "==> Done. Demo project kept at $DEMO_DIR; cache at $CACHE_DIR." diff --git a/Docs/research/poc-step6-results.md b/Docs/research/poc-step6-results.md deleted file mode 100644 index f50fc85..0000000 --- a/Docs/research/poc-step6-results.md +++ /dev/null @@ -1,58 +0,0 @@ -# POC Step 6 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §5 + §6 step 6. Goal: skip the `swift` spawn entirely when the rendered output for an input is already cached, with a key that captures everything that could change the output. - -## What landed - -1. **`Sources/skitrun/OutputCache.swift`.** Three functions: - - `outputCacheKey(inputSource:helpers:libPath:)` — SHA-256 over cache schema version + input bytes + helpers cache key (or `"no-helpers"`) + `swift --version` + libSyntaxKit dylib stamp + sorted `SKITRUN_*` / `SYNTAXKIT_*` env vars. - - `lookupCachedOutput(key:)` — returns the cached `output.swift` bytes from `~/Library/Caches/com.brightdigit.SyntaxKit/outputs//output.swift`, or `nil` on miss. - - `storeCachedOutput(key:data:)` — atomic write via `tmp../` staging dir + rename. Concurrent writers race safely; the loser drops their copy if the destination already exists. - -2. **`Sources/skitrun/Main.swift` wiring.** - - `processFile` reads the input source, computes the cache key, and short-circuits on hit — no temp wrapper, no `swift` spawn, just return the cached bytes with `exitCode = 0`. - - On miss, the normal wrap + spawn path runs; if `swift` returns 0, the rendered output is stored under the key. - - Only successful (exit 0) runs are cached. Failed runs always re-spawn so the user sees fresh diagnostics. - - `CLIArgs` gains `--no-cache` to skip the cache entirely (still useful when chasing flaky output or after manually deleting the cache). - - The flag threads through `runSingleFile` and `runDirectory` so folder mode can opt out wholesale. - -3. **Helpers.swift internal exposure.** `syntaxKitCacheRoot`, `captureSwiftVersion`, and `libStamp` were `private`; they're now `internal` so OutputCache can reuse them rather than duplicating. - -## Verified flows - -From `Docs/research/poc-step6.sh` (single input, no helpers): - -| Run | Real time | Notes | -| --- | ---: | --- | -| Cold (cache cleared) | 0.55s | swift spawn + compile + store | -| Warm (cache hit) | 0.14s | FS read only, no swift spawn | -| `--no-cache` | 0.27s | always spawn, ignore cache | -| After mutation (miss) | 0.41s | new key, recompile, store | -| Warm after mutation | 0.14s | second key cached | - -After mutation, the cache directory contains **two** entries (one per input version) — old keys aren't evicted, which is fine for a per-toolchain cache where stale entries are dead weight, not correctness risks. Eviction can be a follow-up if cache size becomes a complaint. - -## Cache key, written out - -``` -SHA-256( - "v1" // cache schema version - + input.swift bytes // verbatim user input - + helpersCacheKey || "no-helpers" // helpers fingerprint (sibling cache) - + swift --version stdout // toolchain fingerprint - + "/" of libSyntaxKit.dylib // SyntaxKit fingerprint proxy - + sorted SKITRUN_*/SYNTAXKIT_* env (k=v\0) // env override sensitivity -) -``` - -Two cooperating cache layers — helpers (step 5) and outputs (step 6) — sit side-by-side under `~/Library/Caches/com.brightdigit.SyntaxKit/{helpers,outputs}//`. Helpers cache hits are reused across many inputs; output cache hits are per-input. - -## Known rough edges - -- **Output cache stores stdout only.** Stderr from a successful run (e.g. warnings even with `-suppress-warnings` off) is discarded on a hit. With `-suppress-warnings` in `runSwift` this is rarely visible, but it does mean cache hits suppress warnings that would have appeared on a fresh run. Acceptable for a generator; revisit if SyntaxKit grows runtime-side warnings. -- **`libStamp` is a coarse proxy.** Size + mtime catches normal rebuilds but a deterministic rebuild that preserves both would slip past. Hashing the dylib is correct but slow (9.3 MB per invocation defeats the cache). The right long-term fix is embedding a SyntaxKit version constant the bundle exports. -- **No size cap or eviction.** A repo that touches inputs frequently will accrete cache entries. Each is small (a few hundred bytes of rendered Swift) so the practical ceiling is high, but a `--prune` subcommand is a reasonable v1.1 addition. - -## What's next - -Step 7: **Linux smoke test.** Confirm `/usr/bin/env swift` + the bundled-dylib layout works on Linux without `-F` framework search paths — `-I + -L + -lSyntaxKit` should be sufficient per Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch. CryptoKit is macOS-only; Linux will need `swift-crypto` or a small fallback hash impl behind a `#if canImport(CryptoKit)` shim. After that, the 7-step ladder is complete. diff --git a/Docs/research/poc-step6.sh b/Docs/research/poc-step6.sh deleted file mode 100755 index ad90c70..0000000 --- a/Docs/research/poc-step6.sh +++ /dev/null @@ -1,116 +0,0 @@ -#!/usr/bin/env bash -# -# POC step 6 demo: rendered-output cache. skitrun skips the `swift` spawn -# when input + helpers + toolchain are unchanged. -# -# Builds skitrun, stages a runtime lib next to it, runs an input, then -# replays it three ways: -# 1. with cache — should be near-instant (no swift spawn) -# 2. --no-cache — always spawns swift -# 3. mutated input — invalidates the cache key, falls back to swift - -set -euo pipefail - -if [[ "$(uname -s)" != "Darwin" ]]; then - echo "macOS-only. Linux smoke test is POC step 7." >&2 - exit 1 -fi - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -POC_DIR="/tmp/syntaxkit-poc-step6" -DEMO_DIR="$POC_DIR/demo" -OUTPUT_CACHE="$HOME/Library/Caches/com.brightdigit.SyntaxKit/outputs" -PACKAGE_FILE="$REPO_ROOT/Package.swift" -PACKAGE_BACKUP="$(mktemp)" - -cleanup() { - if [[ -s "$PACKAGE_BACKUP" ]]; then - cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" - fi - rm -f "$PACKAGE_BACKUP" -} -trap cleanup EXIT - -cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" - -echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" -python3 - "$PACKAGE_FILE" <<'PY' -import sys, pathlib -p = pathlib.Path(sys.argv[1]) -src = p.read_text() -old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' -new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' -if old not in src: - sys.exit("Package.swift: expected SyntaxKit library product block not found") -p.write_text(src.replace(old, new, 1)) -PY - -cd "$REPO_ROOT" - -echo "==> swift build" -swift build - -BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1)" -if [[ -z "$BUILD_DIR" ]]; then - echo "Could not locate .build//debug" >&2 - exit 1 -fi - -echo "==> Staging $POC_DIR" -rm -rf "$POC_DIR" -mkdir -p "$POC_DIR/lib" "$DEMO_DIR" - -cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" -cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" -cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" -cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ - "$POC_DIR/lib/_SwiftSyntaxCShims-include" - -cat > "$DEMO_DIR/Input.swift" <<'SWIFT' -Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") -} -SWIFT - -echo "==> Clearing output cache to force a cold run" -rm -rf "$OUTPUT_CACHE" - -echo -echo "==> Cold run (cache miss → swift spawn → store):" -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> Warm run (cache hit → no swift spawn):" -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> --no-cache (always spawn swift, even with cache present):" -/usr/bin/time -p "$POC_DIR/skitrun" --no-cache "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> Output cache contents:" -find "$OUTPUT_CACHE" -maxdepth 3 -type f | sed "s|$OUTPUT_CACHE||" | sort - -echo -echo "==> Mutating input invalidates the cache:" -cat > "$DEMO_DIR/Input.swift" <<'SWIFT' -Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") - Variable(.let, name: "email", type: "String") -} -SWIFT -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> Warm run after mutation:" -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> Cache now contains two distinct keys:" -find "$OUTPUT_CACHE" -maxdepth 1 -mindepth 1 -type d | wc -l | xargs -I {} echo " {} cache entries" - -echo -echo "==> Done. Cache at $OUTPUT_CACHE." diff --git a/Docs/research/poc-step7-results.md b/Docs/research/poc-step7-results.md deleted file mode 100644 index a9a40fe..0000000 --- a/Docs/research/poc-step7-results.md +++ /dev/null @@ -1,45 +0,0 @@ -# POC Step 7 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 7. Goal: confirm the `skitrun` flow that worked on macOS (steps 1-6) also works on Linux with the bundled-dylib layout, no framework search paths, and no Apple-only crypto. - -## What landed - -1. **`swift-crypto` swap-in.** `Sources/skitrun/Helpers.swift` and `OutputCache.swift` now `import Crypto` instead of `CryptoKit`. The Apple `swift-crypto` package vends the same `SHA256` API on every platform and re-exports CryptoKit on Apple platforms when available, so there's no `#if` shim needed in skitrun's own code. Added as a Package dependency at `from: "3.0.0"`. - -2. **Platform-aware dylib filename.** A new `dylibFilename(forLibrary:)` helper returns `libX.dylib` on Apple / `libX.so` on Linux. All call sites that hard-coded `libSyntaxKit.dylib` or `libSyntaxKitHelpers.dylib` now go through it: `isLibDir`, `libStamp`, helpers cache-hit check, and the `swiftc -emit-library -o` path. - -3. **Skip `@rpath` install-name on Linux.** `compileHelpers` previously passed `-Xlinker -install_name -Xlinker @rpath/libSyntaxKitHelpers.dylib`. That flag is Mach-O specific; on Linux it errors at link time. Wrapped in `#if !os(Linux)`. The `-Xlinker -rpath` flag still works on both platforms and is what actually locates the dylib at runtime. - -4. **`Docs/research/poc-step7.sh`.** Self-rerunning Docker wrapper: when invoked on the host it re-execs itself inside `swift:6.0-jammy` with the repo bind-mounted; inside the container it flips Package.swift to dynamic, builds with `--build-path .build-linux` (separate from the macOS host's `.build`), stages a `lib/` next to `skitrun`, and runs cold + warm + `--no-cache` against an input that uses helpers. - -## Verified flows - -From `Docs/research/poc-step7.sh` inside `swift:6.0-jammy` (aarch64): - -| Flow | Time | Notes | -| --- | ---: | --- | -| Cold (helpers compile + output cache miss) | 0.73s | swiftc spawns for helpers, swift interprets the wrapped input | -| Warm (output cache hit) | 0.26s | no swift spawn | -| `--no-cache` | 0.30s | swift spawn, helpers reused from cache | - -Rendered output for the demo `Person` struct matches the macOS step 5 output exactly — same SyntaxKit, same generator, no platform-specific quirks in the rendered code. - -## Linux-only surprises - -- **`Process.waitUntilExit()` hangs on already-exited children.** This was the biggest find. Foundation's `Process.waitUntilExit()` on `swift:6.0-jammy/aarch64` blocks indefinitely even when the child has clearly exited (stdout EOF observed, all 76 bytes of `swift --version` already read). Fix applied to all three callers (`captureSwiftVersion`, `compileHelpers`, `runSwift`): set `process.terminationHandler = { _ in semaphore.signal() }` before `run()`, then `semaphore.wait()` instead of `waitUntilExit()`. Took down a 20-minute mystery hang. -- **Stdout/stderr pipe drain order matters.** Linux pipe buffers are ~64 KB; reading sequentially after waiting for child exit deadlocks when the child fills either pipe. `runSwift` now drains both pipes concurrently via `DispatchGroup` + a `PipeDataBox` class (the boxing satisfies Swift 6 strict-concurrency without `@unchecked Sendable` on local vars). `compileHelpers` only needs to drain stderr (stdout goes to `/dev/null`), but drains before the wait for the same reason. -- **`-Xlinker -install_name @rpath/...` is Mach-O specific.** Errors out on Linux's GNU ld. Wrapped in `#if !os(Linux)` in `compileHelpers`. The `-Xlinker -rpath -Xlinker ` flag still works on both platforms and is what actually locates the dylib at runtime. -- **`/usr/bin/time` isn't installed in `swift:6.0-jammy` by default.** Demo script uses the bash builtin `time` for timing, which writes a different format but is portable. -- **`swift build --product skitrun` alone doesn't emit `libSyntaxKit.so`.** The first `poc-step7.sh` draft scoped the build to just the executable, which produced a 60 MB statically-linked binary and no dylib at all. Plain `swift build` (all products) emits both `libSyntaxKit.so` and `skitrun`, matching the macOS steps 5-6 scripts. -- **First-time build cost.** Cold dependency resolution + boringssl C compile (pulled in by `swift-crypto` on Linux where CommonCrypto isn't available) takes ~3 min in `swift:6.0-jammy/aarch64`. Subsequent runs reuse `.build-linux/` and finish in ~40s. -- **Crypto on Linux brings boringssl.** `swift-crypto` statically links boringssl on non-Apple platforms. `skitrun`'s Linux binary is therefore noticeably larger than the macOS one (boringssl C blobs add up). Not a correctness issue, just a size note for future packaging. - -## What's next - -The 7-step POC ladder is **complete**. With this commit: - -- Cold-start cost has been measured on both platforms. -- The bundled-dylib + script-mode `swift` invocation works on macOS and Linux. -- Folder mode, helpers, and the rendered-output cache all behave the same way on both. - -The remaining bullets in [`codegen-cli-design.md` §7](./codegen-cli-design.md#7-what-we-still-need-to-verify) — timeouts, the splice-fidelity audit beyond the demo inputs, `@main`/attribute behavior in script-mode swift, the multi-file output question, and the web-server form — are now the natural next conversation. None of them block productizing the CLI; all of them are scope decisions rather than open technical risks. diff --git a/Docs/research/poc-step7.sh b/Docs/research/poc-step7.sh deleted file mode 100755 index 1d6c98e..0000000 --- a/Docs/research/poc-step7.sh +++ /dev/null @@ -1,135 +0,0 @@ -#!/usr/bin/env bash -# -# POC step 7: Linux smoke test for skitrun. -# -# Runs inside a swift:6.0-jammy container. Builds skitrun, stages a -# runtime lib/ next to it (libSyntaxKit.so + Modules + _SwiftSyntaxCShims -# headers), then exercises single-file mode, helpers, and the output -# cache — the same flows POC steps 5 and 6 verified on macOS. -# -# Usage (from macOS host or Linux host with docker): -# Docs/research/poc-step7.sh -# -# Override the image with $SKITRUN_LINUX_IMAGE. -# -# To save time across runs, the script uses .build-linux/ as a separate -# build directory so the host's .build/ stays clean and SwiftSyntax -# doesn't re-download on every invocation. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -IMAGE="${SKITRUN_LINUX_IMAGE:-swift:6.0-jammy}" - -if [[ ! -f /.dockerenv ]]; then - # ---- Host side: invoke ourselves inside the swift container. ---- - if ! command -v docker >/dev/null; then - echo "docker is required for POC step 7" >&2 - exit 1 - fi - echo "==> Running POC step 7 inside $IMAGE" - exec docker run --rm -t \ - -v "$REPO_ROOT:/workspace" \ - -w /workspace \ - -e SKITRUN_INSIDE_DOCKER=1 \ - "$IMAGE" \ - /workspace/Docs/research/poc-step7.sh -fi - -# ---- Container side: do the real work. ---- - -PACKAGE_FILE="Package.swift" -PACKAGE_BACKUP="$(mktemp)" -cleanup() { - if [[ -s "$PACKAGE_BACKUP" ]]; then - cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" - fi - rm -f "$PACKAGE_BACKUP" -} -trap cleanup EXIT - -cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" - -echo "==> swift --version" -swift --version - -echo -echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" -python3 - "$PACKAGE_FILE" <<'PY' -import sys, pathlib -p = pathlib.Path(sys.argv[1]) -src = p.read_text() -old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' -new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' -if old not in src: - sys.exit("Package.swift: expected SyntaxKit library product block not found") -p.write_text(src.replace(old, new, 1)) -PY - -echo -echo "==> swift build (build path: .build-linux)" -swift build --build-path .build-linux - -BUILD_DIR="$(ls -d .build-linux/*-unknown-linux-gnu/debug 2>/dev/null | head -1)" -if [[ -z "$BUILD_DIR" ]]; then - echo "Could not locate Linux build dir under .build-linux/" >&2 - ls -la .build-linux/ || true - exit 1 -fi - -POC_DIR=/tmp/syntaxkit-poc-step7 -DEMO_DIR="$POC_DIR/demo" -OUTPUT_CACHE="$HOME/.cache/syntaxkit/outputs" - -rm -rf "$POC_DIR" "$HOME/.cache/syntaxkit" -mkdir -p "$POC_DIR/lib" "$DEMO_DIR/Helpers" - -cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" -cp "$BUILD_DIR/libSyntaxKit.so" "$POC_DIR/lib/" -cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" -cp -r .build-linux/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include \ - "$POC_DIR/lib/_SwiftSyntaxCShims-include" - -cat > "$DEMO_DIR/Helpers/Models.swift" <<'SWIFT' -import SyntaxKit - -public func equatableModel( - _ name: String, - fields: [(name: String, type: String)] -) -> any CodeBlock { - Struct(name) { - for field in fields { - Variable(.let, name: field.name, type: field.type) - } - }.inherits("Equatable") -} -SWIFT - -cat > "$DEMO_DIR/Input.swift" <<'SWIFT' -import SyntaxKitHelpers - -equatableModel("Person", fields: [ - ("name", "String"), - ("age", "Int"), -]) -SWIFT - -echo -echo "==> Cold run (helpers compile + output cache miss):" -time "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" - -echo -echo "==> Warm run (output cache hit, no swift spawn):" -time "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> --no-cache (swift spawn, helpers reused):" -time "$POC_DIR/skitrun" --no-cache "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> Output cache entries:" -find "$OUTPUT_CACHE" -maxdepth 2 -type f 2>/dev/null | sed "s|$OUTPUT_CACHE||" | sort - -echo -echo "==> Linux smoke test passed." diff --git a/Docs/skit.md b/Docs/skit.md new file mode 100644 index 0000000..8702ff5 --- /dev/null +++ b/Docs/skit.md @@ -0,0 +1,209 @@ +# `skit` — a SyntaxKit CLI for config-driven Swift codegen + +`skit` is a small CLI that takes a SyntaxKit DSL file as input and writes Swift source out the other side. The vision: pure data — JSON, YAML, your own format — drives a manifest written in the SyntaxKit DSL, and `skit` materializes it into idiomatic Swift you check in alongside everything else. Less hand-maintenance, fewer drift bugs. + +This doc walks through how `skit` is built. Two verbs, one wrap-and-spawn pipeline, two layers of cache, and a careful toolchain story underneath. Where there are sharp edges, this doc names them. + +## Two verbs + +``` +skit run Input.swift # SyntaxKit DSL → Swift source on stdout +skit run Input.swift -o Out.swift +skit run InputDir/ -o OutDir/ # walk **/*.swift and mirror rendered output +skit parse < Input.swift # Swift source → JSON syntax tree +``` + +`run` is the default subcommand. `skit Input.swift` is shorthand for `skit run Input.swift`. `parse` is a one-shot for the inverse direction (Swift source → SwiftSyntax tree as JSON) — useful for tooling that wants to introspect existing code. + +The remainder of this doc is about `run`, which is where the interesting design decisions live. + +## How `skit run` works + +A `.swift` input file looks like a SyntaxKit DSL expression at the top level: + +```swift +// Models.swift +import SyntaxKitHelpers // optional — only if a Helpers/ dir is present + +equatableModel("Person", fields: [ + ("name", "String"), + ("age", "Int"), +]) +``` + +The input is not a complete Swift program. It has no `@main`, no `print`, no `let root = …`. `skit run` adds the boilerplate by wrapping the input in a `Group { … }` builder and a top-level `print` that renders the result: + +```swift +// What `skit run` writes to a temp file before spawning `swift`: +import SyntaxKit +import SyntaxKitHelpers // hoisted from the input + +let __skit_root = Group { +#sourceLocation(file: "/path/to/Models.swift", line: 3) +equatableModel("Person", fields: [ + ("name", "String"), + ("age", "Int"), +]) +#sourceLocation() +} + +print(__skit_root.generateCode()) +``` + +`skit` then spawns `swift` (the script-mode interpreter, not `swiftc`) on this wrapped file, captures stdout, and writes the result to the user's chosen output target. + +Three things deserve a closer look: + +**Imports get hoisted.** The wrapper has to start with `import SyntaxKit` so the DSL types are available. The user's `import`s — typically `import SyntaxKitHelpers` plus anything their Helpers/ module references — need to live at the top of the file, not inside `Group { … }`. `skit` parses the input with SwiftSyntax, peels off the leading `import` declarations, and lifts them into the wrapper preamble. Anything else (declarations, expressions, top-level types) stays in the body. + +**`#sourceLocation` keeps diagnostics readable.** When the spawned `swift` emits a compile error, it reports a line number in the wrapped temp file, which is meaningless to the user. The `#sourceLocation` directive remaps body diagnostics back to the original input path and line. Errors in the wrapper preamble (the `import` block, the `Group { … }` opening) still reference the temp file — `skit` rewrites occurrences of the temp path in stderr to the input path as a fallback, so users see something coherent. + +**`swift` runs in script mode.** Running `swift Input.swift` invokes the Swift interpreter rather than going through `swiftc` + `ld`. Cold-start is around 700ms on macOS; warm spawns are around 110ms. The CLI's hot path leans into this — for batch input via `skit run InputDir/`, we spawn one `swift` per input file in parallel up to the active core count. + +## Caches + +Two layers, both keyed by content hash. They live under `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS, `$XDG_CACHE_HOME/syntaxkit` (or `~/.cache/syntaxkit`) on Linux. + +| Layer | Path | What it skips on hit | +| ------- | ----------------------------- | --------------------------------------------- | +| Helpers | `helpers//` | the `swiftc` compile of `Helpers/*.swift` | +| Output | `outputs//output.swift` | the `swift` spawn for an input | + +**Helpers cache.** Each project's `Helpers/` directory gets compiled into `libSyntaxKitHelpers.{dylib,so}` once and reused. The cache key is a hash of the helper sources, plus the bundled `libSyntaxKit` stamp, plus `swift --version`. Touching a helper file invalidates one shard; updating the toolchain invalidates everything. Hit on a warm cache: skip the ~1–2s `swiftc` compile entirely. + +**Output cache.** The fully-rendered output of an input gets cached by a hash of (input bytes, helpers shard, libSyntaxKit stamp, swift version, sorted `SKIT_*`/`SYNTAXKIT_*` env vars). On hit, `skit` doesn't spawn `swift` at all — total wall time is around 0.14s, dominated by hash + file read. Cold miss matches the warm script-mode baseline (~0.5s). + +`--no-cache` skips the output cache. There's no flag to skip the helpers cache — invalidate it by touching a helper or bumping the toolchain. + +## Toolchain stamping + +Pre-compiled Swift modules (`SyntaxKit.swiftmodule`) are tightly coupled to the compiler that built them. Even patch-level differences fail to load: + +``` +error: module compiled with Swift 6.3 cannot be imported by the Swift 6.3.2 compiler +``` + +If `skit`'s bundled `lib/SyntaxKit.swiftmodule` doesn't match the user's `swift`, the spawned interpreter emits exactly this diagnostic and refuses to compile the wrapped input. The user is left staring at a cryptic message that doesn't name the actual problem. + +`skit` mitigates this by recording the build toolchain at bundle time. `Scripts/build-skit-release.sh` writes `lib/swift-version.txt` containing the output of `swift --version`. On startup, `skit run` reads the stamp and compares it to a freshly-captured local `swift --version`. Three paths: + +- **Match.** Proceed silently. +- **Mismatch.** Print a clear error naming both versions and the rebuild command, exit 2. Skip with `--no-toolchain-check`. +- **Missing stamp.** Print a one-line note and proceed. Older bundles built before this check existed shouldn't break. + +The comparison is exact-string match. Patch-level drift broke the originating bug, so anything less strict would just defer the failure to the cryptic message we're trying to avoid. + +**What this doesn't do**: it doesn't fix the mismatch, just surfaces it. The fix is to re-run the release script with the user's current toolchain. The auto-rebuild path — bundle SyntaxKit sources alongside the prebuilt module and recompile transparently when the stamp doesn't match — is tracked as [issue #157](https://github.com/brightdigit/SyntaxKit/issues/157). + +## Timeout watchdog + +The spawned `swift` is the only unbounded piece of `skit run`'s hot path. The wrap step is microseconds. Helpers compile is cached. The output cache hits or misses in milliseconds. But the spawn itself runs *user code* — and that code is allowed to be arbitrarily slow, recursive, or stuck. + +`skit run` defaults to a 60s per-input timeout. On expiry it sends `SIGTERM`, gives a 5s grace, then `SIGKILL`. The wrapped input exits with code 124 — POSIX `timeout(1)`'s convention. `--timeout ` overrides the default; `--timeout 0` disables the watchdog entirely (useful for debugging genuinely long codegen). + +The implementation is `DispatchSemaphore.wait(timeout: deadline)` paired with a `process.terminationHandler` that signals on child exit. The Linux Foundation `Process.waitUntilExit()` hangs on already-exited children on some configurations, which is why `skit` uses the semaphore-based wait everywhere. Same story for pipe drains — sequential reads after the child exits can deadlock when either pipe (~64 KB buffer on Linux) fills before exit, so both pipes drain concurrently via `DispatchGroup`. + +## Helpers + +Shared codegen utilities live in a `Helpers/` directory. `skit` walks up from the input, finds the nearest `Helpers/`, and compiles its sources into a Swift dylib that the wrapped input can `import SyntaxKitHelpers`. + +``` +project/ +├── Helpers/ +│ └── Models.swift # public func equatableModel(_:fields:) -> any CodeBlock +└── inputs/ + ├── Person.swift # imports SyntaxKitHelpers, calls equatableModel(...) + └── Pet.swift # same +``` + +Files prefixed with `_` are skipped — a convention for private helpers within the helpers module. The module name is hard-coded to `SyntaxKitHelpers`. + +The helpers cache is per-content, so editing a helper triggers a fresh compile but reading the same helper across many inputs hits the cache. + +## Sharp edges + +### `if`-in-`Group` crashes the Swift type-checker — [#158](https://github.com/brightdigit/SyntaxKit/issues/158) + +`CodeBlockBuilderResult` declares all the methods needed for `if`/`else` in a result builder (`buildEither`, `buildOptional`, `buildArray`), but using them surfaces a Swift type-checker bug: + +```swift +let _ = Group { + if true { // ← error: failed to produce diagnostic for expression + Struct("A") { … } + } +} +``` + +This is a Swift compiler bug, not a `skit` bug. The workaround is to hoist the conditional into a helper function that uses **plain Swift `if`/`else`** (not a `Group { if … }` body) to return one of two `CodeBlock`s: + +```swift +// In Helpers/Models.swift +public func optionalDebugField(_ include: Bool) -> any CodeBlock { + if include { + return Variable(.let, name: "debug", type: "Bool") + } else { + return Group {} // empty Group as the "no-op" branch — there's no + // public EmptyCodeBlock type + } +} + +// In Input.swift +Struct("Config") { + Variable(.let, name: "name", type: "String") + optionalDebugField(buildIsDebug) +} +``` + +The helper itself can't use `Group { if … }` either — same crash. Plain Swift control flow only. + +### `@main` and top-level decl attributes don't work + +`@main` and other decl attributes (like top-level `@available`) at the start of the input fail to compile. The wrapper places the user's body inside `Group { … }`, where these attributes try to bind to a function-call expression rather than a declaration. Examples: + +```swift +@main // ❌ error: expected declaration +Struct("Person") { … } + +@available(iOS 17, *) // ❌ error: expected declaration +Struct("ModernView") { … } +``` + +The DSL has its own mechanism for attribute attachment — call `.attribute("Published")`, `.attribute("available", arguments: ["iOS 17", "*"])` on the `Variable`/`Struct`/etc. — which renders the attribute in the *output*, not on the DSL expression itself. That's the path users should take. + +### Comments don't carry through + +Swift comments in the input file (`// MARK: - Models`, block comments, inline `//` after a DSL line) don't render to the output. The input's comments are *only* for the input's author. To emit a comment in the rendered code, attach it to a `CodeBlock` via `.comment { Line(.doc, "…") }`. + +### `#if os(...)` is gen-time, not output + +A `#if os(macOS) … #endif` block in the input is evaluated when the wrapped file compiles, controlling whether the enclosed DSL expressions are part of the builder. It does *not* emit a `#if os(macOS)` directive into the rendered Swift output. If you want compile-time-conditional output, emit the `#if` lines as raw text via `VariableExp("#if os(macOS)")` (or write a small helper). + +## Platform notes + +**macOS** is the primary target. The build and release flows live in `Scripts/`; the bundle is portable across machines with the same Swift version. + +**Linux** is verified on `swift:6.0-jammy/aarch64`. Two adjustments compared to macOS: + +- `swift-crypto` replaces CryptoKit (we depend on Crypto for cache-key hashing). `swift-crypto` statically links boringssl on Linux, so the binary is noticeably larger than the macOS one. +- The Mach-O `install_name` rewrite in `Scripts/build-skit-release.sh` is skipped on Linux. GNU `ld` doesn't accept the flag; the `-rpath` injection (which is what actually locates the dylib at runtime) works on both platforms. + +The Foundation.Process workarounds described earlier are Linux-driven. `Process.waitUntilExit()` blocks indefinitely on already-exited children on `swift:6.0-jammy/aarch64` — `skit` uses `DispatchSemaphore` everywhere a child wait is needed, and drains stdout/stderr pipes concurrently to avoid deadlocks when either pipe buffer fills. + +**Windows** is not supported. + +## What's deferred + +A few things were considered for v1 and explicitly punted: + +- **Auto-rebuild on toolchain mismatch** — [#157](https://github.com/brightdigit/SyntaxKit/issues/157). Today the user gets a clear error and a rebuild command; tomorrow `skit` should rebuild `libSyntaxKit` from bundled sources transparently and cache the result per Swift version. The stamp-and-detect path shipped in this release is the foundation; the rebuild fallback is the natural next step. +- **Multi-file output from a single input** — out of scope. The folder mode handles N-inputs → N-outputs; if you need fan-out from one logical generator, split it into multiple input files. +- **Sandboxing the spawned `swift`** — out of scope. The threat model is "you ran your own code", same as `swift Input.swift` by hand. +- **HTTP / web-server form** — out of scope for the CLI. A long-lived server could share a warm interpreter and the caches across requests, but that's a different shape with its own isolation questions. Revisit later. + +## Reference + +- [`Sources/skit/README.md`](../Sources/skit/README.md) — per-target quick reference (flag table, helpers layout). +- [`Scripts/build-skit-release.sh`](../Scripts/build-skit-release.sh) — release-bundle builder. +- [`Docs/research/tuist-manifest-pipeline.md`](research/tuist-manifest-pipeline.md) — the manifest-pipeline pattern this CLI borrows from. +- [Issue #154](https://github.com/brightdigit/SyntaxKit/issues/154) — original tracking issue. +- [Issue #157](https://github.com/brightdigit/SyntaxKit/issues/157), [Issue #158](https://github.com/brightdigit/SyntaxKit/issues/158) — follow-ups. From e3e243ff06786d167388f538938cb6b0f26ed884 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 16:11:04 -0400 Subject: [PATCH 22/28] Replace swift-crypto/SHA-256 with pure-Swift FNV-1a content hash MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two cache keys (helpers cache, output cache) only need a stable, deterministic, cross-platform hash for content addressing — they aren't security-critical and there's no adversary trying to forge collisions. Drop swift-crypto's SHA-256 in favour of a ~10-line pure-Swift FNV-1a 64-bit hasher. - 64-bit output gives ~10⁻⁹ collision probability at 10⁶ cache entries (well past anything we'll see). - Cache keys go from 64-char hex to 16-char hex. - Drops the swift-crypto dep entirely. On macOS the binary barely changes (~4KB) since swift-crypto used CryptoKit there anyway; on Linux we no longer statically link boringssl, which was the dep's real cost. - New file: Sources/skit/ContentHasher.swift. Same streaming API (update/finalize) so the two call sites are nearly identical to the SHA-256 version. All 10 working examples re-rendered with fresh keys after cache clear; output identical. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/skit.md | 5 +-- Package.resolved | 20 +---------- Package.swift | 2 -- Sources/skit/ContentHasher.swift | 59 ++++++++++++++++++++++++++++++++ Sources/skit/Helpers.swift | 8 ++--- Sources/skit/OutputCache.swift | 12 +++---- Sources/skit/README.md | 2 +- 7 files changed, 72 insertions(+), 36 deletions(-) create mode 100644 Sources/skit/ContentHasher.swift diff --git a/Docs/skit.md b/Docs/skit.md index 8702ff5..35baa07 100644 --- a/Docs/skit.md +++ b/Docs/skit.md @@ -182,10 +182,7 @@ A `#if os(macOS) … #endif` block in the input is evaluated when the wrapped fi **macOS** is the primary target. The build and release flows live in `Scripts/`; the bundle is portable across machines with the same Swift version. -**Linux** is verified on `swift:6.0-jammy/aarch64`. Two adjustments compared to macOS: - -- `swift-crypto` replaces CryptoKit (we depend on Crypto for cache-key hashing). `swift-crypto` statically links boringssl on Linux, so the binary is noticeably larger than the macOS one. -- The Mach-O `install_name` rewrite in `Scripts/build-skit-release.sh` is skipped on Linux. GNU `ld` doesn't accept the flag; the `-rpath` injection (which is what actually locates the dylib at runtime) works on both platforms. +**Linux** is verified on `swift:6.0-jammy/aarch64`. One adjustment compared to macOS: the Mach-O `install_name` rewrite in `Scripts/build-skit-release.sh` is skipped — GNU `ld` doesn't accept the flag. The `-rpath` injection (which is what actually locates the dylib at runtime) works on both platforms. The Foundation.Process workarounds described earlier are Linux-driven. `Process.waitUntilExit()` blocks indefinitely on already-exited children on `swift:6.0-jammy/aarch64` — `skit` uses `DispatchSemaphore` everywhere a child wait is needed, and drains stdout/stderr pipes concurrently to avoid deadlocks when either pipe buffer fills. diff --git a/Package.resolved b/Package.resolved index 07e1898..fc8a607 100644 --- a/Package.resolved +++ b/Package.resolved @@ -1,5 +1,5 @@ { - "originHash" : "36e6466ffd4edf53c9520bb53570c9119c1a34eb0e59dd824b1c4ff4f19189a8", + "originHash" : "fb25157fbc930f88cbd845709bb74c245da530b7a57458eb846e0d74bcd9c062", "pins" : [ { "identity" : "swift-argument-parser", @@ -10,24 +10,6 @@ "version" : "1.7.1" } }, - { - "identity" : "swift-asn1", - "kind" : "remoteSourceControl", - "location" : "https://github.com/apple/swift-asn1.git", - "state" : { - "revision" : "9f542610331815e29cc3821d3b6f488db8715517", - "version" : "1.6.0" - } - }, - { - "identity" : "swift-crypto", - "kind" : "remoteSourceControl", - "location" : "https://github.com/apple/swift-crypto.git", - "state" : { - "revision" : "95ba0316a9b733e92bb6b071255ff46263bbe7dc", - "version" : "3.15.1" - } - }, { "identity" : "swift-docc-plugin", "kind" : "remoteSourceControl", diff --git a/Package.swift b/Package.swift index 551ba63..2ce09bd 100644 --- a/Package.swift +++ b/Package.swift @@ -99,7 +99,6 @@ let package = Package( dependencies: [ .package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.1"), .package(url: "https://github.com/swiftlang/swift-docc-plugin", from: "1.4.0"), - .package(url: "https://github.com/apple/swift-crypto.git", from: "3.0.0"), .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0") ], targets: [ @@ -147,7 +146,6 @@ let package = Package( "SyntaxParser", .product(name: "SwiftSyntax", package: "swift-syntax"), .product(name: "SwiftParser", package: "swift-syntax"), - .product(name: "Crypto", package: "swift-crypto"), .product(name: "ArgumentParser", package: "swift-argument-parser") ], swiftSettings: swiftSettings diff --git a/Sources/skit/ContentHasher.swift b/Sources/skit/ContentHasher.swift new file mode 100644 index 0000000..6ec0e3c --- /dev/null +++ b/Sources/skit/ContentHasher.swift @@ -0,0 +1,59 @@ +// +// ContentHasher.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +/// Non-cryptographic 64-bit FNV-1a hasher used to derive content-addressed +/// cache keys. The cache keys aren't security-critical — there's no +/// adversary trying to forge a collision — so we don't need a cryptographic +/// hash. 64 bits of output gives ~10⁻⁹ collision probability at 10⁶ cache +/// entries, which is well past anything we'll see in practice. +/// +/// FNV-1a is deterministic across processes and platforms (unlike the Swift +/// stdlib `Hasher`, whose seed is randomized per-process) — that +/// determinism is what makes it usable as an on-disk cache key. +internal struct ContentHasher { + private static let offsetBasis: UInt64 = 0xcbf2_9ce4_8422_2325 + private static let prime: UInt64 = 0x0000_0100_0000_01b3 + + private var state: UInt64 = ContentHasher.offsetBasis + + internal mutating func update(data: Data) { + for byte in data { + state ^= UInt64(byte) + state &*= ContentHasher.prime + } + } + + /// Returns the hash as a 16-char lowercase-hex string suitable for use as + /// a directory name. + internal func finalize() -> String { + String(format: "%016x", state) + } +} diff --git a/Sources/skit/Helpers.swift b/Sources/skit/Helpers.swift index 7a33b51..ec74dd3 100644 --- a/Sources/skit/Helpers.swift +++ b/Sources/skit/Helpers.swift @@ -27,7 +27,6 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Crypto import Foundation /// Hardcoded module name for the user's `Helpers/` compilation output. Inputs @@ -118,7 +117,8 @@ internal func buildHelpers( let cacheRoot = try syntaxKitCacheRoot() .appendingPathComponent("helpers") .appendingPathComponent(key) - let dylibPath = cacheRoot + let dylibPath = + cacheRoot .appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path let fm = FileManager.default @@ -220,7 +220,7 @@ private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) t // MARK: - Cache key private func helpersCacheKey(sources: [URL], libPath: String) throws -> String { - var hasher = SHA256() + var hasher = ContentHasher() hasher.update(data: Data(helpersCacheSchemaVersion.utf8)) for source in sources { @@ -236,7 +236,7 @@ private func helpersCacheKey(sources: [URL], libPath: String) throws -> String { hasher.update(data: Data(stamp.utf8)) } - return hasher.finalize().map { String(format: "%02x", $0) }.joined() + return hasher.finalize() } internal func captureSwiftVersion() -> String? { diff --git a/Sources/skit/OutputCache.swift b/Sources/skit/OutputCache.swift index d4a26df..c555702 100644 --- a/Sources/skit/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -27,21 +27,21 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Crypto import Foundation /// Bumped when the output cache layout changes in a way that requires invalidation. private let outputCacheSchemaVersion = "v1" -/// SHA-256 over (cache schema, input source bytes, helpers key, swift version, -/// libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). Any change in -/// these inputs produces a fresh key and forces a recompile. +/// 64-bit content hash over (cache schema, input source bytes, helpers key, +/// swift version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). +/// Any change in these inputs produces a fresh key and forces a recompile. +/// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. internal func outputCacheKey( inputSource: String, helpers: CompiledHelpers?, libPath: String ) -> String { - var hasher = SHA256() + var hasher = ContentHasher() hasher.update(data: Data(outputCacheSchemaVersion.utf8)) hasher.update(data: Data(inputSource.utf8)) @@ -66,7 +66,7 @@ internal func outputCacheKey( hasher.update(data: Data("\(key)=\(value)\0".utf8)) } - return hasher.finalize().map { String(format: "%02x", $0) }.joined() + return hasher.finalize() } /// Returns the cached rendered output for `key`, or nil on miss. diff --git a/Sources/skit/README.md b/Sources/skit/README.md index 5b0f77a..630250a 100644 --- a/Sources/skit/README.md +++ b/Sources/skit/README.md @@ -89,7 +89,7 @@ Output cache hit is roughly ~0.14s on macOS (no spawn at all); cold miss matches ## Platform notes - **macOS** — primary target. All build/release/test flows in `Scripts/`. -- **Linux** — verified on `swift:6.0-jammy/aarch64`. Requires `swift-crypto` instead of CryptoKit (we depend on it). The Mach-O `install_name` step in `Scripts/build-skit-release.sh` is macOS-specific and skipped on Linux. +- **Linux** — verified on `swift:6.0-jammy/aarch64`. The Mach-O `install_name` step in `Scripts/build-skit-release.sh` is macOS-specific and skipped on Linux. - **Windows** — not supported. Known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited children on `swift:6.0-jammy/aarch64`. `Runner.swift` and `Helpers.swift` work around it with `terminationHandler` + `DispatchSemaphore`. From 61349dab49da310bc864fd1950702688ce919584 Mon Sep 17 00:00:00 2001 From: leogdion Date: Wed, 13 May 2026 09:48:16 -0400 Subject: [PATCH 23/28] Move skit to swift-subprocess; revert simulator OS pin to 26.4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 25759444849 broke skit's build on every non-macOS-host target. Foundation.Process is unavailable on watchOS/tvOS and on Ubuntu's wasm + wasm-embedded toolchains, and POSIX kill/SIGKILL aren't in scope on Windows. The previous skit implementation worked around Linux Foundation.Process quirks with a DispatchSemaphore + DispatchGroup pipe-drain ladder and a manual kill(pid, SIGKILL) timeout watchdog — none of which port. This commit replaces all three Foundation.Process call sites (captureSwiftVersion, compileHelpers, runSwift) with swift-subprocess (v0.4.0). The timeout watchdog becomes a withThrowingTaskGroup race against Task.sleep + cancelAll; swift-subprocess's teardown sequence handles SIGTERM → grace → SIGKILL on POSIX and the WM_CLOSE/CTRL_C_EVENT/TerminateProcess equivalent on Windows. ~70 lines of platform workaround dropped. swift-subprocess requires Swift 6.1, so swift-tools-version goes 6.0 → 6.1 (and CLAUDE.md follows). skit itself is gated by `#if canImport(Subprocess)`; on platforms where the Subprocess module isn't built (iOS, watchOS, tvOS, visionOS, Android, WASI) the new SkitStub.swift provides a caseless-enum @main that exits 1 with a clear message. The target still links — just with no usable subcommands. A regression test in Tests/SyntaxKitTests/Integration/ covers swift-subprocess #256 (stream-read hang on cancellation when grandchildren inherit pipe fds). skit's runSwift hits this scenario because swift forks the frontend + linker as grandchildren. Test passes locally in ~1s, bounded at 15s. The iOS/visionOS simulator failures in the same CI run were a separate cause: the May-13 workflow sync bumped osVersion to 26.5, but the macos-26 runner image still only ships OS 26.4 simulators, so download-platform: true couldn't fetch what the matrix asked for. Revert osVersion 26.5 → 26.4 on the iOS/watchOS/tvOS/visionOS rows; Xcode pins unchanged. Tracked for re-bump in #160. Scripts/build-skit-debug.sh is a debug-mode counterpart to the existing release-bundle script, for fast local end-to-end iteration on the DSL transformation (~10s vs. 5-15 min). Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/SyntaxKit.yml | 11 +- CLAUDE.md | 4 +- Package.resolved | 20 +- Package.swift | 21 +- Scripts/build-skit-debug.sh | 81 ++ Sources/skit/ContentHasher.swift | 52 +- Sources/skit/Helpers.swift | 435 ++++--- Sources/skit/OutputCache.swift | 138 +-- Sources/skit/Runner.swift | 1035 ++++++++--------- Sources/skit/Skit.swift | 369 +++--- Sources/skit/SkitStub.swift | 44 + .../SkitSubprocessTimeoutTests.swift | 101 ++ 12 files changed, 1282 insertions(+), 1029 deletions(-) create mode 100755 Scripts/build-skit-debug.sh create mode 100644 Sources/skit/SkitStub.swift create mode 100644 Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift diff --git a/.github/workflows/SyntaxKit.yml b/.github/workflows/SyntaxKit.yml index 1363b82..5fb86e0 100644 --- a/.github/workflows/SyntaxKit.yml +++ b/.github/workflows/SyntaxKit.yml @@ -229,12 +229,15 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" + # iOS / watchOS / tvOS / visionOS osVersion pinned to 26.4 until the + # macos-26 runner image ships Xcode 26.5 simulators — see #160. + # iOS Build Matrix - type: ios runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "iPhone 17 Pro" - osVersion: "26.5" + osVersion: "26.4" download-platform: true # watchOS Build Matrix @@ -242,7 +245,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple Watch Ultra 3 (49mm)" - osVersion: "26.5" + osVersion: "26.4" download-platform: true # tvOS Build Matrix @@ -250,7 +253,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple TV" - osVersion: "26.5" + osVersion: "26.4" download-platform: true # visionOS Build Matrix @@ -258,7 +261,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple Vision Pro" - osVersion: "26.5" + osVersion: "26.4" download-platform: true steps: diff --git a/CLAUDE.md b/CLAUDE.md index a09a578..008b2dd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -93,8 +93,8 @@ Sources/SyntaxKit/ 2. **skit Executable** - Command-line tool for parsing Swift code to JSON ### Platform Support -- Swift 6.0+ required -- Xcode 16.0+ for development +- Swift 6.1+ required +- Xcode 16.3+ for development ### Testing - Uses modern Swift Testing framework (`@Test` syntax) diff --git a/Package.resolved b/Package.resolved index fc8a607..99337ef 100644 --- a/Package.resolved +++ b/Package.resolved @@ -1,5 +1,5 @@ { - "originHash" : "fb25157fbc930f88cbd845709bb74c245da530b7a57458eb846e0d74bcd9c062", + "originHash" : "899b1fb6639e07a99f4d6f6c1e22b7c90948b2df293879cf18e8b0f87500bf16", "pins" : [ { "identity" : "swift-argument-parser", @@ -28,6 +28,15 @@ "version" : "1.0.0" } }, + { + "identity" : "swift-subprocess", + "kind" : "remoteSourceControl", + "location" : "https://github.com/swiftlang/swift-subprocess.git", + "state" : { + "revision" : "13d087685b95d64d6aac9b94500d347bbe84c39b", + "version" : "0.4.0" + } + }, { "identity" : "swift-syntax", "kind" : "remoteSourceControl", @@ -36,6 +45,15 @@ "revision" : "f99ae8aa18f0cf0d53481901f88a0991dc3bd4a2", "version" : "601.0.1" } + }, + { + "identity" : "swift-system", + "kind" : "remoteSourceControl", + "location" : "https://github.com/apple/swift-system", + "state" : { + "revision" : "7c6ad0fc39d0763e0b699210e4124afd5041c5df", + "version" : "1.6.4" + } } ], "version" : 3 diff --git a/Package.swift b/Package.swift index 2ce09bd..ae34413 100644 --- a/Package.swift +++ b/Package.swift @@ -1,4 +1,4 @@ -// swift-tools-version: 6.0 +// swift-tools-version: 6.1 // The swift-tools-version declares the minimum version of Swift required to build this package. import PackageDescription @@ -99,7 +99,8 @@ let package = Package( dependencies: [ .package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.1"), .package(url: "https://github.com/swiftlang/swift-docc-plugin", from: "1.4.0"), - .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0") + .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0"), + .package(url: "https://github.com/swiftlang/swift-subprocess.git", from: "0.4.0") ], targets: [ .target( @@ -146,13 +147,25 @@ let package = Package( "SyntaxParser", .product(name: "SwiftSyntax", package: "swift-syntax"), .product(name: "SwiftParser", package: "swift-syntax"), - .product(name: "ArgumentParser", package: "swift-argument-parser") + .product(name: "ArgumentParser", package: "swift-argument-parser"), + .product( + name: "Subprocess", + package: "swift-subprocess", + condition: .when(platforms: [.macOS, .linux, .windows]) + ) ], swiftSettings: swiftSettings ), .testTarget( name: "SyntaxKitTests", - dependencies: ["SyntaxKit"], + dependencies: [ + "SyntaxKit", + .product( + name: "Subprocess", + package: "swift-subprocess", + condition: .when(platforms: [.macOS, .linux, .windows]) + ) + ], swiftSettings: swiftSettings ), .testTarget( diff --git a/Scripts/build-skit-debug.sh b/Scripts/build-skit-debug.sh new file mode 100755 index 0000000..6e93156 --- /dev/null +++ b/Scripts/build-skit-debug.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# +# Build a self-contained skit DEBUG bundle for fast local iteration. +# +# Identical layout to Scripts/build-skit-release.sh but skips release-mode +# optimization (5-15 minute SwiftSyntax compile → ~10 seconds). Use this when +# you want to exercise the end-to-end DSL→Swift transformation locally; use +# the release script when staging an actual release bundle. +# +# Output: .build/skit-debug/{skit, lib/} + +set -euo pipefail + +if [[ "$(uname -s)" != "Darwin" ]]; then + echo "macOS-only. Linux uses a parallel flow." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +OUTPUT_DIR="$REPO_ROOT/.build/skit-debug" +PACKAGE_FILE="$REPO_ROOT/Package.swift" +PACKAGE_BACKUP="$(mktemp)" + +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +new_block = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if new_block in src: + print("Package.swift already has type: .dynamic — leaving as-is.") + sys.exit(0) +old_block = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +if old_block not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found") +p.write_text(src.replace(old_block, new_block, 1)) +PY + +cd "$REPO_ROOT" + +echo "==> swift build --product skit" +swift build --product skit + +echo "==> swift build --product SyntaxKit" +swift build --product SyntaxKit + +BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate debug build dir under .build//debug" >&2 + exit 1 +fi + +echo "==> Staging $OUTPUT_DIR" +rm -rf "$OUTPUT_DIR" +mkdir -p "$OUTPUT_DIR/lib" + +cp "$BUILD_DIR/skit" "$OUTPUT_DIR/skit" +cp "$BUILD_DIR/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/" +cp -r "$BUILD_DIR/Modules/." "$OUTPUT_DIR/lib/" +cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ + "$OUTPUT_DIR/lib/_SwiftSyntaxCShims-include" + +install_name_tool -id "@rpath/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/libSyntaxKit.dylib" 2>/dev/null || true + +swift --version > "$OUTPUT_DIR/lib/swift-version.txt" + +echo +echo "==> Debug bundle ready at $OUTPUT_DIR" +echo "==> Try it:" +echo " $OUTPUT_DIR/skit Examples/Completed/card_game/dsl.swift --no-cache" diff --git a/Sources/skit/ContentHasher.swift b/Sources/skit/ContentHasher.swift index 6ec0e3c..39227b5 100644 --- a/Sources/skit/ContentHasher.swift +++ b/Sources/skit/ContentHasher.swift @@ -27,33 +27,37 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation +#if canImport(Subprocess) -/// Non-cryptographic 64-bit FNV-1a hasher used to derive content-addressed -/// cache keys. The cache keys aren't security-critical — there's no -/// adversary trying to forge a collision — so we don't need a cryptographic -/// hash. 64 bits of output gives ~10⁻⁹ collision probability at 10⁶ cache -/// entries, which is well past anything we'll see in practice. -/// -/// FNV-1a is deterministic across processes and platforms (unlike the Swift -/// stdlib `Hasher`, whose seed is randomized per-process) — that -/// determinism is what makes it usable as an on-disk cache key. -internal struct ContentHasher { - private static let offsetBasis: UInt64 = 0xcbf2_9ce4_8422_2325 - private static let prime: UInt64 = 0x0000_0100_0000_01b3 + import Foundation - private var state: UInt64 = ContentHasher.offsetBasis + /// Non-cryptographic 64-bit FNV-1a hasher used to derive content-addressed + /// cache keys. The cache keys aren't security-critical — there's no + /// adversary trying to forge a collision — so we don't need a cryptographic + /// hash. 64 bits of output gives ~10⁻⁹ collision probability at 10⁶ cache + /// entries, which is well past anything we'll see in practice. + /// + /// FNV-1a is deterministic across processes and platforms (unlike the Swift + /// stdlib `Hasher`, whose seed is randomized per-process) — that + /// determinism is what makes it usable as an on-disk cache key. + internal struct ContentHasher { + private static let offsetBasis: UInt64 = 0xcbf2_9ce4_8422_2325 + private static let prime: UInt64 = 0x0000_0100_0000_01b3 - internal mutating func update(data: Data) { - for byte in data { - state ^= UInt64(byte) - state &*= ContentHasher.prime + private var state: UInt64 = ContentHasher.offsetBasis + + internal mutating func update(data: Data) { + for byte in data { + state ^= UInt64(byte) + state &*= ContentHasher.prime + } } - } - /// Returns the hash as a 16-char lowercase-hex string suitable for use as - /// a directory name. - internal func finalize() -> String { - String(format: "%016x", state) + /// Returns the hash as a 16-char lowercase-hex string suitable for use as + /// a directory name. + internal func finalize() -> String { + String(format: "%016x", state) + } } -} + +#endif diff --git a/Sources/skit/Helpers.swift b/Sources/skit/Helpers.swift index ec74dd3..f9511ae 100644 --- a/Sources/skit/Helpers.swift +++ b/Sources/skit/Helpers.swift @@ -27,250 +27,237 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation - -/// Hardcoded module name for the user's `Helpers/` compilation output. Inputs -/// reach the compiled helpers via `import SyntaxKitHelpers`. -internal let helpersModuleName = "SyntaxKitHelpers" - -/// Platform-specific shared-library filename for a Swift library product. -internal func dylibFilename(forLibrary name: String) -> String { - #if os(Linux) - return "lib\(name).so" - #else - return "lib\(name).dylib" - #endif -} - -/// Bumped when the cache layout changes in a way that requires invalidation. -private let helpersCacheSchemaVersion = "v1" - -/// A compiled `Helpers/` directory ready to splice into the input spawn. -internal struct CompiledHelpers: Sendable { - /// Directory containing `libSyntaxKitHelpers.dylib` + `.swiftmodule` files. - let outputDir: URL - /// Whether the build was reused from cache (false = freshly compiled). - let cacheHit: Bool -} - -// MARK: - Discovery - -/// Walks up from `inputURL` looking for a `Helpers/` directory. Returns the -/// first one found, or nil if no ancestor contains one. -/// -/// When `inputURL` is a file, the search starts from its parent. When it's a -/// directory, the search starts from the directory itself. -internal func discoverHelpersDir(near inputURL: URL) -> URL? { - let fm = FileManager.default - var isDirectory: ObjCBool = false - let exists = fm.fileExists(atPath: inputURL.path, isDirectory: &isDirectory) - var dir = (exists && isDirectory.boolValue) ? inputURL : inputURL.deletingLastPathComponent() - dir = dir.standardizedFileURL - - while true { - let candidate = dir.appendingPathComponent("Helpers") - var isDir: ObjCBool = false - if fm.fileExists(atPath: candidate.path, isDirectory: &isDir), isDir.boolValue { - return candidate.standardizedFileURL - } - let parent = dir.deletingLastPathComponent().standardizedFileURL - if parent.path == dir.path { return nil } - dir = parent +#if canImport(Subprocess) + + import Foundation + import Subprocess + + /// Hardcoded module name for the user's `Helpers/` compilation output. Inputs + /// reach the compiled helpers via `import SyntaxKitHelpers`. + internal let helpersModuleName = "SyntaxKitHelpers" + + /// Platform-specific shared-library filename for a Swift library product. + internal func dylibFilename(forLibrary name: String) -> String { + #if os(Linux) + return "lib\(name).so" + #else + return "lib\(name).dylib" + #endif } -} - -/// Globs `**/*.swift` under `helpersDir`, skipping files prefixed with `_`. -internal func collectHelperSources(in helpersDir: URL) throws -> [URL] { - guard - let enumerator = FileManager.default.enumerator( - at: helpersDir, - includingPropertiesForKeys: [.isRegularFileKey], - options: [.skipsHiddenFiles] - ) - else { - throw CLIError(message: "could not enumerate \(helpersDir.path)") + + /// Bumped when the cache layout changes in a way that requires invalidation. + private let helpersCacheSchemaVersion = "v1" + + /// A compiled `Helpers/` directory ready to splice into the input spawn. + internal struct CompiledHelpers: Sendable { + /// Directory containing `libSyntaxKitHelpers.dylib` + `.swiftmodule` files. + let outputDir: URL + /// Whether the build was reused from cache (false = freshly compiled). + let cacheHit: Bool } - var result: [URL] = [] - for case let url as URL in enumerator { - let values = try url.resourceValues(forKeys: [.isRegularFileKey]) - guard values.isRegularFile == true else { continue } - guard url.pathExtension == "swift" else { continue } - guard !url.lastPathComponent.hasPrefix("_") else { continue } - result.append(url.standardizedFileURL) + // MARK: - Discovery + + /// Walks up from `inputURL` looking for a `Helpers/` directory. Returns the + /// first one found, or nil if no ancestor contains one. + /// + /// When `inputURL` is a file, the search starts from its parent. When it's a + /// directory, the search starts from the directory itself. + internal func discoverHelpersDir(near inputURL: URL) -> URL? { + let fm = FileManager.default + var isDirectory: ObjCBool = false + let exists = fm.fileExists(atPath: inputURL.path, isDirectory: &isDirectory) + var dir = (exists && isDirectory.boolValue) ? inputURL : inputURL.deletingLastPathComponent() + dir = dir.standardizedFileURL + + while true { + let candidate = dir.appendingPathComponent("Helpers") + var isDir: ObjCBool = false + if fm.fileExists(atPath: candidate.path, isDirectory: &isDir), isDir.boolValue { + return candidate.standardizedFileURL + } + let parent = dir.deletingLastPathComponent().standardizedFileURL + if parent.path == dir.path { return nil } + dir = parent + } } - return result.sorted { $0.path < $1.path } -} - -// MARK: - Build pipeline - -/// Compiles helper sources into a per-key cache directory and returns the -/// directory plus a cache-hit flag. Returns nil when the helpers dir is empty. -internal func buildHelpers( - helpersDir: URL, - libPath: String -) throws -> CompiledHelpers? { - let sources = try collectHelperSources(in: helpersDir) - if sources.isEmpty { return nil } - - let key = try helpersCacheKey(sources: sources, libPath: libPath) - let cacheRoot = try syntaxKitCacheRoot() - .appendingPathComponent("helpers") - .appendingPathComponent(key) - let dylibPath = - cacheRoot - .appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path - - let fm = FileManager.default - if fm.fileExists(atPath: dylibPath) { - return CompiledHelpers(outputDir: cacheRoot, cacheHit: true) + + /// Globs `**/*.swift` under `helpersDir`, skipping files prefixed with `_`. + internal func collectHelperSources(in helpersDir: URL) throws -> [URL] { + guard + let enumerator = FileManager.default.enumerator( + at: helpersDir, + includingPropertiesForKeys: [.isRegularFileKey], + options: [.skipsHiddenFiles] + ) + else { + throw CLIError(message: "could not enumerate \(helpersDir.path)") + } + + var result: [URL] = [] + for case let url as URL in enumerator { + let values = try url.resourceValues(forKeys: [.isRegularFileKey]) + guard values.isRegularFile == true else { continue } + guard url.pathExtension == "swift" else { continue } + guard !url.lastPathComponent.hasPrefix("_") else { continue } + result.append(url.standardizedFileURL) + } + return result.sorted { $0.path < $1.path } } - try fm.createDirectory( - at: cacheRoot.deletingLastPathComponent(), - withIntermediateDirectories: true - ) + // MARK: - Build pipeline + + /// Compiles helper sources into a per-key cache directory and returns the + /// directory plus a cache-hit flag. Returns nil when the helpers dir is empty. + internal func buildHelpers( + helpersDir: URL, + libPath: String + ) async throws -> CompiledHelpers? { + let sources = try collectHelperSources(in: helpersDir) + if sources.isEmpty { return nil } + + let key = try await helpersCacheKey(sources: sources, libPath: libPath) + let cacheRoot = try syntaxKitCacheRoot() + .appendingPathComponent("helpers") + .appendingPathComponent(key) + let dylibPath = + cacheRoot + .appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path + + let fm = FileManager.default + if fm.fileExists(atPath: dylibPath) { + return CompiledHelpers(outputDir: cacheRoot, cacheHit: true) + } - let staging = cacheRoot.deletingLastPathComponent() - .appendingPathComponent("tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)") - try fm.createDirectory(at: staging, withIntermediateDirectories: true) + try fm.createDirectory( + at: cacheRoot.deletingLastPathComponent(), + withIntermediateDirectories: true + ) - do { - try compileHelpers(sources: sources, into: staging, libPath: libPath) - } catch { - try? fm.removeItem(at: staging) - throw error - } + let staging = cacheRoot.deletingLastPathComponent() + .appendingPathComponent( + "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)") + try fm.createDirectory(at: staging, withIntermediateDirectories: true) - // Atomic rename into the cache path. If a peer beat us to it (rename failed - // because the destination now exists), keep theirs and drop ours. - do { - try fm.moveItem(at: staging, to: cacheRoot) - } catch { - try? fm.removeItem(at: staging) - if !fm.fileExists(atPath: dylibPath) { + do { + try await compileHelpers(sources: sources, into: staging, libPath: libPath) + } catch { + try? fm.removeItem(at: staging) throw error } + + // Atomic rename into the cache path. If a peer beat us to it (rename failed + // because the destination now exists), keep theirs and drop ours. + do { + try fm.moveItem(at: staging, to: cacheRoot) + } catch { + try? fm.removeItem(at: staging) + if !fm.fileExists(atPath: dylibPath) { + throw error + } + } + + return CompiledHelpers(outputDir: cacheRoot, cacheHit: false) } - return CompiledHelpers(outputDir: cacheRoot, cacheHit: false) -} - -private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) throws { - let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - let dylib = outDir.appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path - let modulePath = outDir.appendingPathComponent("\(helpersModuleName).swiftmodule").path - - let process = Process() - process.executableURL = URL(fileURLWithPath: "/usr/bin/env") - var args: [String] = [ - "swiftc", - "-module-name", helpersModuleName, - "-emit-module", - "-emit-module-path", modulePath, - "-parse-as-library", - "-emit-library", - "-o", dylib, - "-suppress-warnings", - "-I", libPath, - "-L", libPath, - "-lSyntaxKit", - "-Xcc", "-I", "-Xcc", cShimsInclude, - "-Xlinker", "-rpath", "-Xlinker", libPath, - ] - #if !os(Linux) - // @rpath install_name is macOS-only; on Linux SONAME isn't needed because - // we use rpath-based loading and the dylib lives in a cache path that's - // known at link time. - args.append(contentsOf: [ - "-Xlinker", "-install_name", - "-Xlinker", "@rpath/\(dylibFilename(forLibrary: helpersModuleName))", - ]) - #endif - args.append(contentsOf: sources.map(\.path)) - process.arguments = args - - let stderrPipe = Pipe() - process.standardOutput = FileHandle.nullDevice - process.standardError = stderrPipe - - // Linux Foundation's `Process.waitUntilExit()` blocks indefinitely on - // already-exited children in some configurations; terminationHandler + - // semaphore is the workaround used elsewhere in this file. - let semaphore = DispatchSemaphore(value: 0) - process.terminationHandler = { _ in semaphore.signal() } - - try process.run() - - // Drain stderr BEFORE waiting on the semaphore — Linux pipe buffers are - // ~64 KB; if the child fills them we deadlock waiting for an exit that - // can't happen until we read. - let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() - semaphore.wait() - guard process.terminationStatus == 0 else { - let stderr = String(decoding: stderrData, as: UTF8.self) - throw CLIError( - message: """ - skit: failed to compile Helpers/ (exit \(process.terminationStatus)) - \(stderr) - """) + private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) async throws { + let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + let dylib = outDir.appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path + let modulePath = outDir.appendingPathComponent("\(helpersModuleName).swiftmodule").path + + var args: [String] = [ + "-module-name", helpersModuleName, + "-emit-module", + "-emit-module-path", modulePath, + "-parse-as-library", + "-emit-library", + "-o", dylib, + "-suppress-warnings", + "-I", libPath, + "-L", libPath, + "-lSyntaxKit", + "-Xcc", "-I", "-Xcc", cShimsInclude, + "-Xlinker", "-rpath", "-Xlinker", libPath, + ] + #if !os(Linux) + // @rpath install_name is macOS-only; on Linux SONAME isn't needed because + // we use rpath-based loading and the dylib lives in a cache path that's + // known at link time. + args.append(contentsOf: [ + "-Xlinker", "-install_name", + "-Xlinker", "@rpath/\(dylibFilename(forLibrary: helpersModuleName))", + ]) + #endif + args.append(contentsOf: sources.map(\.path)) + + let result = try await run( + .name("swiftc"), + arguments: Arguments(args), + output: .discarded, + error: .string(limit: 1 * 1_024 * 1_024) + ) + + guard result.terminationStatus.isSuccess else { + let stderr = result.standardError ?? "" + throw CLIError( + message: """ + skit: failed to compile Helpers/ (\(result.terminationStatus)) + \(stderr) + """) + } } -} -// MARK: - Cache key + // MARK: - Cache key -private func helpersCacheKey(sources: [URL], libPath: String) throws -> String { - var hasher = ContentHasher() - hasher.update(data: Data(helpersCacheSchemaVersion.utf8)) + private func helpersCacheKey(sources: [URL], libPath: String) async throws -> String { + var hasher = ContentHasher() + hasher.update(data: Data(helpersCacheSchemaVersion.utf8)) - for source in sources { - let data = try Data(contentsOf: source) - hasher.update(data: Data(source.lastPathComponent.utf8)) - hasher.update(data: data) + for source in sources { + let data = try Data(contentsOf: source) + hasher.update(data: Data(source.lastPathComponent.utf8)) + hasher.update(data: data) + } + + if let swiftVersion = await captureSwiftVersion() { + hasher.update(data: Data(swiftVersion.utf8)) + } + if let stamp = libStamp(libPath: libPath) { + hasher.update(data: Data(stamp.utf8)) + } + + return hasher.finalize() } - if let swiftVersion = captureSwiftVersion() { - hasher.update(data: Data(swiftVersion.utf8)) + internal func captureSwiftVersion() async -> String? { + let result = try? await run( + .name("swift"), + arguments: ["--version"], + output: .string(limit: 4_096), + error: .discarded + ) + return result?.standardOutput } - if let stamp = libStamp(libPath: libPath) { - hasher.update(data: Data(stamp.utf8)) + + internal func libStamp(libPath: String) -> String? { + let dylib = "\(libPath)/\(dylibFilename(forLibrary: "SyntaxKit"))" + guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } + let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 + let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 + return "\(size)/\(Int(mtime))" } - return hasher.finalize() -} - -internal func captureSwiftVersion() -> String? { - let process = Process() - process.executableURL = URL(fileURLWithPath: "/usr/bin/env") - process.arguments = ["swift", "--version"] - let pipe = Pipe() - process.standardOutput = pipe - process.standardError = FileHandle.nullDevice - let semaphore = DispatchSemaphore(value: 0) - process.terminationHandler = { _ in semaphore.signal() } - do { try process.run() } catch { return nil } - let data = pipe.fileHandleForReading.readDataToEndOfFile() - semaphore.wait() - return String(decoding: data, as: UTF8.self) -} - -internal func libStamp(libPath: String) -> String? { - let dylib = "\(libPath)/\(dylibFilename(forLibrary: "SyntaxKit"))" - guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } - let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 - let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 - return "\(size)/\(Int(mtime))" -} - -internal func syntaxKitCacheRoot() throws -> URL { - if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { - return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") + internal func syntaxKitCacheRoot() throws -> URL { + if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { + return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") + } + let home = NSHomeDirectory() + #if os(macOS) + return URL(fileURLWithPath: home) + .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") + #else + return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") + #endif } - let home = NSHomeDirectory() - #if os(macOS) - return URL(fileURLWithPath: home) - .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") - #else - return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") - #endif -} + +#endif diff --git a/Sources/skit/OutputCache.swift b/Sources/skit/OutputCache.swift index c555702..47f7972 100644 --- a/Sources/skit/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -27,85 +27,89 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation +#if canImport(Subprocess) -/// Bumped when the output cache layout changes in a way that requires invalidation. -private let outputCacheSchemaVersion = "v1" + import Foundation -/// 64-bit content hash over (cache schema, input source bytes, helpers key, -/// swift version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). -/// Any change in these inputs produces a fresh key and forces a recompile. -/// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. -internal func outputCacheKey( - inputSource: String, - helpers: CompiledHelpers?, - libPath: String -) -> String { - var hasher = ContentHasher() - hasher.update(data: Data(outputCacheSchemaVersion.utf8)) - hasher.update(data: Data(inputSource.utf8)) + /// Bumped when the output cache layout changes in a way that requires invalidation. + private let outputCacheSchemaVersion = "v1" - if let helpers { - // Helpers cache dir name *is* the helpers cache key (per Helpers.swift). - hasher.update(data: Data(helpers.outputDir.lastPathComponent.utf8)) - } else { - hasher.update(data: Data("no-helpers".utf8)) - } + /// 64-bit content hash over (cache schema, input source bytes, helpers key, + /// swift version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). + /// Any change in these inputs produces a fresh key and forces a recompile. + /// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. + internal func outputCacheKey( + inputSource: String, + helpers: CompiledHelpers?, + libPath: String + ) async -> String { + var hasher = ContentHasher() + hasher.update(data: Data(outputCacheSchemaVersion.utf8)) + hasher.update(data: Data(inputSource.utf8)) - if let version = captureSwiftVersion() { - hasher.update(data: Data(version.utf8)) - } - if let stamp = libStamp(libPath: libPath) { - hasher.update(data: Data(stamp.utf8)) - } + if let helpers { + // Helpers cache dir name *is* the helpers cache key (per Helpers.swift). + hasher.update(data: Data(helpers.outputDir.lastPathComponent.utf8)) + } else { + hasher.update(data: Data("no-helpers".utf8)) + } - let env = ProcessInfo.processInfo.environment - .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } - .sorted { $0.key < $1.key } - for (key, value) in env { - hasher.update(data: Data("\(key)=\(value)\0".utf8)) - } + if let version = await captureSwiftVersion() { + hasher.update(data: Data(version.utf8)) + } + if let stamp = libStamp(libPath: libPath) { + hasher.update(data: Data(stamp.utf8)) + } - return hasher.finalize() -} + let env = ProcessInfo.processInfo.environment + .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } + .sorted { $0.key < $1.key } + for (key, value) in env { + hasher.update(data: Data("\(key)=\(value)\0".utf8)) + } -/// Returns the cached rendered output for `key`, or nil on miss. -internal func lookupCachedOutput(key: String) -> Data? { - guard let dir = try? outputCacheDir(for: key) else { return nil } - return try? Data(contentsOf: dir.appendingPathComponent("output.swift")) -} + return hasher.finalize() + } -/// Atomically stores `data` under `key`. Concurrent writers race via a -/// `tmp../` staging dir + rename; the loser drops their copy. -internal func storeCachedOutput(key: String, data: Data) throws { - let cacheRoot = try outputCacheDir(for: key) - let final = cacheRoot.appendingPathComponent("output.swift") - let fm = FileManager.default + /// Returns the cached rendered output for `key`, or nil on miss. + internal func lookupCachedOutput(key: String) -> Data? { + guard let dir = try? outputCacheDir(for: key) else { return nil } + return try? Data(contentsOf: dir.appendingPathComponent("output.swift")) + } - try fm.createDirectory( - at: cacheRoot.deletingLastPathComponent(), - withIntermediateDirectories: true - ) + /// Atomically stores `data` under `key`. Concurrent writers race via a + /// `tmp../` staging dir + rename; the loser drops their copy. + internal func storeCachedOutput(key: String, data: Data) throws { + let cacheRoot = try outputCacheDir(for: key) + let final = cacheRoot.appendingPathComponent("output.swift") + let fm = FileManager.default - let staging = cacheRoot.deletingLastPathComponent() - .appendingPathComponent( - "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)" + try fm.createDirectory( + at: cacheRoot.deletingLastPathComponent(), + withIntermediateDirectories: true ) - try fm.createDirectory(at: staging, withIntermediateDirectories: true) - try data.write(to: staging.appendingPathComponent("output.swift")) - do { - try fm.moveItem(at: staging, to: cacheRoot) - } catch { - try? fm.removeItem(at: staging) - if !fm.fileExists(atPath: final.path) { - throw error + let staging = cacheRoot.deletingLastPathComponent() + .appendingPathComponent( + "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)" + ) + try fm.createDirectory(at: staging, withIntermediateDirectories: true) + try data.write(to: staging.appendingPathComponent("output.swift")) + + do { + try fm.moveItem(at: staging, to: cacheRoot) + } catch { + try? fm.removeItem(at: staging) + if !fm.fileExists(atPath: final.path) { + throw error + } } } -} -private func outputCacheDir(for key: String) throws -> URL { - try syntaxKitCacheRoot() - .appendingPathComponent("outputs") - .appendingPathComponent(key) -} + private func outputCacheDir(for key: String) throws -> URL { + try syntaxKitCacheRoot() + .appendingPathComponent("outputs") + .appendingPathComponent(key) + } + +#endif diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index 957df18..d9407f4 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -27,585 +27,578 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation -import SwiftParser -import SwiftSyntax - -// MARK: - Helpers resolution - -internal enum HelpersOptions { - case auto - case disabled - case explicit(String) -} - -internal func resolveHelpers( - nearInputPath path: String, - libPath: String, - options: HelpersOptions -) throws -> CompiledHelpers? { - let helpersDir: URL? - switch options { - case .disabled: - return nil - case .auto: - helpersDir = discoverHelpersDir(near: URL(fileURLWithPath: path).standardizedFileURL) - case .explicit(let dir): - let url = URL(fileURLWithPath: dir).standardizedFileURL - var isDir: ObjCBool = false - guard FileManager.default.fileExists(atPath: url.path, isDirectory: &isDir), - isDir.boolValue - else { - throw CLIError(message: "--helpers path is not a directory: \(dir)") - } - helpersDir = url - } - guard let helpersDir else { return nil } +#if canImport(Subprocess) + + import Foundation + import Subprocess + import SwiftParser + import SwiftSyntax - guard let compiled = try buildHelpers(helpersDir: helpersDir, libPath: libPath) else { - return nil + // MARK: - Helpers resolution + + internal enum HelpersOptions { + case auto + case disabled + case explicit(String) } - let suffix = compiled.cacheHit ? "cached" : "compiled" - FileHandle.standardError.write( - Data( - "skit: helpers \(suffix) at \(helpersDir.path)\n".utf8 - )) - return compiled -} - -// MARK: - Resource location - -/// Resolves the directory containing `libSyntaxKit.dylib` + module files, -/// in priority order: explicit flag → env var → adjacent-to-binary -/// (`/lib/`) → Homebrew layout (`/../lib/skit/`). -internal func resolveLibPath(override: String?) throws -> String { - if let override { - guard isLibDir(override) else { - throw CLIError(message: "--lib path does not look like a SyntaxKit lib dir: \(override)") + + internal func resolveHelpers( + nearInputPath path: String, + libPath: String, + options: HelpersOptions + ) async throws -> CompiledHelpers? { + let helpersDir: URL? + switch options { + case .disabled: + return nil + case .auto: + helpersDir = discoverHelpersDir(near: URL(fileURLWithPath: path).standardizedFileURL) + case .explicit(let dir): + let url = URL(fileURLWithPath: dir).standardizedFileURL + var isDir: ObjCBool = false + guard FileManager.default.fileExists(atPath: url.path, isDirectory: &isDir), + isDir.boolValue + else { + throw CLIError(message: "--helpers path is not a directory: \(dir)") + } + helpersDir = url } - return override - } + guard let helpersDir else { return nil } - if let env = ProcessInfo.processInfo.environment["SKIT_LIB_DIR"], !env.isEmpty { - guard isLibDir(env) else { - throw CLIError(message: "SKIT_LIB_DIR is set but path is not a lib dir: \(env)") + guard let compiled = try await buildHelpers(helpersDir: helpersDir, libPath: libPath) else { + return nil } - return env + let suffix = compiled.cacheHit ? "cached" : "compiled" + FileHandle.standardError.write( + Data( + "skit: helpers \(suffix) at \(helpersDir.path)\n".utf8 + )) + return compiled } - if let execURL = Bundle.main.executableURL?.resolvingSymlinksInPath() { - let execDir = execURL.deletingLastPathComponent() + // MARK: - Resource location - let adjacent = execDir.appendingPathComponent("lib").path - if isLibDir(adjacent) { return adjacent } + /// Resolves the directory containing `libSyntaxKit.dylib` + module files, + /// in priority order: explicit flag → env var → adjacent-to-binary + /// (`/lib/`) → Homebrew layout (`/../lib/skit/`). + internal func resolveLibPath(override: String?) throws -> String { + if let override { + guard isLibDir(override) else { + throw CLIError(message: "--lib path does not look like a SyntaxKit lib dir: \(override)") + } + return override + } - let brewLayout = execDir.deletingLastPathComponent() - .appendingPathComponent("lib/skit").path - if isLibDir(brewLayout) { return brewLayout } - } + if let env = ProcessInfo.processInfo.environment["SKIT_LIB_DIR"], !env.isEmpty { + guard isLibDir(env) else { + throw CLIError(message: "SKIT_LIB_DIR is set but path is not a lib dir: \(env)") + } + return env + } - throw CLIError( - message: """ - Could not locate SyntaxKit lib directory. Looked for: - 1. --lib (not provided) - 2. $SKIT_LIB_DIR (not set) - 3. /lib/ (not found) - 4. /../lib/skit/ (not found) - Run Scripts/build-skit-release.sh to produce a self-contained - release bundle under .build/skit-release/. - """) -} - -private func isLibDir(_ path: String) -> Bool { - let fm = FileManager.default - var isDir: ObjCBool = false - guard fm.fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { return false } - return fm.fileExists(atPath: "\(path)/\(dylibFilename(forLibrary: "SyntaxKit"))") -} - -// MARK: - Toolchain check - -/// Filename for the bundle's recorded build-toolchain version. -internal let toolchainStampFilename = "swift-version.txt" - -internal enum ToolchainCheckResult { - /// Bundle stamp matches the local `swift --version` exactly. - case match - /// `/swift-version.txt` is missing (older bundle that predates - /// the stamp). skit prints a one-line note and proceeds. - case stampMissing - case mismatch(bundle: String, local: String) -} - -/// Compares `/swift-version.txt` to `captureSwiftVersion()`. -/// The swiftmodule format isn't reliably forward-compatible across even -/// patch-level Swift releases (originating bug: 6.3.0 → 6.3.2 rejected the -/// swiftmodule), so the comparison is exact-string after normalising -/// trailing whitespace. -internal func toolchainCheck(libPath: String) -> ToolchainCheckResult { - let stampURL = URL(fileURLWithPath: libPath).appendingPathComponent(toolchainStampFilename) - guard let stampData = try? Data(contentsOf: stampURL), - let stampRaw = String(data: stampData, encoding: .utf8) - else { - FileHandle.standardError.write( - Data("skit: bundle has no toolchain stamp; skipping check\n".utf8) - ) - return .stampMissing - } - guard let localRaw = captureSwiftVersion() else { - FileHandle.standardError.write( - Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) - ) - return .stampMissing - } - let bundle = stampRaw.trimmingCharacters(in: .whitespacesAndNewlines) - let local = localRaw.trimmingCharacters(in: .whitespacesAndNewlines) - return bundle == local ? .match : .mismatch(bundle: bundle, local: local) -} - -internal func toolchainMismatchMessage(bundle: String, local: String) -> String { - """ - skit: toolchain mismatch - bundle: \(bundle) - local: \(local) - The bundle's libSyntaxKit was built against a different `swift` than the - one on your PATH. Swift swiftmodules aren't reliably compatible across - versions, so spawning `swift` would fail with a cryptic module-version - diagnostic. - - Rebuild the bundle with: - Scripts/build-skit-release.sh - Or pass --no-toolchain-check to try anyway. - - """ -} - -// MARK: - Single-file mode - -internal func runSingleFile( - inputPath: String, - outputPath: String?, - libPath: String, - helpers: CompiledHelpers?, - useCache: Bool, - timeoutSeconds: Int -) throws { - let result = try processFile( - inputPath: inputPath, - libPath: libPath, - helpers: helpers, - useCache: useCache, - timeoutSeconds: timeoutSeconds - ) - if !result.stderr.isEmpty { - FileHandle.standardError.write(Data(result.stderr.utf8)) + if let execURL = Bundle.main.executableURL?.resolvingSymlinksInPath() { + let execDir = execURL.deletingLastPathComponent() + + let adjacent = execDir.appendingPathComponent("lib").path + if isLibDir(adjacent) { return adjacent } + + let brewLayout = execDir.deletingLastPathComponent() + .appendingPathComponent("lib/skit").path + if isLibDir(brewLayout) { return brewLayout } + } + + throw CLIError( + message: """ + Could not locate SyntaxKit lib directory. Looked for: + 1. --lib (not provided) + 2. $SKIT_LIB_DIR (not set) + 3. /lib/ (not found) + 4. /../lib/skit/ (not found) + Run Scripts/build-skit-release.sh to produce a self-contained + release bundle under .build/skit-release/. + """) } - guard result.exitCode == 0 else { - exit(result.exitCode) + + private func isLibDir(_ path: String) -> Bool { + let fm = FileManager.default + var isDir: ObjCBool = false + guard fm.fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { return false } + return fm.fileExists(atPath: "\(path)/\(dylibFilename(forLibrary: "SyntaxKit"))") } - if let outputPath { - try result.stdout.write(to: URL(fileURLWithPath: outputPath)) - } else { - FileHandle.standardOutput.write(result.stdout) + + // MARK: - Toolchain check + + /// Filename for the bundle's recorded build-toolchain version. + internal let toolchainStampFilename = "swift-version.txt" + + internal enum ToolchainCheckResult { + /// Bundle stamp matches the local `swift --version` exactly. + case match + /// `/swift-version.txt` is missing (older bundle that predates + /// the stamp). skit prints a one-line note and proceeds. + case stampMissing + case mismatch(bundle: String, local: String) } -} - -// MARK: - Folder mode - -internal func runDirectory( - inputDir: String, - outputDir: String, - libPath: String, - helpers: CompiledHelpers?, - useCache: Bool, - timeoutSeconds: Int -) async -> Int32 { - let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL - let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL - - let inputs: [URL] - do { - inputs = try collectInputs(at: inputURL, excluding: helpersExcludePath(inputDir: inputURL)) - } catch { - FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) - return 1 + + /// Compares `/swift-version.txt` to `captureSwiftVersion()`. + /// The swiftmodule format isn't reliably forward-compatible across even + /// patch-level Swift releases (originating bug: 6.3.0 → 6.3.2 rejected the + /// swiftmodule), so the comparison is exact-string after normalising + /// trailing whitespace. + internal func toolchainCheck(libPath: String) async -> ToolchainCheckResult { + let stampURL = URL(fileURLWithPath: libPath).appendingPathComponent(toolchainStampFilename) + guard let stampData = try? Data(contentsOf: stampURL), + let stampRaw = String(data: stampData, encoding: .utf8) + else { + FileHandle.standardError.write( + Data("skit: bundle has no toolchain stamp; skipping check\n".utf8) + ) + return .stampMissing + } + guard let localRaw = await captureSwiftVersion() else { + FileHandle.standardError.write( + Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) + ) + return .stampMissing + } + let bundle = stampRaw.trimmingCharacters(in: .whitespacesAndNewlines) + let local = localRaw.trimmingCharacters(in: .whitespacesAndNewlines) + return bundle == local ? .match : .mismatch(bundle: bundle, local: local) } - if inputs.isEmpty { - FileHandle.standardError.write(Data("skit: no .swift inputs under \(inputDir)\n".utf8)) - return 0 + internal func toolchainMismatchMessage(bundle: String, local: String) -> String { + """ + skit: toolchain mismatch + bundle: \(bundle) + local: \(local) + The bundle's libSyntaxKit was built against a different `swift` than the + one on your PATH. Swift swiftmodules aren't reliably compatible across + versions, so spawning `swift` would fail with a cryptic module-version + diagnostic. + + Rebuild the bundle with: + Scripts/build-skit-release.sh + Or pass --no-toolchain-check to try anyway. + + """ } - let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) + // MARK: - Single-file mode + + internal func runSingleFile( + inputPath: String, + outputPath: String?, + libPath: String, + helpers: CompiledHelpers?, + useCache: Bool, + timeoutSeconds: Int + ) async throws { + let result = try await processFile( + inputPath: inputPath, + libPath: libPath, + helpers: helpers, + useCache: useCache, + timeoutSeconds: timeoutSeconds + ) + if !result.stderr.isEmpty { + FileHandle.standardError.write(Data(result.stderr.utf8)) + } + guard result.exitCode == 0 else { + exit(result.exitCode) + } + if let outputPath { + try result.stdout.write(to: URL(fileURLWithPath: outputPath)) + } else { + FileHandle.standardOutput.write(result.stdout) + } + } - var outcomes: [FileOutcome] = [] - var iterator = inputs.makeIterator() + // MARK: - Folder mode + + internal func runDirectory( + inputDir: String, + outputDir: String, + libPath: String, + helpers: CompiledHelpers?, + useCache: Bool, + timeoutSeconds: Int + ) async -> Int32 { + let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL + let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL + + let inputs: [URL] + do { + inputs = try collectInputs(at: inputURL, excluding: helpersExcludePath(inputDir: inputURL)) + } catch { + FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) + return 1 + } - await withTaskGroup(of: FileOutcome.self) { group in - for _ in 0.. -} - -private func runOne( - _ input: URL, - libPath: String, - helpers: CompiledHelpers?, - useCache: Bool, - timeoutSeconds: Int -) -> FileOutcome { - do { - let result = try processFile( - inputPath: input.path, - libPath: libPath, - helpers: helpers, - useCache: useCache, - timeoutSeconds: timeoutSeconds - ) - return FileOutcome(input: input, result: .success(result)) - } catch { - return FileOutcome(input: input, result: .failure(error)) + private struct FileOutcome: Sendable { + let input: URL + let result: Result } -} - -/// Returns the path of a `Helpers/` directory living directly under `inputDir`, -/// so the folder-mode enumerator can skip its descendants. Helpers that live -/// outside the input tree don't need to be excluded (they aren't enumerated). -private func helpersExcludePath(inputDir: URL) -> String? { - let candidate = inputDir.appendingPathComponent("Helpers").standardizedFileURL - var isDir: ObjCBool = false - guard FileManager.default.fileExists(atPath: candidate.path, isDirectory: &isDir), - isDir.boolValue - else { - return nil + + private func runOne( + _ input: URL, + libPath: String, + helpers: CompiledHelpers?, + useCache: Bool, + timeoutSeconds: Int + ) async -> FileOutcome { + do { + let result = try await processFile( + inputPath: input.path, + libPath: libPath, + helpers: helpers, + useCache: useCache, + timeoutSeconds: timeoutSeconds + ) + return FileOutcome(input: input, result: .success(result)) + } catch { + return FileOutcome(input: input, result: .failure(error)) + } } - return candidate.path -} - -private func collectInputs(at inputDir: URL, excluding excludedDir: String?) throws -> [URL] { - guard - let enumerator = FileManager.default.enumerator( - at: inputDir, - includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], - options: [.skipsHiddenFiles] - ) - else { - throw CLIError(message: "could not enumerate \(inputDir.path)") + + /// Returns the path of a `Helpers/` directory living directly under `inputDir`, + /// so the folder-mode enumerator can skip its descendants. Helpers that live + /// outside the input tree don't need to be excluded (they aren't enumerated). + private func helpersExcludePath(inputDir: URL) -> String? { + let candidate = inputDir.appendingPathComponent("Helpers").standardizedFileURL + var isDir: ObjCBool = false + guard FileManager.default.fileExists(atPath: candidate.path, isDirectory: &isDir), + isDir.boolValue + else { + return nil + } + return candidate.path } - var result: [URL] = [] - for case let url as URL in enumerator { - let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) - if values.isDirectory == true { - if let excludedDir, url.standardizedFileURL.path == excludedDir { - enumerator.skipDescendants() + private func collectInputs(at inputDir: URL, excluding excludedDir: String?) throws -> [URL] { + guard + let enumerator = FileManager.default.enumerator( + at: inputDir, + includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], + options: [.skipsHiddenFiles] + ) + else { + throw CLIError(message: "could not enumerate \(inputDir.path)") + } + + var result: [URL] = [] + for case let url as URL in enumerator { + let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + if values.isDirectory == true { + if let excludedDir, url.standardizedFileURL.path == excludedDir { + enumerator.skipDescendants() + } + continue } - continue + guard values.isRegularFile == true else { continue } + guard url.pathExtension == "swift" else { continue } + guard !url.lastPathComponent.hasPrefix("_") else { continue } + result.append(url.standardizedFileURL) } - guard values.isRegularFile == true else { continue } - guard url.pathExtension == "swift" else { continue } - guard !url.lastPathComponent.hasPrefix("_") else { continue } - result.append(url.standardizedFileURL) - } - return result.sorted { $0.path < $1.path } -} - -// MARK: - Per-file work - -private struct ProcessResult { - let exitCode: Int32 - let stdout: Data - let stderr: String -} - -private func processFile( - inputPath: String, - libPath: String, - helpers: CompiledHelpers?, - useCache: Bool, - timeoutSeconds: Int -) throws -> ProcessResult { - let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL - let absoluteInputPath = inputURL.path - let source = try String(contentsOf: inputURL, encoding: .utf8) - - let cacheKey: String? = - useCache - ? outputCacheKey(inputSource: source, helpers: helpers, libPath: libPath) - : nil - if let cacheKey, let cached = lookupCachedOutput(key: cacheKey) { - return ProcessResult(exitCode: 0, stdout: cached, stderr: "") + return result.sorted { $0.path < $1.path } } - let wrapped = wrap(source: source, originalPath: absoluteInputPath) - - let tmpDir = FileManager.default.temporaryDirectory - .appendingPathComponent("skit-\(UUID().uuidString)") - try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) - defer { try? FileManager.default.removeItem(at: tmpDir) } - - let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") - try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) - - let raw = try runSwift( - wrappedPath: wrappedURL.path, - libPath: libPath, - helpers: helpers, - timeoutSeconds: timeoutSeconds - ) - // #sourceLocation maps body diagnostics back to the input file. Errors in - // the preamble (lines outside the body) still reference the wrapper — - // rewrite literal occurrences of its path so users see something coherent. - let stderr = raw.stderr.replacingOccurrences( - of: wrappedURL.path, - with: absoluteInputPath - ) - - if let cacheKey, raw.exitCode == 0 { - try? storeCachedOutput(key: cacheKey, data: raw.stdout) + // MARK: - Per-file work + + private struct ProcessResult: Sendable { + let exitCode: Int32 + let stdout: Data + let stderr: String } - return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) -} - -internal struct CLIError: Error, CustomStringConvertible { - let message: String - var description: String { message } -} - -// MARK: - Wrapping - -/// Splits the input into hoisted `import` declarations and a verbatim body, -/// returning a complete Swift program that runs SyntaxKit on the body. -/// -/// The body is fenced in `#sourceLocation` directives so compiler diagnostics -/// in the body reference the original input file and line numbers. -internal func wrap(source: String, originalPath: String) -> String { - let tree = Parser.parse(source: source) - let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) - - // Find the first non-import top-level statement; everything before it that - // is an import gets hoisted, anything before that which is *not* an import - // stays in the body (e.g. a top-level `// comment` is left alone). - var hoisted: [String] = [] - var firstBodyByte: AbsolutePosition? - - for item in tree.statements { - if let importDecl = item.item.as(ImportDeclSyntax.self), - firstBodyByte == nil - { - hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) - continue + private func processFile( + inputPath: String, + libPath: String, + helpers: CompiledHelpers?, + useCache: Bool, + timeoutSeconds: Int + ) async throws -> ProcessResult { + let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL + let absoluteInputPath = inputURL.path + let source = try String(contentsOf: inputURL, encoding: .utf8) + + let cacheKey: String? = + useCache + ? await outputCacheKey(inputSource: source, helpers: helpers, libPath: libPath) + : nil + if let cacheKey, let cached = lookupCachedOutput(key: cacheKey) { + return ProcessResult(exitCode: 0, stdout: cached, stderr: "") } - firstBodyByte = item.position - break - } - let body: String - let firstBodyLine: Int - if let firstBodyByte { - let start = source.utf8.index(source.utf8.startIndex, offsetBy: firstBodyByte.utf8Offset) - body = String(source[start...]) - firstBodyLine = locConverter.location(for: firstBodyByte).line - } else { - body = "" - firstBodyLine = 1 - } + let wrapped = wrap(source: source, originalPath: absoluteInputPath) + + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("skit-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") + try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) + + let raw = try await runSwift( + wrappedPath: wrappedURL.path, + libPath: libPath, + helpers: helpers, + timeoutSeconds: timeoutSeconds + ) + // #sourceLocation maps body diagnostics back to the input file. Errors in + // the preamble (lines outside the body) still reference the wrapper — + // rewrite literal occurrences of its path so users see something coherent. + let stderr = raw.stderr.replacingOccurrences( + of: wrappedURL.path, + with: absoluteInputPath + ) - let hoistedBlock = hoisted.isEmpty ? "" : hoisted.joined(separator: "\n") + "\n" - - // #sourceLocation must use a forward-slash path; escape backslashes/quotes - // defensively even though macOS paths shouldn't contain them. - let escapedPath = - originalPath - .replacingOccurrences(of: "\\", with: "\\\\") - .replacingOccurrences(of: "\"", with: "\\\"") - - return """ - import SyntaxKit - \(hoistedBlock) - let __skit_root = Group { - #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) - \(body) - #sourceLocation() + if let cacheKey, raw.exitCode == 0 { + try? storeCachedOutput(key: cacheKey, data: raw.stdout) } - print(__skit_root.generateCode()) - """ -} - -// MARK: - Spawning swift - -/// Exit code returned when the spawned `swift` is killed by skit's timeout -/// watchdog. Matches POSIX `timeout(1)`. -private let timeoutExitCode: Int32 = 124 - -/// Grace period between SIGTERM and SIGKILL when the child won't exit on its own. -private let killGraceSeconds: Int = 5 - -private func runSwift( - wrappedPath: String, - libPath: String, - helpers: CompiledHelpers?, - timeoutSeconds: Int -) throws -> ProcessResult { - let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - - var arguments: [String] = [ - "swift", - "-suppress-warnings", - "-I", libPath, - "-L", libPath, - "-lSyntaxKit", - "-Xcc", "-I", "-Xcc", cShimsInclude, - "-Xlinker", "-rpath", "-Xlinker", libPath, - ] - - if let helpers { - let helpersPath = helpers.outputDir.path - arguments.append(contentsOf: [ - "-I", helpersPath, - "-L", helpersPath, - "-l\(helpersModuleName)", - "-Xlinker", "-rpath", "-Xlinker", helpersPath, - ]) + return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } - arguments.append(wrappedPath) - - let process = Process() - process.executableURL = URL(fileURLWithPath: "/usr/bin/env") - process.arguments = arguments - - let stdoutPipe = Pipe() - let stderrPipe = Pipe() - process.standardOutput = stdoutPipe - process.standardError = stderrPipe - - // Linux Foundation's `Process.waitUntilExit()` blocks indefinitely on - // already-exited children in some configurations; terminationHandler + - // semaphore is the workaround. - let exitSemaphore = DispatchSemaphore(value: 0) - process.terminationHandler = { _ in exitSemaphore.signal() } - - try process.run() - - // Drain both pipes concurrently — reading sequentially deadlocks on Linux - // when either pipe (~64 KB buffer) fills before the child exits. Box the - // buffers in classes so Swift 6 strict-concurrency is satisfied without - // `@unchecked Sendable` on local vars. - let outBox = PipeDataBox() - let errBox = PipeDataBox() - let group = DispatchGroup() - group.enter() - DispatchQueue.global().async { - outBox.value = stdoutPipe.fileHandleForReading.readDataToEndOfFile() - group.leave() - } - group.enter() - DispatchQueue.global().async { - errBox.value = stderrPipe.fileHandleForReading.readDataToEndOfFile() - group.leave() + internal struct CLIError: Error, CustomStringConvertible { + let message: String + var description: String { message } } - // Timeout watchdog: wait for the child with a deadline. On expiry, send - // SIGTERM, give a fixed grace, then SIGKILL. timeoutSeconds == 0 disables. - let timedOut: Bool - if timeoutSeconds > 0 { - let deadline: DispatchTime = .now() + .seconds(timeoutSeconds) - if exitSemaphore.wait(timeout: deadline) == .timedOut { - process.terminate() // SIGTERM - if exitSemaphore.wait(timeout: .now() + .seconds(killGraceSeconds)) == .timedOut { - kill(process.processIdentifier, SIGKILL) - exitSemaphore.wait() + // MARK: - Wrapping + + /// Splits the input into hoisted `import` declarations and a verbatim body, + /// returning a complete Swift program that runs SyntaxKit on the body. + /// + /// The body is fenced in `#sourceLocation` directives so compiler diagnostics + /// in the body reference the original input file and line numbers. + internal func wrap(source: String, originalPath: String) -> String { + let tree = Parser.parse(source: source) + let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) + + // Find the first non-import top-level statement; everything before it that + // is an import gets hoisted, anything before that which is *not* an import + // stays in the body (e.g. a top-level `// comment` is left alone). + var hoisted: [String] = [] + var firstBodyByte: AbsolutePosition? + + for item in tree.statements { + if let importDecl = item.item.as(ImportDeclSyntax.self), + firstBodyByte == nil + { + hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) + continue } - timedOut = true + firstBodyByte = item.position + break + } + + let body: String + let firstBodyLine: Int + if let firstBodyByte { + let start = source.utf8.index(source.utf8.startIndex, offsetBy: firstBodyByte.utf8Offset) + body = String(source[start...]) + firstBodyLine = locConverter.location(for: firstBodyByte).line } else { - timedOut = false + body = "" + firstBodyLine = 1 } - } else { - exitSemaphore.wait() - timedOut = false + + let hoistedBlock = hoisted.isEmpty ? "" : hoisted.joined(separator: "\n") + "\n" + + // #sourceLocation must use a forward-slash path; escape backslashes/quotes + // defensively even though macOS paths shouldn't contain them. + let escapedPath = + originalPath + .replacingOccurrences(of: "\\", with: "\\\\") + .replacingOccurrences(of: "\"", with: "\\\"") + + return """ + import SyntaxKit + \(hoistedBlock) + let __skit_root = Group { + #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) + \(body) + #sourceLocation() + } + + print(__skit_root.generateCode()) + """ } - // Child is dead now — pipes get EOF, drain completes shortly. - group.wait() - if timedOut { - let prefix = Data("skit: timed out after \(timeoutSeconds)s\n".utf8) - let stderr = String(decoding: prefix + errBox.value, as: UTF8.self) - return ProcessResult(exitCode: timeoutExitCode, stdout: outBox.value, stderr: stderr) + // MARK: - Spawning swift + + /// Exit code returned when the spawned `swift` is killed by skit's timeout + /// watchdog. Matches POSIX `timeout(1)`. + private let timeoutExitCode: Int32 = 124 + + /// Bounded output capacity for the spawned `swift` (16 MiB). Generated DSL + /// output above this size is exotic; if we ever hit it we'll see a clear + /// SubprocessError rather than a silent truncation. + private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024 + private let stderrLimitBytes: Int = 1 * 1_024 * 1_024 + + private enum SwiftRunOutcome: Sendable { + case completed(exitCode: Int32, stdout: Data, stderr: String) + case timedOut } - return ProcessResult( - exitCode: process.terminationStatus, - stdout: outBox.value, - stderr: String(decoding: errBox.value, as: UTF8.self) - ) -} + private func runSwift( + wrappedPath: String, + libPath: String, + helpers: CompiledHelpers?, + timeoutSeconds: Int + ) async throws -> ProcessResult { + let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + + var arguments: [String] = [ + "-suppress-warnings", + "-I", libPath, + "-L", libPath, + "-lSyntaxKit", + "-Xcc", "-I", "-Xcc", cShimsInclude, + "-Xlinker", "-rpath", "-Xlinker", libPath, + ] + + if let helpers { + let helpersPath = helpers.outputDir.path + arguments.append(contentsOf: [ + "-I", helpersPath, + "-L", helpersPath, + "-l\(helpersModuleName)", + "-Xlinker", "-rpath", "-Xlinker", helpersPath, + ]) + } + + arguments.append(wrappedPath) + let argumentsCopy = arguments + + let invocation: @Sendable () async throws -> SwiftRunOutcome = { + let record = try await run( + .name("swift"), + arguments: Arguments(argumentsCopy), + output: .string(limit: stdoutLimitBytes), + error: .string(limit: stderrLimitBytes) + ) + return .completed( + exitCode: exitCode(from: record.terminationStatus), + stdout: Data((record.standardOutput ?? "").utf8), + stderr: record.standardError ?? "" + ) + } + + let outcome: SwiftRunOutcome + if timeoutSeconds <= 0 { + outcome = try await invocation() + } else { + outcome = try await withThrowingTaskGroup(of: SwiftRunOutcome.self) { group in + group.addTask { try await invocation() } + group.addTask { + try await Task.sleep(for: .seconds(timeoutSeconds)) + return .timedOut + } + let first = try await group.next()! + group.cancelAll() + return first + } + } + + switch outcome { + case .completed(let exitCode, let stdout, let stderr): + return ProcessResult(exitCode: exitCode, stdout: stdout, stderr: stderr) + case .timedOut: + return ProcessResult( + exitCode: timeoutExitCode, + stdout: Data(), + stderr: "skit: timed out after \(timeoutSeconds)s\n" + ) + } + } + + private func exitCode(from status: TerminationStatus) -> Int32 { + switch status { + case .exited(let code): + return Int32(truncatingIfNeeded: code) + #if !os(Windows) + case .signaled(let signal): + // Match shell convention: 128 + signal number. + return 128 + Int32(truncatingIfNeeded: signal) + #endif + } + } -private final class PipeDataBox: @unchecked Sendable { - var value = Data() -} +#endif diff --git a/Sources/skit/Skit.swift b/Sources/skit/Skit.swift index 710a725..bd594e0 100644 --- a/Sources/skit/Skit.swift +++ b/Sources/skit/Skit.swift @@ -27,217 +27,222 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import ArgumentParser -import Foundation -import SyntaxParser - -@main -internal struct Skit: AsyncParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "skit", - abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", - subcommands: [Run.self, Parse.self], - defaultSubcommand: Run.self - ) -} - -// MARK: - skit run - -extension Skit { - internal struct Run: AsyncParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "run", - abstract: "Render SyntaxKit DSL input(s) into Swift source.", - discussion: """ - Wraps each input in a `Group { … }` closure and spawns `swift` to - evaluate it. The rendered output is written to stdout (single-file - mode) or mirrored into an output directory (folder mode). - - Forms: - skit run Input.swift — render to stdout - skit run Input.swift -o Out.swift — render to a file - skit run InputDir/ -o OutDir/ — walk **/*.swift (skipping - files prefixed with '_') - and mirror rendered output - into OutDir/ - """ - ) +#if canImport(Subprocess) - @Argument(help: "Path to a .swift input file or a directory of inputs.") - internal var input: String + import ArgumentParser + import Foundation + import SyntaxParser - @Option( - name: [.short, .customLong("output")], - help: "Output file (single-file mode) or directory (folder mode)." + @main + internal struct Skit: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "skit", + abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", + subcommands: [Run.self, Parse.self], + defaultSubcommand: Run.self ) - internal var output: String? + } - @Option( - name: .customLong("lib"), - help: ArgumentHelp( - "Directory containing libSyntaxKit.dylib + module files.", - discussion: """ - When omitted, skit searches: $SKIT_LIB_DIR, then /lib/, - then /../lib/skit/. Build a self-contained bundle with - Scripts/build-skit-release.sh. - """ - ) - ) - internal var libPath: String? + // MARK: - skit run - @Option( - name: .customLong("helpers"), - help: ArgumentHelp( - "Override Helpers/ directory location.", + extension Skit { + internal struct Run: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "run", + abstract: "Render SyntaxKit DSL input(s) into Swift source.", discussion: """ - By default skit walks up from the input looking for one. Helper - sources are pre-compiled into libSyntaxKitHelpers.dylib and made - importable via `import SyntaxKitHelpers`. + Wraps each input in a `Group { … }` closure and spawns `swift` to + evaluate it. The rendered output is written to stdout (single-file + mode) or mirrored into an output directory (folder mode). + + Forms: + skit run Input.swift — render to stdout + skit run Input.swift -o Out.swift — render to a file + skit run InputDir/ -o OutDir/ — walk **/*.swift (skipping + files prefixed with '_') + and mirror rendered output + into OutDir/ """ ) - ) - internal var helpersDir: String? - @Flag( - name: .customLong("no-helpers"), - help: "Skip helpers discovery entirely." - ) - internal var noHelpers: Bool = false + @Argument(help: "Path to a .swift input file or a directory of inputs.") + internal var input: String - @Flag( - name: .customLong("no-cache"), - help: ArgumentHelp( - "Skip the rendered-output cache (always run swift).", - discussion: """ - The cache lives at /outputs// and is keyed - on input bytes, helpers, swift version, libSyntaxKit stamp, and - SKIT_*/SYNTAXKIT_* env. - """ + @Option( + name: [.short, .customLong("output")], + help: "Output file (single-file mode) or directory (folder mode)." ) - ) - internal var noCache: Bool = false - - @Option( - name: .customLong("timeout"), - help: ArgumentHelp( - "Per-input timeout for the spawned `swift` (seconds).", - discussion: """ - Default 60. On expiry: SIGTERM, then SIGKILL after a 5s grace; the - file exits with code 124. Pass 0 to disable the watchdog. - """ + internal var output: String? + + @Option( + name: .customLong("lib"), + help: ArgumentHelp( + "Directory containing libSyntaxKit.dylib + module files.", + discussion: """ + When omitted, skit searches: $SKIT_LIB_DIR, then /lib/, + then /../lib/skit/. Build a self-contained bundle with + Scripts/build-skit-release.sh. + """ + ) ) - ) - internal var timeoutSeconds: Int = 60 - - @Flag( - name: .customLong("no-toolchain-check"), - help: ArgumentHelp( - "Skip the bundle/local Swift-toolchain comparison.", - discussion: """ - skit compares /swift-version.txt to `swift --version` at - startup and refuses to spawn `swift` on mismatch — swiftmodules - aren't reliably compatible across compiler versions. See issue - #157 for the auto-rebuild plan. - """ + internal var libPath: String? + + @Option( + name: .customLong("helpers"), + help: ArgumentHelp( + "Override Helpers/ directory location.", + discussion: """ + By default skit walks up from the input looking for one. Helper + sources are pre-compiled into libSyntaxKitHelpers.dylib and made + importable via `import SyntaxKitHelpers`. + """ + ) ) - ) - internal var noToolchainCheck: Bool = false + internal var helpersDir: String? - internal func validate() throws { - guard timeoutSeconds >= 0 else { - throw ValidationError( - "--timeout expects a non-negative integer (seconds), got: \(timeoutSeconds)" + @Flag( + name: .customLong("no-helpers"), + help: "Skip helpers discovery entirely." + ) + internal var noHelpers: Bool = false + + @Flag( + name: .customLong("no-cache"), + help: ArgumentHelp( + "Skip the rendered-output cache (always run swift).", + discussion: """ + The cache lives at /outputs// and is keyed + on input bytes, helpers, swift version, libSyntaxKit stamp, and + SKIT_*/SYNTAXKIT_* env. + """ ) - } - } + ) + internal var noCache: Bool = false + + @Option( + name: .customLong("timeout"), + help: ArgumentHelp( + "Per-input timeout for the spawned `swift` (seconds).", + discussion: """ + Default 60. On expiry, skit cancels the spawn and exits with code + 124. Pass 0 to disable the watchdog. + """ + ) + ) + internal var timeoutSeconds: Int = 60 + + @Flag( + name: .customLong("no-toolchain-check"), + help: ArgumentHelp( + "Skip the bundle/local Swift-toolchain comparison.", + discussion: """ + skit compares /swift-version.txt to `swift --version` at + startup and refuses to spawn `swift` on mismatch — swiftmodules + aren't reliably compatible across compiler versions. See issue + #157 for the auto-rebuild plan. + """ + ) + ) + internal var noToolchainCheck: Bool = false - internal func run() async throws { - let libPath: String - do { - libPath = try resolveLibPath(override: self.libPath) - } catch { - FileHandle.standardError.write(Data("\(error)\n".utf8)) - throw ExitCode(2) + internal func validate() throws { + guard timeoutSeconds >= 0 else { + throw ValidationError( + "--timeout expects a non-negative integer (seconds), got: \(timeoutSeconds)" + ) + } } - if !noToolchainCheck { - switch toolchainCheck(libPath: libPath) { - case .match, .stampMissing: - break - case .mismatch(let bundle, let local): - FileHandle.standardError.write( - Data(toolchainMismatchMessage(bundle: bundle, local: local).utf8)) + internal func run() async throws { + let libPath: String + do { + libPath = try resolveLibPath(override: self.libPath) + } catch { + FileHandle.standardError.write(Data("\(error)\n".utf8)) throw ExitCode(2) } - } - let helpersOptions: HelpersOptions - if noHelpers { - helpersOptions = .disabled - } else if let dir = helpersDir { - helpersOptions = .explicit(dir) - } else { - helpersOptions = .auto - } + if !noToolchainCheck { + switch await toolchainCheck(libPath: libPath) { + case .match, .stampMissing: + break + case .mismatch(let bundle, let local): + FileHandle.standardError.write( + Data(toolchainMismatchMessage(bundle: bundle, local: local).utf8)) + throw ExitCode(2) + } + } - var isDirectory: ObjCBool = false - guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { - throw ValidationError("input does not exist: \(input)") - } + let helpersOptions: HelpersOptions + if noHelpers { + helpersOptions = .disabled + } else if let dir = helpersDir { + helpersOptions = .explicit(dir) + } else { + helpersOptions = .auto + } - if isDirectory.boolValue { - guard let output else { - throw ValidationError("directory inputs require -o ") + var isDirectory: ObjCBool = false + guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { + throw ValidationError("input does not exist: \(input)") + } + + if isDirectory.boolValue { + guard let output else { + throw ValidationError("directory inputs require -o ") + } + let helpers = try await resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + let exitCode = await runDirectory( + inputDir: input, + outputDir: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) + throw ExitCode(exitCode) + } else { + let helpers = try await resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + try await runSingleFile( + inputPath: input, + outputPath: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) } - let helpers = try resolveHelpers( - nearInputPath: input, - libPath: libPath, - options: helpersOptions - ) - let exitCode = await runDirectory( - inputDir: input, - outputDir: output, - libPath: libPath, - helpers: helpers, - useCache: !noCache, - timeoutSeconds: timeoutSeconds - ) - throw ExitCode(exitCode) - } else { - let helpers = try resolveHelpers( - nearInputPath: input, - libPath: libPath, - options: helpersOptions - ) - try runSingleFile( - inputPath: input, - outputPath: output, - libPath: libPath, - helpers: helpers, - useCache: !noCache, - timeoutSeconds: timeoutSeconds - ) } } } -} -// MARK: - skit parse + // MARK: - skit parse -extension Skit { - internal struct Parse: ParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "parse", - abstract: "Parse Swift source on stdin into a JSON syntax tree on stdout." - ) + extension Skit { + internal struct Parse: ParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "parse", + abstract: "Parse Swift source on stdin into a JSON syntax tree on stdout." + ) - internal func run() throws { - let code = String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" - let treeNodes = SyntaxParser.parse(code: code) - let encoder = JSONEncoder() - let data = try encoder.encode(treeNodes) - let json = String(decoding: data, as: UTF8.self) - print(json) + internal func run() throws { + let code = + String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" + let treeNodes = SyntaxParser.parse(code: code) + let encoder = JSONEncoder() + let data = try encoder.encode(treeNodes) + let json = String(decoding: data, as: UTF8.self) + print(json) + } } } -} + +#endif diff --git a/Sources/skit/SkitStub.swift b/Sources/skit/SkitStub.swift new file mode 100644 index 0000000..5d3a7d1 --- /dev/null +++ b/Sources/skit/SkitStub.swift @@ -0,0 +1,44 @@ +// +// SkitStub.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if !canImport(Subprocess) + + import Foundation + + @main + internal enum SkitStub { + internal static func main() { + FileHandle.standardError.write( + Data("skit: this platform is not supported (no Subprocess backend).\n".utf8) + ) + exit(1) + } + } + +#endif diff --git a/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift b/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift new file mode 100644 index 0000000..ebd083e --- /dev/null +++ b/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift @@ -0,0 +1,101 @@ +// +// SkitSubprocessTimeoutTests.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Foundation + import Subprocess + import Testing + + // Regression test for swift-subprocess #256: + // + // + // The skit timeout watchdog (Sources/skit/Runner.swift) races a Subprocess + // run() call against `Task.sleep(timeout)`. On timeout it calls + // `group.cancelAll()`, which Subprocess turns into a teardown sequence on + // the spawned `swift`. The reported bug: when the spawned child has a + // grandchild that inherited the pipe FDs, the parent's stream read can hang + // waiting for an EOF that won't arrive until the grandchild exits. + // + // skit's real-world trigger is `swift Input.swift`, which fork-exec's the + // Swift frontend + linker as grandchildren. We approximate that here with a + // shell pipeline that forks a background `sleep` holding stderr open. + @Suite("Subprocess timeout-cancel") + internal struct SkitSubprocessTimeoutTests { + @Test( + "cancel-on-timeout completes within a bounded wall-time when grandchildren hold pipe fds" + ) + internal func cancelWithBackgroundedGrandchild() async throws { + let start = ContinuousClock.now + + let outcome: Outcome = try await withThrowingTaskGroup(of: Outcome.self) { group in + group.addTask { + // Outer shell spawns a backgrounded grandchild that holds stderr + // open for 30s, then itself sleeps 30s. Both must be forcibly + // killed by the teardown to free our stream reads. + let record = try await run( + .name("sh"), + arguments: ["-c", "(sleep 30 >/dev/null 2>&1 &) ; sleep 30"], + output: .discarded, + error: .discarded + ) + return .completed(record.terminationStatus) + } + group.addTask { + try await Task.sleep(for: .seconds(1)) + return .timedOut + } + let first = try await group.next()! + group.cancelAll() + return first + } + + let elapsed = ContinuousClock.now - start + + #expect( + outcome == .timedOut, + "timeout task should win the race against a 30s sleep" + ) + // Generous bound: cancellation + Subprocess teardown should complete + // well under 15s. If swift-subprocess #256 triggers, this test fails + // (the run task hangs ≥30s waiting for the grandchild to release the + // stderr pipe). + #expect( + elapsed < .seconds(15), + "timeout-cancel took \(elapsed); possible swift-subprocess #256 regression" + ) + } + + private enum Outcome: Equatable, Sendable { + case completed(TerminationStatus) + case timedOut + } + } + +#endif From 675882212d6b69452f49047e311766e1f52ef006 Mon Sep 17 00:00:00 2001 From: leogdion Date: Wed, 13 May 2026 14:42:04 +0000 Subject: [PATCH 24/28] Fixing CI --- .github/workflows/SyntaxKit.yml | 15 ++++++--------- .github/workflows/swift-source-compat.yml | 1 - 2 files changed, 6 insertions(+), 10 deletions(-) diff --git a/.github/workflows/SyntaxKit.yml b/.github/workflows/SyntaxKit.yml index 5fb86e0..451d097 100644 --- a/.github/workflows/SyntaxKit.yml +++ b/.github/workflows/SyntaxKit.yml @@ -65,7 +65,7 @@ jobs: run: | if [[ "${{ steps.check.outputs.full }}" == "true" ]]; then echo 'ubuntu-os=["noble","jammy"]' >> "$GITHUB_OUTPUT" - echo 'ubuntu-swift=[{"version":"6.0"},{"version":"6.1"},{"version":"6.2"},{"version":"6.3"}]' >> "$GITHUB_OUTPUT" + echo 'ubuntu-swift=[{"version":"6.1"},{"version":"6.2"},{"version":"6.3"}]' >> "$GITHUB_OUTPUT" echo 'ubuntu-type=["","wasm","wasm-embedded"]' >> "$GITHUB_OUTPUT" else echo 'ubuntu-os=["noble"]' >> "$GITHUB_OUTPUT" @@ -79,15 +79,12 @@ jobs: runs-on: ubuntu-latest container: ${{ matrix.swift.nightly && format('swiftlang/swift:nightly-{0}-{1}', matrix.swift.version, matrix.os) || format('swift:{0}-{1}', matrix.swift.version, matrix.os) }} strategy: + fail-fast: false matrix: os: ${{ fromJSON(needs.configure.outputs.ubuntu-os) }} swift: ${{ fromJSON(needs.configure.outputs.ubuntu-swift) }} type: ${{ fromJSON(needs.configure.outputs.ubuntu-type) }} exclude: - - swift: {version: "6.0"} - type: wasm - - swift: {version: "6.0"} - type: wasm-embedded - swift: {version: "6.1"} type: wasm - swift: {version: "6.1"} @@ -237,7 +234,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "iPhone 17 Pro" - osVersion: "26.4" + osVersion: "26.4.1" download-platform: true # watchOS Build Matrix @@ -245,7 +242,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple Watch Ultra 3 (49mm)" - osVersion: "26.4" + osVersion: "26.4.1" download-platform: true # tvOS Build Matrix @@ -253,7 +250,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple TV" - osVersion: "26.4" + osVersion: "26.4.1" download-platform: true # visionOS Build Matrix @@ -261,7 +258,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple Vision Pro" - osVersion: "26.4" + osVersion: "26.4.1" download-platform: true steps: diff --git a/.github/workflows/swift-source-compat.yml b/.github/workflows/swift-source-compat.yml index a8c05a0..4ab0b69 100644 --- a/.github/workflows/swift-source-compat.yml +++ b/.github/workflows/swift-source-compat.yml @@ -16,7 +16,6 @@ jobs: fail-fast: false matrix: container: - - swift:6.0 - swift:6.1 - swift:6.2 - swift:6.3 From fbcc8fa4566208945b7fdb51eb8499ecf7a05be3 Mon Sep 17 00:00:00 2001 From: leogdion Date: Wed, 13 May 2026 11:20:01 -0400 Subject: [PATCH 25/28] Update OS version to 26.5 in SyntaxKit.yml --- .github/workflows/SyntaxKit.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/SyntaxKit.yml b/.github/workflows/SyntaxKit.yml index 451d097..fb5cd8c 100644 --- a/.github/workflows/SyntaxKit.yml +++ b/.github/workflows/SyntaxKit.yml @@ -242,7 +242,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple Watch Ultra 3 (49mm)" - osVersion: "26.4.1" + osVersion: "26.5" download-platform: true # tvOS Build Matrix @@ -250,7 +250,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple TV" - osVersion: "26.4.1" + osVersion: "26.5" download-platform: true # visionOS Build Matrix From 89fad366e2a6edbb1bbe00ede9a4ae6d1cba2220 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 13 May 2026 14:58:01 -0400 Subject: [PATCH 26/28] Fix two PR review regressions in Examples/Completed dsl files [skip ci] - for_loops/dsl.swift: restore `For(_, in:, where:, then:)` with where clause filtering; the earlier rewrite to an inner-body `If` was a semantic change (iterate-all-then-branch vs filter-at-iteration). - errors_async/dsl.swift: restore the original `TupleAssignment(...).async().throwing()` line emitting `let (fetchedData, fetchedPosts) = try await (data, posts)`. - Promote `TupleAssignment` (struct, syntax property, init, `.async()`/`.throwing()`/`.asyncSet()`) to `public` so the example compiles via skit, and switch its `import SwiftSyntax` to `public import SwiftSyntax`. Co-Authored-By: Claude Opus 4.7 (1M context) --- Examples/Completed/errors_async/dsl.swift | 11 ++++------- Examples/Completed/for_loops/dsl.swift | 16 ++++++++++------ .../SyntaxKit/Collections/TupleAssignment.swift | 14 +++++++------- 3 files changed, 21 insertions(+), 20 deletions(-) diff --git a/Examples/Completed/errors_async/dsl.swift b/Examples/Completed/errors_async/dsl.swift index 6dc71f1..aad5087 100644 --- a/Examples/Completed/errors_async/dsl.swift +++ b/Examples/Completed/errors_async/dsl.swift @@ -57,13 +57,10 @@ Do { ParameterExp(name: "id", value: Literal.integer(1)) } }.async() - // The original example used `TupleAssignment([...], equals: Tuple {...})` - // to emit `let (fetchedData, fetchedPosts) = try await (data, posts)`, but - // `TupleAssignment` is internal in the current API. Emit two equivalent - // single-variable bindings instead — same observable behaviour for the - // catch block below. - Variable(.let, name: "fetchedData") { VariableExp("data") } - Variable(.let, name: "fetchedPosts") { VariableExp("posts") } + TupleAssignment(["fetchedData", "fetchedPosts"], equals: Tuple { + VariableExp("data") + VariableExp("posts") + }).async().throwing() } catch: { Catch(EnumCase("fetchError")) { // Example catch for async/await diff --git a/Examples/Completed/for_loops/dsl.swift b/Examples/Completed/for_loops/dsl.swift index 753e78a..74b1d1f 100644 --- a/Examples/Completed/for_loops/dsl.swift +++ b/Examples/Completed/for_loops/dsl.swift @@ -39,12 +39,16 @@ Group { } Variable(.let, name: "numbers", equals: Literal.array([Literal.integer(1), Literal.integer(2), Literal.integer(3), Literal.integer(4), Literal.integer(5), Literal.integer(6), Literal.integer(7), Literal.integer(8), Literal.integer(9), Literal.integer(10)])) - For(VariableExp("number"), in: VariableExp("numbers"), then: { - If(VariableExp("number % 2 == 0"), then: { - Call("print") { - ParameterExp(unlabeled: "\"Even number: \\(number)\"") - } - }) + For(VariableExp("number"), in: VariableExp("numbers"), where: { + Infix("==", + lhs: Infix("%", + lhs: VariableExp("number"), + rhs: Literal.integer(2)), + rhs: Literal.integer(0)) + }, then: { + Call("print") { + ParameterExp(unlabeled: "\"Even number: \\(number)\"") + } }) // MARK: - For-in with Dictionary diff --git a/Sources/SyntaxKit/Collections/TupleAssignment.swift b/Sources/SyntaxKit/Collections/TupleAssignment.swift index fbea0ba..1c30d1c 100644 --- a/Sources/SyntaxKit/Collections/TupleAssignment.swift +++ b/Sources/SyntaxKit/Collections/TupleAssignment.swift @@ -28,10 +28,10 @@ // import Foundation -import SwiftSyntax +public import SwiftSyntax /// A tuple assignment statement for destructuring multiple values. -internal struct TupleAssignment: CodeBlock { +public struct TupleAssignment: CodeBlock { private let elements: [String] private let value: any CodeBlock private var isAsync: Bool = false @@ -39,7 +39,7 @@ internal struct TupleAssignment: CodeBlock { private var isAsyncSet: Bool = false /// The syntax representation of this tuple assignment. - internal var syntax: any SyntaxProtocol { + public var syntax: any SyntaxProtocol { if isAsyncSet { return generateAsyncSetSyntax() } @@ -50,14 +50,14 @@ internal struct TupleAssignment: CodeBlock { /// - Parameters: /// - elements: The names of the variables to destructure into. /// - value: The expression to destructure. - internal init(_ elements: [String], equals value: any CodeBlock) { + public init(_ elements: [String], equals value: any CodeBlock) { self.elements = elements self.value = value } /// Marks this destructuring as async. /// - Returns: A copy of the destructuring marked as async. - internal func async() -> Self { + public func async() -> Self { var copy = self copy.isAsync = true return copy @@ -65,7 +65,7 @@ internal struct TupleAssignment: CodeBlock { /// Marks this destructuring as throwing. /// - Returns: A copy of the destructuring marked as throwing. - internal func throwing() -> Self { + public func throwing() -> Self { var copy = self copy.isThrowing = true return copy @@ -73,7 +73,7 @@ internal struct TupleAssignment: CodeBlock { /// Marks this destructuring as concurrent async (async let set). /// - Returns: A copy of the destructuring marked as async set. - internal func asyncSet() -> Self { + public func asyncSet() -> Self { var copy = self copy.isAsyncSet = true return copy From 824fb3ceb1d7e586b90300ba25002f5eb538e25c Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 13 May 2026 15:25:46 -0400 Subject: [PATCH 27/28] Narrow Subprocess guard in skit and split subcommands into their own files [skip ci] Drop the file-level `#if canImport(Subprocess)` around the entire Skit type and the separate SkitStub `@main`. The single `@main Skit` now covers all platforms; only the body of `Run.run()` is guarded, so `skit parse` works without a Subprocess backend. Each subcommand lives in its own extension file (Skit+Run.swift, Skit+Parse.swift), and Run's options shed their verbose `discussion:` text in favor of ArgumentParser's default `--help` rendering. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../skit/{SkitStub.swift => Skit+Parse.swift} | 31 ++- Sources/skit/Skit+Run.swift | 169 +++++++++++++ Sources/skit/Skit.swift | 230 +----------------- 3 files changed, 199 insertions(+), 231 deletions(-) rename Sources/skit/{SkitStub.swift => Skit+Parse.swift} (64%) create mode 100644 Sources/skit/Skit+Run.swift diff --git a/Sources/skit/SkitStub.swift b/Sources/skit/Skit+Parse.swift similarity index 64% rename from Sources/skit/SkitStub.swift rename to Sources/skit/Skit+Parse.swift index 5d3a7d1..47ad22d 100644 --- a/Sources/skit/SkitStub.swift +++ b/Sources/skit/Skit+Parse.swift @@ -1,5 +1,5 @@ // -// SkitStub.swift +// Skit+Parse.swift // SyntaxKit // // Created by Leo Dion. @@ -27,18 +27,25 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if !canImport(Subprocess) +import ArgumentParser +import Foundation +import SyntaxParser - import Foundation +extension Skit { + internal struct Parse: ParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "parse", + abstract: "Parse Swift source on stdin into a JSON syntax tree on stdout." + ) - @main - internal enum SkitStub { - internal static func main() { - FileHandle.standardError.write( - Data("skit: this platform is not supported (no Subprocess backend).\n".utf8) - ) - exit(1) + internal func run() throws { + let code = + String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" + let treeNodes = SyntaxParser.parse(code: code) + let encoder = JSONEncoder() + let data = try encoder.encode(treeNodes) + let json = String(decoding: data, as: UTF8.self) + print(json) } } - -#endif +} diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift new file mode 100644 index 0000000..dff3a80 --- /dev/null +++ b/Sources/skit/Skit+Run.swift @@ -0,0 +1,169 @@ +// +// Skit+Run.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import ArgumentParser +import Foundation + +extension Skit { + internal struct Run: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "run", + abstract: "Render SyntaxKit DSL input(s) into Swift source." + ) + + @Argument(help: "Path to a .swift input file or a directory of inputs.") + internal var input: String + + @Option( + name: [.short, .customLong("output")], + help: "Output file (single-file mode) or directory (folder mode)." + ) + internal var output: String? + + @Option( + name: .customLong("lib"), + help: "Directory containing libSyntaxKit.dylib + module files." + ) + internal var libPath: String? + + @Option( + name: .customLong("helpers"), + help: "Override Helpers/ directory location." + ) + internal var helpersDir: String? + + @Flag( + name: .customLong("no-helpers"), + help: "Skip helpers discovery entirely." + ) + internal var noHelpers: Bool = false + + @Flag( + name: .customLong("no-cache"), + help: "Skip the rendered-output cache (always run swift)." + ) + internal var noCache: Bool = false + + @Option( + name: .customLong("timeout"), + help: "Per-input timeout for the spawned `swift` in seconds (0 disables)." + ) + internal var timeoutSeconds: Int = 60 + + @Flag( + name: .customLong("no-toolchain-check"), + help: "Skip the bundle/local Swift-toolchain comparison." + ) + internal var noToolchainCheck: Bool = false + + internal func validate() throws { + guard timeoutSeconds >= 0 else { + throw ValidationError( + "--timeout expects a non-negative integer (seconds), got: \(timeoutSeconds)" + ) + } + } + + internal func run() async throws { + #if canImport(Subprocess) + let libPath: String + do { + libPath = try resolveLibPath(override: self.libPath) + } catch { + FileHandle.standardError.write(Data("\(error)\n".utf8)) + throw ExitCode(2) + } + + if !noToolchainCheck { + switch await toolchainCheck(libPath: libPath) { + case .match, .stampMissing: + break + case .mismatch(let bundle, let local): + FileHandle.standardError.write( + Data(toolchainMismatchMessage(bundle: bundle, local: local).utf8)) + throw ExitCode(2) + } + } + + let helpersOptions: HelpersOptions + if noHelpers { + helpersOptions = .disabled + } else if let dir = helpersDir { + helpersOptions = .explicit(dir) + } else { + helpersOptions = .auto + } + + var isDirectory: ObjCBool = false + guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { + throw ValidationError("input does not exist: \(input)") + } + + if isDirectory.boolValue { + guard let output else { + throw ValidationError("directory inputs require -o ") + } + let helpers = try await resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + let exitCode = await runDirectory( + inputDir: input, + outputDir: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) + throw ExitCode(exitCode) + } else { + let helpers = try await resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + try await runSingleFile( + inputPath: input, + outputPath: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) + } + #else + FileHandle.standardError.write( + Data("skit: run is not supported on this platform (no Subprocess backend).\n".utf8) + ) + throw ExitCode(1) + #endif + } + } +} diff --git a/Sources/skit/Skit.swift b/Sources/skit/Skit.swift index bd594e0..92d8c1a 100644 --- a/Sources/skit/Skit.swift +++ b/Sources/skit/Skit.swift @@ -27,222 +27,14 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) - - import ArgumentParser - import Foundation - import SyntaxParser - - @main - internal struct Skit: AsyncParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "skit", - abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", - subcommands: [Run.self, Parse.self], - defaultSubcommand: Run.self - ) - } - - // MARK: - skit run - - extension Skit { - internal struct Run: AsyncParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "run", - abstract: "Render SyntaxKit DSL input(s) into Swift source.", - discussion: """ - Wraps each input in a `Group { … }` closure and spawns `swift` to - evaluate it. The rendered output is written to stdout (single-file - mode) or mirrored into an output directory (folder mode). - - Forms: - skit run Input.swift — render to stdout - skit run Input.swift -o Out.swift — render to a file - skit run InputDir/ -o OutDir/ — walk **/*.swift (skipping - files prefixed with '_') - and mirror rendered output - into OutDir/ - """ - ) - - @Argument(help: "Path to a .swift input file or a directory of inputs.") - internal var input: String - - @Option( - name: [.short, .customLong("output")], - help: "Output file (single-file mode) or directory (folder mode)." - ) - internal var output: String? - - @Option( - name: .customLong("lib"), - help: ArgumentHelp( - "Directory containing libSyntaxKit.dylib + module files.", - discussion: """ - When omitted, skit searches: $SKIT_LIB_DIR, then /lib/, - then /../lib/skit/. Build a self-contained bundle with - Scripts/build-skit-release.sh. - """ - ) - ) - internal var libPath: String? - - @Option( - name: .customLong("helpers"), - help: ArgumentHelp( - "Override Helpers/ directory location.", - discussion: """ - By default skit walks up from the input looking for one. Helper - sources are pre-compiled into libSyntaxKitHelpers.dylib and made - importable via `import SyntaxKitHelpers`. - """ - ) - ) - internal var helpersDir: String? - - @Flag( - name: .customLong("no-helpers"), - help: "Skip helpers discovery entirely." - ) - internal var noHelpers: Bool = false - - @Flag( - name: .customLong("no-cache"), - help: ArgumentHelp( - "Skip the rendered-output cache (always run swift).", - discussion: """ - The cache lives at /outputs// and is keyed - on input bytes, helpers, swift version, libSyntaxKit stamp, and - SKIT_*/SYNTAXKIT_* env. - """ - ) - ) - internal var noCache: Bool = false - - @Option( - name: .customLong("timeout"), - help: ArgumentHelp( - "Per-input timeout for the spawned `swift` (seconds).", - discussion: """ - Default 60. On expiry, skit cancels the spawn and exits with code - 124. Pass 0 to disable the watchdog. - """ - ) - ) - internal var timeoutSeconds: Int = 60 - - @Flag( - name: .customLong("no-toolchain-check"), - help: ArgumentHelp( - "Skip the bundle/local Swift-toolchain comparison.", - discussion: """ - skit compares /swift-version.txt to `swift --version` at - startup and refuses to spawn `swift` on mismatch — swiftmodules - aren't reliably compatible across compiler versions. See issue - #157 for the auto-rebuild plan. - """ - ) - ) - internal var noToolchainCheck: Bool = false - - internal func validate() throws { - guard timeoutSeconds >= 0 else { - throw ValidationError( - "--timeout expects a non-negative integer (seconds), got: \(timeoutSeconds)" - ) - } - } - - internal func run() async throws { - let libPath: String - do { - libPath = try resolveLibPath(override: self.libPath) - } catch { - FileHandle.standardError.write(Data("\(error)\n".utf8)) - throw ExitCode(2) - } - - if !noToolchainCheck { - switch await toolchainCheck(libPath: libPath) { - case .match, .stampMissing: - break - case .mismatch(let bundle, let local): - FileHandle.standardError.write( - Data(toolchainMismatchMessage(bundle: bundle, local: local).utf8)) - throw ExitCode(2) - } - } - - let helpersOptions: HelpersOptions - if noHelpers { - helpersOptions = .disabled - } else if let dir = helpersDir { - helpersOptions = .explicit(dir) - } else { - helpersOptions = .auto - } - - var isDirectory: ObjCBool = false - guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { - throw ValidationError("input does not exist: \(input)") - } - - if isDirectory.boolValue { - guard let output else { - throw ValidationError("directory inputs require -o ") - } - let helpers = try await resolveHelpers( - nearInputPath: input, - libPath: libPath, - options: helpersOptions - ) - let exitCode = await runDirectory( - inputDir: input, - outputDir: output, - libPath: libPath, - helpers: helpers, - useCache: !noCache, - timeoutSeconds: timeoutSeconds - ) - throw ExitCode(exitCode) - } else { - let helpers = try await resolveHelpers( - nearInputPath: input, - libPath: libPath, - options: helpersOptions - ) - try await runSingleFile( - inputPath: input, - outputPath: output, - libPath: libPath, - helpers: helpers, - useCache: !noCache, - timeoutSeconds: timeoutSeconds - ) - } - } - } - } - - // MARK: - skit parse - - extension Skit { - internal struct Parse: ParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "parse", - abstract: "Parse Swift source on stdin into a JSON syntax tree on stdout." - ) - - internal func run() throws { - let code = - String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" - let treeNodes = SyntaxParser.parse(code: code) - let encoder = JSONEncoder() - let data = try encoder.encode(treeNodes) - let json = String(decoding: data, as: UTF8.self) - print(json) - } - } - } - -#endif +import ArgumentParser + +@main +internal struct Skit: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "skit", + abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", + subcommands: [Run.self, Parse.self], + defaultSubcommand: Run.self + ) +} From 74f6c6ac986c32d212a75dee3ca692c138a96f1c Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 13 May 2026 19:44:10 -0400 Subject: [PATCH 28/28] Add lifecycle map and inline phase comments across Sources/skit/ [skip ci] Documents the `skit run` pipeline so a reader can trace it end-to-end from the source files alone: top-of-Runner.swift lifecycle map, numbered phases in Skit.Run.run(), docstrings on every previously undocumented orchestrator, and inline step labels inside processFile, runDirectory, runSwift, wrap, buildHelpers, compileHelpers, outputCacheKey, and storeCachedOutput. No behaviour changes. Co-Authored-By: Claude Opus 4.7 (1M context) --- Sources/skit/Helpers.swift | 35 +++++++++ Sources/skit/OutputCache.swift | 19 ++++- Sources/skit/Runner.swift | 126 +++++++++++++++++++++++++++++++-- Sources/skit/Skit+Run.swift | 29 ++++++++ Sources/skit/Skit.swift | 6 ++ 5 files changed, 209 insertions(+), 6 deletions(-) diff --git a/Sources/skit/Helpers.swift b/Sources/skit/Helpers.swift index f9511ae..12430a4 100644 --- a/Sources/skit/Helpers.swift +++ b/Sources/skit/Helpers.swift @@ -109,13 +109,21 @@ /// Compiles helper sources into a per-key cache directory and returns the /// directory plus a cache-hit flag. Returns nil when the helpers dir is empty. + /// + /// Concurrent invocations are tolerated via the staging-dir + atomic-rename + /// pattern: if two processes race to compile the same key, the loser's + /// rename fails and we keep the winner's artefact. internal func buildHelpers( helpersDir: URL, libPath: String ) async throws -> CompiledHelpers? { + // Collect helper sources. An empty Helpers/ dir is "no helpers" rather + // than an error — the caller will fall back to no-helpers mode. let sources = try collectHelperSources(in: helpersDir) if sources.isEmpty { return nil } + // Compute the content-keyed cache path. The dylib's presence under that + // path is what makes a build "cached". let key = try await helpersCacheKey(sources: sources, libPath: libPath) let cacheRoot = try syntaxKitCacheRoot() .appendingPathComponent("helpers") @@ -124,21 +132,28 @@ cacheRoot .appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path + // Cache hit: artefact already present, skip the whole compile. let fm = FileManager.default if fm.fileExists(atPath: dylibPath) { return CompiledHelpers(outputDir: cacheRoot, cacheHit: true) } + // Ensure the parent of the cache key dir exists. We don't create the + // key dir itself — the atomic move below installs it. try fm.createDirectory( at: cacheRoot.deletingLastPathComponent(), withIntermediateDirectories: true ) + // Compile into a per-pid + uuid staging dir, then atomically rename into + // place. This is what lets concurrent skit invocations co-exist safely. let staging = cacheRoot.deletingLastPathComponent() .appendingPathComponent( "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)") try fm.createDirectory(at: staging, withIntermediateDirectories: true) + // Run swiftc into the staging dir. Clean up on failure so we don't leak + // half-baked artefacts in the cache root. do { try await compileHelpers(sources: sources, into: staging, libPath: libPath) } catch { @@ -160,11 +175,18 @@ return CompiledHelpers(outputDir: cacheRoot, cacheHit: false) } + /// Invokes `swiftc` to build `sources` into a Swift module + dylib under + /// `outDir`. The dylib is named `lib.{dylib,so}` and the + /// module file is `.swiftmodule`. Output (stdout) + /// is discarded; stderr is captured for the error path. private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) async throws { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" let dylib = outDir.appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path let modulePath = outDir.appendingPathComponent("\(helpersModuleName).swiftmodule").path + // Base swiftc arguments: emit a library + module file linking against + // libSyntaxKit, with rpath set so the dylib can find libSyntaxKit at + // load time. var args: [String] = [ "-module-name", helpersModuleName, "-emit-module", @@ -188,8 +210,11 @@ "-Xlinker", "@rpath/\(dylibFilename(forLibrary: helpersModuleName))", ]) #endif + // Append source files last so the leading flags apply to all of them. args.append(contentsOf: sources.map(\.path)) + // Spawn swiftc. Stderr is captured (1 MiB cap) so a compile failure can + // surface the diagnostic verbatim in a CLIError. let result = try await run( .name("swiftc"), arguments: Arguments(args), @@ -209,16 +234,21 @@ // MARK: - Cache key + /// Content-addressed cache key mixing schema version, each helper source's + /// filename + bytes, `swift --version`, and the libSyntaxKit stamp. private func helpersCacheKey(sources: [URL], libPath: String) async throws -> String { var hasher = ContentHasher() hasher.update(data: Data(helpersCacheSchemaVersion.utf8)) + // Filename matters as well as bytes — two same-content files with + // different names produce different symbols. for source in sources { let data = try Data(contentsOf: source) hasher.update(data: Data(source.lastPathComponent.utf8)) hasher.update(data: data) } + // Toolchain + dylib stamp invalidate on cross-version or in-place rebuild. if let swiftVersion = await captureSwiftVersion() { hasher.update(data: Data(swiftVersion.utf8)) } @@ -229,6 +259,7 @@ return hasher.finalize() } + /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. internal func captureSwiftVersion() async -> String? { let result = try? await run( .name("swift"), @@ -239,6 +270,8 @@ return result?.standardOutput } + /// `/` fingerprint of the bundled libSyntaxKit dylib, or nil + /// if unreadable. Catches in-place rebuilds without a version bump. internal func libStamp(libPath: String) -> String? { let dylib = "\(libPath)/\(dylibFilename(forLibrary: "SyntaxKit"))" guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } @@ -247,6 +280,8 @@ return "\(size)/\(Int(mtime))" } + /// Root for all skit caches. Honours `XDG_CACHE_HOME`, else macOS + /// `~/Library/Caches/...` or Linux `~/.cache/syntaxkit`. internal func syntaxKitCacheRoot() throws -> URL { if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") diff --git a/Sources/skit/OutputCache.swift b/Sources/skit/OutputCache.swift index 47f7972..e68dd84 100644 --- a/Sources/skit/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -44,23 +44,33 @@ libPath: String ) async -> String { var hasher = ContentHasher() + // Schema version: bump to invalidate every existing cache entry at once. hasher.update(data: Data(outputCacheSchemaVersion.utf8)) + // Input source bytes: the primary driver of the key. hasher.update(data: Data(inputSource.utf8)) + // Helpers fingerprint. The helpers cache dir name *is* the helpers cache + // key (per Helpers.swift), so re-mixing it here cheaply propagates any + // helpers change into this key. if let helpers { - // Helpers cache dir name *is* the helpers cache key (per Helpers.swift). hasher.update(data: Data(helpers.outputDir.lastPathComponent.utf8)) } else { hasher.update(data: Data("no-helpers".utf8)) } + // Toolchain version. Different `swift` builds emit different bytes for + // the same DSL input. if let version = await captureSwiftVersion() { hasher.update(data: Data(version.utf8)) } + // libSyntaxKit stamp. A rebuilt dylib can change the rendered output + // even without a Swift-version bump. if let stamp = libStamp(libPath: libPath) { hasher.update(data: Data(stamp.utf8)) } + // SKIT_*/SYNTAXKIT_* env vars. Sorted so the cache key is stable, and + // NUL-terminated so `"AB=" + "C"` doesn't collide with `"A=" + "BC"`. let env = ProcessInfo.processInfo.environment .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } .sorted { $0.key < $1.key } @@ -84,11 +94,15 @@ let final = cacheRoot.appendingPathComponent("output.swift") let fm = FileManager.default + // Ensure the parent of the cache key dir exists. The key dir itself is + // installed by the atomic rename below. try fm.createDirectory( at: cacheRoot.deletingLastPathComponent(), withIntermediateDirectories: true ) + // Stage the payload in a per-pid + uuid sibling dir so it can be renamed + // into place as a single atomic step. let staging = cacheRoot.deletingLastPathComponent() .appendingPathComponent( "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)" @@ -96,6 +110,9 @@ try fm.createDirectory(at: staging, withIntermediateDirectories: true) try data.write(to: staging.appendingPathComponent("output.swift")) + // Atomic rename into the cache path. If a peer already populated this + // key, swallow the rename error and drop our staging copy. Re-throw only + // if the destination is still missing afterwards. do { try fm.moveItem(at: staging, to: cacheRoot) } catch { diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index d9407f4..4a8513d 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -34,19 +34,42 @@ import SwiftParser import SwiftSyntax + // Run lifecycle (per `skit run` invocation): + // 1. resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) + // 2. toolchainCheck — compare bundle stamp to `swift --version` + // 3. resolveHelpers — discover + compile Helpers/ (memoised on disk) + // 4. runSingleFile / runDirectory — dispatch to single- or batch-input mode + // 5. processFile (per input) — load → cache lookup → wrap → spawn → cache store + // 6. wrap — hoist imports, wrap body in Group { … }, #sourceLocation + // 7. runSwift — spawn `swift` with timeout watchdog + // See Docs/skit.md for design rationale and trade-offs. + // MARK: - Helpers resolution + /// How `skit run` should treat a `Helpers/` directory for this invocation. internal enum HelpersOptions { + /// Walk up from the input looking for `Helpers/`. The default. case auto + /// Skip helpers discovery entirely (`--no-helpers`). The wrapped input + /// won't be able to `import SyntaxKitHelpers`. case disabled + /// Use the directory at the given path (`--helpers `). Validated as + /// an existing directory in `resolveHelpers`. case explicit(String) } + /// Resolves a `Helpers/` directory and compiles it (or returns the cached + /// build). Returns nil when helpers are disabled, when no `Helpers/` was + /// found in auto mode, or when the directory exists but contains no `.swift` + /// sources. On success, writes a one-line "skit: helpers cached/compiled at + /// " note to stderr so users can see whether the cache hit. internal func resolveHelpers( nearInputPath path: String, libPath: String, options: HelpersOptions ) async throws -> CompiledHelpers? { + // Pick the helpers dir according to the mode: walk up the tree, accept an + // explicit override (after validating it's a directory), or bail out. let helpersDir: URL? switch options { case .disabled: @@ -65,6 +88,8 @@ } guard let helpersDir else { return nil } + // Compile (or reuse the cached build). An empty Helpers/ dir is treated + // as "no helpers" rather than an error. guard let compiled = try await buildHelpers(helpersDir: helpersDir, libPath: libPath) else { return nil } @@ -185,6 +210,11 @@ // MARK: - Single-file mode + /// Runs `processFile` on a single input and writes its rendered Swift to + /// `outputPath` (or stdout when nil). Any stderr from the spawned `swift` + /// is surfaced verbatim. On a non-zero subprocess exit, calls `exit()` + /// directly — the caller in `Skit.Run.run()` won't see a thrown error in + /// that path. internal func runSingleFile( inputPath: String, outputPath: String?, @@ -193,6 +223,8 @@ useCache: Bool, timeoutSeconds: Int ) async throws { + // Render the input. `processFile` may hit the output cache and skip the + // spawn entirely; either way the result has the same shape. let result = try await processFile( inputPath: inputPath, libPath: libPath, @@ -200,12 +232,16 @@ useCache: useCache, timeoutSeconds: timeoutSeconds ) + // Surface diagnostics from the spawned `swift` before deciding success. if !result.stderr.isEmpty { FileHandle.standardError.write(Data(result.stderr.utf8)) } + // Non-zero subprocess exit propagates as a process exit. We don't write + // partial output in that case. guard result.exitCode == 0 else { exit(result.exitCode) } + // Deliver the rendered output to file or stdout. if let outputPath { try result.stdout.write(to: URL(fileURLWithPath: outputPath)) } else { @@ -215,6 +251,10 @@ // MARK: - Folder mode + /// Walks `inputDir` for `.swift` inputs, processes them concurrently (up to + /// the active core count), and mirrors the rendered output into `outputDir`. + /// A failure on one input does not abort the batch — successful peers are + /// still written. Returns 0 if every input succeeded, 1 otherwise. internal func runDirectory( inputDir: String, outputDir: String, @@ -226,6 +266,8 @@ let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL + // Phase 1: enumerate inputs. Top-level `Helpers/` is excluded so its + // sources aren't processed as DSL inputs. let inputs: [URL] do { inputs = try collectInputs(at: inputURL, excluding: helpersExcludePath(inputDir: inputURL)) @@ -239,12 +281,15 @@ return 0 } + // Phase 2: bounded-concurrency processing. Cap is the active core count + // so a 200-file batch doesn't fork 200 simultaneous `swift` processes. let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) var outcomes: [FileOutcome] = [] var iterator = inputs.makeIterator() await withTaskGroup(of: FileOutcome.self) { group in + // Seed the group up to the concurrency cap… for _ in 0.. } + /// `processFile` adapter that catches errors into the `FileOutcome` result + /// so a single failure doesn't tear down the surrounding `TaskGroup`. private func runOne( _ input: URL, libPath: String, @@ -348,6 +402,9 @@ return candidate.path } + /// Returns every `.swift` file under `inputDir` (recursive), sorted, with + /// hidden files, files prefixed by `_`, and the `excludedDir` subtree + /// removed. Sorted output keeps batch behaviour deterministic across runs. private func collectInputs(at inputDir: URL, excluding excludedDir: String?) throws -> [URL] { guard let enumerator = FileManager.default.enumerator( @@ -362,12 +419,16 @@ var result: [URL] = [] for case let url as URL in enumerator { let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + // Directories aren't outputs; if this is the excluded `Helpers/` dir, + // prune the whole subtree. if values.isDirectory == true { if let excludedDir, url.standardizedFileURL.path == excludedDir { enumerator.skipDescendants() } continue } + // Filter for `.swift` regular files, skipping the `_`-prefixed + // convention for "not an input" sources. guard values.isRegularFile == true else { continue } guard url.pathExtension == "swift" else { continue } guard !url.lastPathComponent.hasPrefix("_") else { continue } @@ -378,12 +439,19 @@ // MARK: - Per-file work + /// Raw outcome of rendering one input — what `processFile` returns to its + /// caller. `exitCode == 0` indicates the spawned `swift` succeeded (or that + /// the output cache hit, which is treated identically). private struct ProcessResult: Sendable { let exitCode: Int32 let stdout: Data let stderr: String } + /// The per-input render pipeline: load source → consult the output cache → + /// (on miss) wrap → spawn `swift` → rewrite diagnostics → store the result + /// in the cache. The temp wrapper file is created in a per-run tmp dir and + /// torn down by `defer` whether the spawn succeeded or not. private func processFile( inputPath: String, libPath: String, @@ -391,20 +459,30 @@ useCache: Bool, timeoutSeconds: Int ) async throws -> ProcessResult { + // Load the input source. Anything past this point keys off these bytes. let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL let absoluteInputPath = inputURL.path let source = try String(contentsOf: inputURL, encoding: .utf8) + // Compute the output cache key (skipped under `--no-cache`). Mixes input + // bytes, toolchain version, helpers fingerprint, libSyntaxKit stamp, and + // sorted SKIT_*/SYNTAXKIT_* env vars — see `outputCacheKey`. let cacheKey: String? = useCache ? await outputCacheKey(inputSource: source, helpers: helpers, libPath: libPath) : nil + // Cache hit: skip the wrap+spawn entirely and return the stored output. if let cacheKey, let cached = lookupCachedOutput(key: cacheKey) { return ProcessResult(exitCode: 0, stdout: cached, stderr: "") } + // Wrap the user's input into a complete Swift program that imports + // SyntaxKit, runs the body inside a Group { … } builder, and prints the + // result. See `wrap` for the exact template. let wrapped = wrap(source: source, originalPath: absoluteInputPath) + // Spill the wrapped program to a per-invocation temp dir. The dir is + // cleaned up unconditionally so a failed spawn doesn't leak files. let tmpDir = FileManager.default.temporaryDirectory .appendingPathComponent("skit-\(UUID().uuidString)") try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) @@ -413,6 +491,8 @@ let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) + // Spawn `swift` on the wrapped file (with timeout watchdog). stdout is + // the rendered Swift source; stderr is compiler diagnostics, if any. let raw = try await runSwift( wrappedPath: wrappedURL.path, libPath: libPath, @@ -427,6 +507,8 @@ with: absoluteInputPath ) + // Store on the way out. `try?` is deliberate: a cache write failure is + // not a render failure. The next run will simply miss and re-spawn. if let cacheKey, raw.exitCode == 0 { try? storeCachedOutput(key: cacheKey, data: raw.stdout) } @@ -434,6 +516,8 @@ return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } + /// Throwable error wrapper for skit's user-facing diagnostics. The message + /// is printed verbatim — keep it actionable (path, hint, next step). internal struct CLIError: Error, CustomStringConvertible { let message: String var description: String { message } @@ -447,12 +531,16 @@ /// The body is fenced in `#sourceLocation` directives so compiler diagnostics /// in the body reference the original input file and line numbers. internal func wrap(source: String, originalPath: String) -> String { + // Parse the input with SwiftSyntax. The location converter is needed to + // map the body's starting byte offset back to a 1-based line number for + // the `#sourceLocation` directive. let tree = Parser.parse(source: source) let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) - // Find the first non-import top-level statement; everything before it that - // is an import gets hoisted, anything before that which is *not* an import - // stays in the body (e.g. a top-level `// comment` is left alone). + // Scan top-level statements for hoistable imports. Everything before the + // first non-import statement that *is* an import gets hoisted; anything + // before that which is *not* an import stays in the body (e.g. a top-level + // `// comment` is left alone). var hoisted: [String] = [] var firstBodyByte: AbsolutePosition? @@ -467,6 +555,8 @@ break } + // Compute the body slice (source from the first non-import byte onward) + // and the 1-based line number it lives on in the original file. let body: String let firstBodyLine: Int if let firstBodyByte { @@ -478,6 +568,9 @@ firstBodyLine = 1 } + // Render the hoisted-imports block. Trailing newline only if non-empty so + // the wrapper doesn't grow an extra blank line in the common no-imports + // case. let hoistedBlock = hoisted.isEmpty ? "" : hoisted.joined(separator: "\n") + "\n" // #sourceLocation must use a forward-slash path; escape backslashes/quotes @@ -487,6 +580,8 @@ .replacingOccurrences(of: "\\", with: "\\\\") .replacingOccurrences(of: "\"", with: "\\\"") + // Build the final wrapper. Layout: SyntaxKit import → hoisted imports → + // Group { #sourceLocation(...) #sourceLocation() } → print. return """ import SyntaxKit \(hoistedBlock) @@ -512,11 +607,19 @@ private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024 private let stderrLimitBytes: Int = 1 * 1_024 * 1_024 + /// Either the spawned `swift` ran to completion (success or failure) or + /// the watchdog elapsed first. The completed payload is normalized to the + /// shape callers want regardless of platform. private enum SwiftRunOutcome: Sendable { case completed(exitCode: Int32, stdout: Data, stderr: String) case timedOut } + /// Spawns `swift` (script-mode interpreter) on the wrapped input file, + /// optionally splicing in flags to import a precompiled helpers module. + /// When `timeoutSeconds > 0` the spawn races a sleep task in a throwing + /// task group; the loser is cancelled. On timeout, returns exit 124 with + /// a one-line stderr message — matching POSIX `timeout(1)`'s convention. private func runSwift( wrappedPath: String, libPath: String, @@ -525,6 +628,8 @@ ) async throws -> ProcessResult { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + // Build the base argument list: link against libSyntaxKit, include the + // CShims headers, set rpath so the dylib loads at runtime. var arguments: [String] = [ "-suppress-warnings", "-I", libPath, @@ -534,6 +639,9 @@ "-Xlinker", "-rpath", "-Xlinker", libPath, ] + // Splice in helpers-module flags only when a compiled helpers dylib is + // available. Skipping these makes `import SyntaxKitHelpers` fail in the + // wrapped input, which is fine when no Helpers/ dir was discovered. if let helpers { let helpersPath = helpers.outputDir.path arguments.append(contentsOf: [ @@ -547,6 +655,8 @@ arguments.append(wrappedPath) let argumentsCopy = arguments + // The actual subprocess call, wrapped in a closure so the task-group race + // below can hold a single Sendable reference to it. let invocation: @Sendable () async throws -> SwiftRunOutcome = { let record = try await run( .name("swift"), @@ -561,6 +671,9 @@ ) } + // Race the invocation against a sleep watchdog; whichever finishes first + // wins, the other is cancelled. `timeoutSeconds <= 0` opts out of the + // race entirely (useful for debugging genuinely long codegen). let outcome: SwiftRunOutcome if timeoutSeconds <= 0 { outcome = try await invocation() @@ -577,6 +690,7 @@ } } + // Normalize both outcomes into a single ProcessResult shape. switch outcome { case .completed(let exitCode, let stdout, let stderr): return ProcessResult(exitCode: exitCode, stdout: stdout, stderr: stderr) @@ -589,6 +703,8 @@ } } + /// Collapses Subprocess's `TerminationStatus` into a single Int32 exit code, + /// using the shell convention (128 + signal number) for signalled deaths. private func exitCode(from status: TerminationStatus) -> Int32 { switch status { case .exited(let code): diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index dff3a80..baf8d7b 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -31,6 +31,12 @@ import ArgumentParser import Foundation extension Skit { + /// Render one or more SyntaxKit DSL files into Swift source. + /// + /// `Run` is the default subcommand. It accepts either a single `.swift` + /// file or a directory of `.swift` files; in directory mode the rendered + /// output is written into a mirrored tree under `-o`. The actual work is + /// delegated to free functions in `Runner.swift`. internal struct Run: AsyncParsableCommand { internal static let configuration = CommandConfiguration( commandName: "run", @@ -92,6 +98,9 @@ extension Skit { internal func run() async throws { #if canImport(Subprocess) + // 1. Resolve the libSyntaxKit bundle dir. Failure here is fatal — we + // can't spawn `swift` without knowing where the dylib + swiftmodules + // live. The error message lists the four lookup paths in priority order. let libPath: String do { libPath = try resolveLibPath(override: self.libPath) @@ -100,6 +109,10 @@ extension Skit { throw ExitCode(2) } + // 2. Compare the bundle's recorded `swift --version` against the local + // one. swiftmodules aren't reliably forward-compatible across compiler + // versions, so a mismatch produces a clear error rather than letting + // the spawned `swift` emit a cryptic module-version diagnostic. if !noToolchainCheck { switch await toolchainCheck(libPath: libPath) { case .match, .stampMissing: @@ -111,6 +124,8 @@ extension Skit { } } + // 3. Decide which helpers-resolution mode this invocation is in. + // The actual discovery / compilation happens later in `resolveHelpers`. let helpersOptions: HelpersOptions if noHelpers { helpersOptions = .disabled @@ -120,6 +135,9 @@ extension Skit { helpersOptions = .auto } + // 4. Stat the input to pick single-file vs. directory mode. Directory + // mode requires an explicit `-o` output dir; single-file mode falls + // back to stdout. var isDirectory: ObjCBool = false guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { throw ValidationError("input does not exist: \(input)") @@ -129,11 +147,16 @@ extension Skit { guard let output else { throw ValidationError("directory inputs require -o ") } + // 5a. Resolve helpers relative to the input root. This is the only + // place we compile `Helpers/`; the result is reused across every + // input file in the directory. let helpers = try await resolveHelpers( nearInputPath: input, libPath: libPath, options: helpersOptions ) + // 6a. Hand off to the directory orchestrator and surface its exit + // code via ExitCode (so a partial-failure batch returns 1). let exitCode = await runDirectory( inputDir: input, outputDir: output, @@ -144,11 +167,15 @@ extension Skit { ) throw ExitCode(exitCode) } else { + // 5b. Resolve helpers relative to this single file's parent. let helpers = try await resolveHelpers( nearInputPath: input, libPath: libPath, options: helpersOptions ) + // 6b. Hand off to the single-file orchestrator. It calls `exit()` + // directly on non-zero subprocess exit, so a thrown ExitCode here + // would be unreachable in that path. try await runSingleFile( inputPath: input, outputPath: output, @@ -159,6 +186,8 @@ extension Skit { ) } #else + // Subprocess is the only backend skit knows how to use to spawn + // `swift`/`swiftc`. Without it (Windows, embedded), `run` cannot work. FileHandle.standardError.write( Data("skit: run is not supported on this platform (no Subprocess backend).\n".utf8) ) diff --git a/Sources/skit/Skit.swift b/Sources/skit/Skit.swift index 92d8c1a..d5cb936 100644 --- a/Sources/skit/Skit.swift +++ b/Sources/skit/Skit.swift @@ -29,6 +29,12 @@ import ArgumentParser +/// The `skit` CLI entry point. +/// +/// `Skit` itself is just an ArgumentParser shell that wires up two subcommands: +/// `Run` (the default, rendering SyntaxKit DSL into Swift source) and `Parse` +/// (the inverse, reading Swift source on stdin and emitting JSON). Their bodies +/// live in `Skit+Run.swift` and `Skit+Parse.swift` respectively. @main internal struct Skit: AsyncParsableCommand { internal static let configuration = CommandConfiguration(