Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
67107ea
Add research docs for SyntaxKit-driven codegen CLI (#154)
leogdion May 11, 2026
e492503
POC step 1: wrap+spawn flow works at ~720ms cold (#154)
leogdion May 11, 2026
cf9f02a
Add POC step 1 reproducer script (#154)
leogdion May 11, 2026
840cb0c
Release-config dylib size: 9.3 MB stripped (#154)
leogdion May 11, 2026
9e5affa
POC step 2: skitrun CLI wraps + spawns SyntaxKit DSL inputs (#154)
leogdion May 11, 2026
dd209c9
POC step 3: skitrun folder mode (#154)
leogdion May 12, 2026
c6daced
POC step 4: self-contained skitrun release bundle (#154)
leogdion May 12, 2026
3d3b631
POC step 5: Helpers/ discovery + per-toolchain cache (#154)
leogdion May 12, 2026
2adc3cd
POC step 6: rendered-output cache + --no-cache (#154)
leogdion May 12, 2026
68e49f0
Note web-server form as a post-CLI follow-up (#154)
leogdion May 12, 2026
65f2103
POC step 7: Linux smoke test + Foundation.Process workarounds (#154)
leogdion May 12, 2026
2836cf7
Add skitrun README scoped to the target dir (#154)
leogdion May 12, 2026
c6d9e7d
Fix skitrun release bundle and update blackjack example
leogdion May 12, 2026
84b497b
Update Examples/Completed/*/dsl.swift to current API + skitrun shape
leogdion May 12, 2026
207cb7e
Add --timeout watchdog to skitrun
leogdion May 12, 2026
a7d8876
Stamp + detect toolchain mismatch at skitrun startup
leogdion May 12, 2026
47e5be8
Update 3 more Examples/Completed/*/dsl.swift to current API
leogdion May 12, 2026
b0008fc
Move enum_generator out of Examples/Completed/
leogdion May 12, 2026
18f7097
Unify skit + skitrun into one binary with ArgumentParser subcommands
leogdion May 12, 2026
f3a1912
Productize Sources/skit/README.md for v0.0.5
leogdion May 12, 2026
05e8502
Replace POC research log with Docs/skit.md
leogdion May 12, 2026
e3e243f
Replace swift-crypto/SHA-256 with pure-Swift FNV-1a content hash
leogdion May 12, 2026
61349da
Move skit to swift-subprocess; revert simulator OS pin to 26.4
leogdion May 13, 2026
6758822
Fixing CI
leogdion May 13, 2026
fbcc8fa
Update OS version to 26.5 in SyntaxKit.yml
leogdion May 13, 2026
89fad36
Fix two PR review regressions in Examples/Completed dsl files [skip ci]
leogdion May 13, 2026
824fb3c
Narrow Subprocess guard in skit and split subcommands into their own …
leogdion May 13, 2026
74f6c6a
Add lifecycle map and inline phase comments across Sources/skit/ [ski…
leogdion May 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions .github/workflows/SyntaxKit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ jobs:
run: |
if [[ "${{ steps.check.outputs.full }}" == "true" ]]; then
echo 'ubuntu-os=["noble","jammy"]' >> "$GITHUB_OUTPUT"
echo 'ubuntu-swift=[{"version":"6.0"},{"version":"6.1"},{"version":"6.2"},{"version":"6.3"}]' >> "$GITHUB_OUTPUT"
echo 'ubuntu-swift=[{"version":"6.1"},{"version":"6.2"},{"version":"6.3"}]' >> "$GITHUB_OUTPUT"
echo 'ubuntu-type=["","wasm","wasm-embedded"]' >> "$GITHUB_OUTPUT"
else
echo 'ubuntu-os=["noble"]' >> "$GITHUB_OUTPUT"
Expand All @@ -79,15 +79,12 @@ jobs:
runs-on: ubuntu-latest
container: ${{ matrix.swift.nightly && format('swiftlang/swift:nightly-{0}-{1}', matrix.swift.version, matrix.os) || format('swift:{0}-{1}', matrix.swift.version, matrix.os) }}
strategy:
fail-fast: false
matrix:
os: ${{ fromJSON(needs.configure.outputs.ubuntu-os) }}
swift: ${{ fromJSON(needs.configure.outputs.ubuntu-swift) }}
type: ${{ fromJSON(needs.configure.outputs.ubuntu-type) }}
exclude:
- swift: {version: "6.0"}
type: wasm
- swift: {version: "6.0"}
type: wasm-embedded
- swift: {version: "6.1"}
type: wasm
- swift: {version: "6.1"}
Expand Down Expand Up @@ -229,12 +226,15 @@ jobs:
runs-on: macos-26
xcode: "/Applications/Xcode_26.5.app"

# iOS / watchOS / tvOS / visionOS osVersion pinned to 26.4 until the
# macos-26 runner image ships Xcode 26.5 simulators — see #160.

# iOS Build Matrix
- type: ios
runs-on: macos-26
xcode: "/Applications/Xcode_26.5.app"
deviceName: "iPhone 17 Pro"
osVersion: "26.5"
osVersion: "26.4.1"
download-platform: true

# watchOS Build Matrix
Expand All @@ -258,7 +258,7 @@ jobs:
runs-on: macos-26
xcode: "/Applications/Xcode_26.5.app"
deviceName: "Apple Vision Pro"
osVersion: "26.5"
osVersion: "26.4.1"
download-platform: true

steps:
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/swift-source-compat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ jobs:
fail-fast: false
matrix:
container:
- swift:6.0
- swift:6.1
- swift:6.2
- swift:6.3
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ playground.xcworkspace
.swiftpm

.build/
.build-linux/

# CocoaPods
# We recommend against adding the Pods directory to your .gitignore. However
Expand Down
20 changes: 20 additions & 0 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,26 @@
"preLaunchTask": "swift: Build Release skit",
"target": "skit",
"configuration": "release"
},
{
"type": "swift",
"request": "launch",
"args": [],
"cwd": "${workspaceFolder:SyntaxKit}",
"name": "Debug skitrun",
"target": "skitrun",
"configuration": "debug",
"preLaunchTask": "swift: Build Debug skitrun"
},
{
"type": "swift",
"request": "launch",
"args": [],
"cwd": "${workspaceFolder:SyntaxKit}",
"name": "Release skitrun",
"target": "skitrun",
"configuration": "release",
"preLaunchTask": "swift: Build Release skitrun"
}
]
}
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,8 @@ Sources/SyntaxKit/
2. **skit Executable** - Command-line tool for parsing Swift code to JSON

### Platform Support
- Swift 6.0+ required
- Xcode 16.0+ for development
- Swift 6.1+ required
- Xcode 16.3+ for development

### Testing
- Uses modern Swift Testing framework (`@Test` syntax)
Expand Down
265 changes: 265 additions & 0 deletions Docs/research/tuist-manifest-pipeline.md

Large diffs are not rendered by default.

206 changes: 206 additions & 0 deletions Docs/skit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
# `skit` — a SyntaxKit CLI for config-driven Swift codegen

`skit` is a small CLI that takes a SyntaxKit DSL file as input and writes Swift source out the other side. The vision: pure data — JSON, YAML, your own format — drives a manifest written in the SyntaxKit DSL, and `skit` materializes it into idiomatic Swift you check in alongside everything else. Less hand-maintenance, fewer drift bugs.

This doc walks through how `skit` is built. Two verbs, one wrap-and-spawn pipeline, two layers of cache, and a careful toolchain story underneath. Where there are sharp edges, this doc names them.

## Two verbs

```
skit run Input.swift # SyntaxKit DSL → Swift source on stdout
skit run Input.swift -o Out.swift
skit run InputDir/ -o OutDir/ # walk **/*.swift and mirror rendered output
skit parse < Input.swift # Swift source → JSON syntax tree
```

`run` is the default subcommand. `skit Input.swift` is shorthand for `skit run Input.swift`. `parse` is a one-shot for the inverse direction (Swift source → SwiftSyntax tree as JSON) — useful for tooling that wants to introspect existing code.

The remainder of this doc is about `run`, which is where the interesting design decisions live.

## How `skit run` works

A `.swift` input file looks like a SyntaxKit DSL expression at the top level:

```swift
// Models.swift
import SyntaxKitHelpers // optional — only if a Helpers/ dir is present

equatableModel("Person", fields: [
("name", "String"),
("age", "Int"),
])
```

The input is not a complete Swift program. It has no `@main`, no `print`, no `let root = …`. `skit run` adds the boilerplate by wrapping the input in a `Group { … }` builder and a top-level `print` that renders the result:

```swift
// What `skit run` writes to a temp file before spawning `swift`:
import SyntaxKit
import SyntaxKitHelpers // hoisted from the input

let __skit_root = Group {
#sourceLocation(file: "/path/to/Models.swift", line: 3)
equatableModel("Person", fields: [
("name", "String"),
("age", "Int"),
])
#sourceLocation()
}

print(__skit_root.generateCode())
```

`skit` then spawns `swift` (the script-mode interpreter, not `swiftc`) on this wrapped file, captures stdout, and writes the result to the user's chosen output target.

Three things deserve a closer look:

**Imports get hoisted.** The wrapper has to start with `import SyntaxKit` so the DSL types are available. The user's `import`s — typically `import SyntaxKitHelpers` plus anything their Helpers/ module references — need to live at the top of the file, not inside `Group { … }`. `skit` parses the input with SwiftSyntax, peels off the leading `import` declarations, and lifts them into the wrapper preamble. Anything else (declarations, expressions, top-level types) stays in the body.

**`#sourceLocation` keeps diagnostics readable.** When the spawned `swift` emits a compile error, it reports a line number in the wrapped temp file, which is meaningless to the user. The `#sourceLocation` directive remaps body diagnostics back to the original input path and line. Errors in the wrapper preamble (the `import` block, the `Group { … }` opening) still reference the temp file — `skit` rewrites occurrences of the temp path in stderr to the input path as a fallback, so users see something coherent.

**`swift` runs in script mode.** Running `swift Input.swift` invokes the Swift interpreter rather than going through `swiftc` + `ld`. Cold-start is around 700ms on macOS; warm spawns are around 110ms. The CLI's hot path leans into this — for batch input via `skit run InputDir/`, we spawn one `swift` per input file in parallel up to the active core count.

## Caches

Two layers, both keyed by content hash. They live under `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS, `$XDG_CACHE_HOME/syntaxkit` (or `~/.cache/syntaxkit`) on Linux.

| Layer | Path | What it skips on hit |
| ------- | ----------------------------- | --------------------------------------------- |
| Helpers | `helpers/<sha>/` | the `swiftc` compile of `Helpers/*.swift` |
| Output | `outputs/<sha>/output.swift` | the `swift` spawn for an input |

**Helpers cache.** Each project's `Helpers/` directory gets compiled into `libSyntaxKitHelpers.{dylib,so}` once and reused. The cache key is a hash of the helper sources, plus the bundled `libSyntaxKit` stamp, plus `swift --version`. Touching a helper file invalidates one shard; updating the toolchain invalidates everything. Hit on a warm cache: skip the ~1–2s `swiftc` compile entirely.

**Output cache.** The fully-rendered output of an input gets cached by a hash of (input bytes, helpers shard, libSyntaxKit stamp, swift version, sorted `SKIT_*`/`SYNTAXKIT_*` env vars). On hit, `skit` doesn't spawn `swift` at all — total wall time is around 0.14s, dominated by hash + file read. Cold miss matches the warm script-mode baseline (~0.5s).

`--no-cache` skips the output cache. There's no flag to skip the helpers cache — invalidate it by touching a helper or bumping the toolchain.

## Toolchain stamping

Pre-compiled Swift modules (`SyntaxKit.swiftmodule`) are tightly coupled to the compiler that built them. Even patch-level differences fail to load:

```
error: module compiled with Swift 6.3 cannot be imported by the Swift 6.3.2 compiler
```

If `skit`'s bundled `lib/SyntaxKit.swiftmodule` doesn't match the user's `swift`, the spawned interpreter emits exactly this diagnostic and refuses to compile the wrapped input. The user is left staring at a cryptic message that doesn't name the actual problem.

`skit` mitigates this by recording the build toolchain at bundle time. `Scripts/build-skit-release.sh` writes `lib/swift-version.txt` containing the output of `swift --version`. On startup, `skit run` reads the stamp and compares it to a freshly-captured local `swift --version`. Three paths:

- **Match.** Proceed silently.
- **Mismatch.** Print a clear error naming both versions and the rebuild command, exit 2. Skip with `--no-toolchain-check`.
- **Missing stamp.** Print a one-line note and proceed. Older bundles built before this check existed shouldn't break.

The comparison is exact-string match. Patch-level drift broke the originating bug, so anything less strict would just defer the failure to the cryptic message we're trying to avoid.

**What this doesn't do**: it doesn't fix the mismatch, just surfaces it. The fix is to re-run the release script with the user's current toolchain. The auto-rebuild path — bundle SyntaxKit sources alongside the prebuilt module and recompile transparently when the stamp doesn't match — is tracked as [issue #157](https://github.com/brightdigit/SyntaxKit/issues/157).

## Timeout watchdog

The spawned `swift` is the only unbounded piece of `skit run`'s hot path. The wrap step is microseconds. Helpers compile is cached. The output cache hits or misses in milliseconds. But the spawn itself runs *user code* — and that code is allowed to be arbitrarily slow, recursive, or stuck.

`skit run` defaults to a 60s per-input timeout. On expiry it sends `SIGTERM`, gives a 5s grace, then `SIGKILL`. The wrapped input exits with code 124 — POSIX `timeout(1)`'s convention. `--timeout <s>` overrides the default; `--timeout 0` disables the watchdog entirely (useful for debugging genuinely long codegen).

The implementation is `DispatchSemaphore.wait(timeout: deadline)` paired with a `process.terminationHandler` that signals on child exit. The Linux Foundation `Process.waitUntilExit()` hangs on already-exited children on some configurations, which is why `skit` uses the semaphore-based wait everywhere. Same story for pipe drains — sequential reads after the child exits can deadlock when either pipe (~64 KB buffer on Linux) fills before exit, so both pipes drain concurrently via `DispatchGroup`.

## Helpers

Shared codegen utilities live in a `Helpers/` directory. `skit` walks up from the input, finds the nearest `Helpers/`, and compiles its sources into a Swift dylib that the wrapped input can `import SyntaxKitHelpers`.

```
project/
├── Helpers/
│ └── Models.swift # public func equatableModel(_:fields:) -> any CodeBlock
└── inputs/
├── Person.swift # imports SyntaxKitHelpers, calls equatableModel(...)
└── Pet.swift # same
```

Files prefixed with `_` are skipped — a convention for private helpers within the helpers module. The module name is hard-coded to `SyntaxKitHelpers`.

The helpers cache is per-content, so editing a helper triggers a fresh compile but reading the same helper across many inputs hits the cache.

## Sharp edges

### `if`-in-`Group` crashes the Swift type-checker — [#158](https://github.com/brightdigit/SyntaxKit/issues/158)

`CodeBlockBuilderResult` declares all the methods needed for `if`/`else` in a result builder (`buildEither`, `buildOptional`, `buildArray`), but using them surfaces a Swift type-checker bug:

```swift
let _ = Group {
if true { // ← error: failed to produce diagnostic for expression
Struct("A") { … }
}
}
```

This is a Swift compiler bug, not a `skit` bug. The workaround is to hoist the conditional into a helper function that uses **plain Swift `if`/`else`** (not a `Group { if … }` body) to return one of two `CodeBlock`s:

```swift
// In Helpers/Models.swift
public func optionalDebugField(_ include: Bool) -> any CodeBlock {
if include {
return Variable(.let, name: "debug", type: "Bool")
} else {
return Group {} // empty Group as the "no-op" branch — there's no
// public EmptyCodeBlock type
}
}

// In Input.swift
Struct("Config") {
Variable(.let, name: "name", type: "String")
optionalDebugField(buildIsDebug)
}
```

The helper itself can't use `Group { if … }` either — same crash. Plain Swift control flow only.

### `@main` and top-level decl attributes don't work

`@main` and other decl attributes (like top-level `@available`) at the start of the input fail to compile. The wrapper places the user's body inside `Group { … }`, where these attributes try to bind to a function-call expression rather than a declaration. Examples:

```swift
@main // ❌ error: expected declaration
Struct("Person") { … }

@available(iOS 17, *) // ❌ error: expected declaration
Struct("ModernView") { … }
```

The DSL has its own mechanism for attribute attachment — call `.attribute("Published")`, `.attribute("available", arguments: ["iOS 17", "*"])` on the `Variable`/`Struct`/etc. — which renders the attribute in the *output*, not on the DSL expression itself. That's the path users should take.

### Comments don't carry through

Swift comments in the input file (`// MARK: - Models`, block comments, inline `//` after a DSL line) don't render to the output. The input's comments are *only* for the input's author. To emit a comment in the rendered code, attach it to a `CodeBlock` via `.comment { Line(.doc, "…") }`.

### `#if os(...)` is gen-time, not output

A `#if os(macOS) … #endif` block in the input is evaluated when the wrapped file compiles, controlling whether the enclosed DSL expressions are part of the builder. It does *not* emit a `#if os(macOS)` directive into the rendered Swift output. If you want compile-time-conditional output, emit the `#if` lines as raw text via `VariableExp("#if os(macOS)")` (or write a small helper).

## Platform notes

**macOS** is the primary target. The build and release flows live in `Scripts/`; the bundle is portable across machines with the same Swift version.

**Linux** is verified on `swift:6.0-jammy/aarch64`. One adjustment compared to macOS: the Mach-O `install_name` rewrite in `Scripts/build-skit-release.sh` is skipped — GNU `ld` doesn't accept the flag. The `-rpath` injection (which is what actually locates the dylib at runtime) works on both platforms.

The Foundation.Process workarounds described earlier are Linux-driven. `Process.waitUntilExit()` blocks indefinitely on already-exited children on `swift:6.0-jammy/aarch64` — `skit` uses `DispatchSemaphore` everywhere a child wait is needed, and drains stdout/stderr pipes concurrently to avoid deadlocks when either pipe buffer fills.

**Windows** is not supported.

## What's deferred

A few things were considered for v1 and explicitly punted:

- **Auto-rebuild on toolchain mismatch** — [#157](https://github.com/brightdigit/SyntaxKit/issues/157). Today the user gets a clear error and a rebuild command; tomorrow `skit` should rebuild `libSyntaxKit` from bundled sources transparently and cache the result per Swift version. The stamp-and-detect path shipped in this release is the foundation; the rebuild fallback is the natural next step.
- **Multi-file output from a single input** — out of scope. The folder mode handles N-inputs → N-outputs; if you need fan-out from one logical generator, split it into multiple input files.
- **Sandboxing the spawned `swift`** — out of scope. The threat model is "you ran your own code", same as `swift Input.swift` by hand.
- **HTTP / web-server form** — out of scope for the CLI. A long-lived server could share a warm interpreter and the caches across requests, but that's a different shape with its own isolation questions. Revisit later.

## Reference

- [`Sources/skit/README.md`](../Sources/skit/README.md) — per-target quick reference (flag table, helpers layout).
- [`Scripts/build-skit-release.sh`](../Scripts/build-skit-release.sh) — release-bundle builder.
- [`Docs/research/tuist-manifest-pipeline.md`](research/tuist-manifest-pipeline.md) — the manifest-pipeline pattern this CLI borrows from.
- [Issue #154](https://github.com/brightdigit/SyntaxKit/issues/154) — original tracking issue.
- [Issue #157](https://github.com/brightdigit/SyntaxKit/issues/157), [Issue #158](https://github.com/brightdigit/SyntaxKit/issues/158) — follow-ups.
9 changes: 6 additions & 3 deletions Examples/Completed/attributes/dsl.swift
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
Class("Foo") {
Variable(.var, name: "bar", type: "String", defaultValue: "bar").attribute("Published")
Variable(.var, name: "bar", type: "String", equals: "bar").attribute("Published")
Function("bar") {
print("bar")
Call("print") {
ParameterExp(unlabeled: Literal.string("bar"))
}
}.attribute("available", arguments: ["iOS 17.0", "*"])
Function("baz") {
}.attribute("objc")}.attribute("objc")
}.attribute("objc")
}
Loading