Embed tree-sitter wasm as ~268 chunked base64 string literals

Three previous approaches all failed on Windows in subtly different ways:

 1. Single 274KB base64 string literal: bun's Windows minifier dropped
    or transformed it (build verified the prefix wasn't in the binary
    even though the embed step wrote the file).
 2. `with { type: 'file' }` from a node_modules subpath: bytes ended up
    in the binary but the import variable was bound to undefined at
    runtime — bun on Windows mishandles the JS-level binding for that
    attribute.
 3. `with { type: 'file' }` from a relative path (wasm copied into
    pre-init/): same as #2 — confirms it's not subpath-vs-relative,
    it's a bun/Windows bug with the import-attribute binding.

Round 4: write the base64 as ~268 small chunks (1024 chars each) in an
exported array, joined and decoded at runtime in the pre-init. Each
chunk is referenced unconditionally at runtime via .join(''), so DCE
can't eliminate it; each is small enough that no minifier heuristic
would treat it as a special "huge string literal" worth dropping.

- cli/scripts/build-binary.ts: embedTreeSitterWasmAsChunks() writes the
  full array, returns sample chunks (start/middle/end) for the post-
  build verification scan to look for in the compiled binary. Restores
  the empty stub eagerly + via process.on('exit').
- cli/src/pre-init/tree-sitter-wasm-bytes.ts: re-introduced as a stub
  exporting an empty readonly string[]. Dev-mode and unit tests see
  the empty stub; production builds get the real chunks written in by
  build-binary.ts.
- cli/src/pre-init/tree-sitter-wasm.ts: import the chunks, .join(''),
  Buffer.from(_, 'base64'), publish on globalThis. The if() guard
  remains because dev mode legitimately has zero chunks.

Verified locally: build embeds 268 chunks, post-build verifies 3 sample
chunks at distinct offsets in the compiled binary, --smoke-tree-sitter
exits 0 with "tree-sitter smoke ok (wasmBinary, 205488 bytes)", full
smoke passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: jahooma

cli/.gitignore

@@ -7,7 +7,3 @@ debug/
 
 # Generated files
 src/agents/bundled-agents.generated.ts
-
-# Staged by build-binary.ts before `bun build --compile`, removed after.
-# See cli/src/pre-init/tree-sitter-wasm.ts for why we copy this in.
-src/pre-init/tree-sitter.wasm

cli/scripts/build-binary.ts

@@ -145,9 +145,10 @@ async function main() {
   patchOpenTuiAssetPaths()
   await ensureOpenTuiNativeBundle(targetInfo)
 
-  const wasmCopy = stagePreInitWasm()
-  // Even on a build-script crash, leave the developer's working tree clean.
-  process.on('exit', wasmCopy.cleanup)
+  const treeSitterEmbed = embedTreeSitterWasmAsChunks()
+  // Even on a build-script crash, restore the empty stub so a developer's
+  // working tree doesn't end up with a multi-MB diff.
+  process.on('exit', treeSitterEmbed.restore)
 
   const outputFilename =
     targetInfo.platform === 'win32' ? `${binaryName}.exe` : binaryName
@@ -190,17 +191,16 @@ async function main() {
 
   runCommand('bun', buildArgs, { cwd: cliRoot })
 
-  // Remove the staged pre-init wasm now that the build has read it. Eager
-  // cleanup keeps a successful build clean; the exit handler above is a
-  // backstop for crashes between stage and now.
-  wasmCopy.cleanup()
+  // Restore the empty stub now that the build read the chunks. Eager
+  // cleanup keeps a successful build clean; the exit handler is a
+  // backstop for crashes between embed and now.
+  treeSitterEmbed.restore()
 
-  // Fail the build if the wasm asset didn't actually make it into the
-  // compiled binary. The pre-init imports tree-sitter.wasm with `with {
-  // type: 'file' }`, which Bun should embed; this scan catches silent
-  // regressions (e.g. tree-shaking eliminating the import) before we ship
-  // a broken artifact.
-  verifyTreeSitterWasmEmbedded(outputFile)
+  // Fail the build if the chunks didn't actually make it into the
+  // compiled binary. Catches silent regressions (tree-shaking, minifier
+  // dropping literals, file-write timing) before we upload an artifact
+  // that would crash for users.
+  verifyTreeSitterWasmEmbedded(outputFile, treeSitterEmbed.sampleChunks)
 
   if (targetInfo.platform !== 'win32') {
     chmodSync(outputFile, 0o755)
@@ -247,82 +247,107 @@ function findWebTreeSitterWasm(): string {
 }
 
 /**
- * Copy `tree-sitter.wasm` into `cli/src/pre-init/` so the pre-init module
- * can import it via a relative `with { type: 'file' }` path. We can't
- * import it directly as a node_modules subpath: on Windows, bun's
- * `with { type: 'file' }` resolution returned falsy at runtime for
- * `web-tree-sitter/tree-sitter.wasm` even though the bytes ended up in
- * the binary, breaking the pre-init's runtime path lookup. OpenTUI's own
- * tree-sitter assets work because they're imported relatively from
- * inside the package — same trick here.
+ * Inline `tree-sitter.wasm` into the binary as base64-encoded string
+ * literals — but split into many small chunks. A single 274KB string
+ * literal got dropped/transformed by bun's Windows minifier in an
+ * earlier attempt; small chunks are individually unremarkable to the
+ * minifier and survive intact. The pre-init joins them at runtime and
+ * decodes back to the wasm bytes.
  *
- * Returns a cleanup function. The build calls it eagerly after compile
- * and registers it as an exit handler so a mid-build crash doesn't leave
- * a multi-MB untracked file in the working tree.
+ * Returns a `restore` function (resets the stub) and a small set of
+ * `sampleChunks` for the post-build verification step to look for in
+ * the compiled binary. Always invoke `restore` (eagerly + on exit) so
+ * a developer's working tree doesn't end up with a multi-MB diff after
+ * a build.
  */
-function stagePreInitWasm(): { cleanup: () => void } {
-  const sourceWasm = findWebTreeSitterWasm()
-  const stagedPath = join(cliRoot, 'src', 'pre-init', 'tree-sitter.wasm')
-  let cleaned = false
-  const cleanup = (): void => {
-    if (cleaned) return
-    cleaned = true
-    if (existsSync(stagedPath)) {
-      try {
-        rmSync(stagedPath)
-      } catch (error) {
-        console.error('Failed to remove staged pre-init wasm:', error)
-      }
+function embedTreeSitterWasmAsChunks(): {
+  restore: () => void
+  sampleChunks: string[]
+} {
+  const stubPath = join(cliRoot, 'src', 'pre-init', 'tree-sitter-wasm-bytes.ts')
+  const originalStub = readFileSync(stubPath, 'utf8')
+  let restored = false
+  const restore = (): void => {
+    if (restored) return
+    restored = true
+    try {
+      writeFileSync(stubPath, originalStub)
+    } catch (error) {
+      console.error('Failed to restore tree-sitter-wasm-bytes stub:', error)
     }
   }
 
-  // Read + write rather than copyFile so we don't accidentally hardlink
-  // (some Windows hosts fail to delete hardlinks while bun has the file
-  // mmapped from the compile step).
-  writeFileSync(stagedPath, readFileSync(sourceWasm))
-  logAlways(`Staged pre-init wasm: ${sourceWasm} → ${stagedPath}`)
-  return { cleanup }
+  const sourceWasm = findWebTreeSitterWasm()
+  const wasmBytes = readFileSync(sourceWasm)
+  const fullBase64 = wasmBytes.toString('base64')
+
+  // ~1KB per chunk: well under any plausible minifier-dropped-literal
+  // threshold, and small enough that even a heavy-handed inliner would
+  // emit them as runtime references rather than evaluating the whole
+  // .join() at compile time. Keeps total chunk count manageable too
+  // (~270 chunks for a 205KB wasm).
+  const CHUNK_SIZE = 1024
+  const chunks: string[] = []
+  for (let i = 0; i < fullBase64.length; i += CHUNK_SIZE) {
+    chunks.push(fullBase64.slice(i, i + CHUNK_SIZE))
+  }
+
+  const generated =
+    `// AUTO-GENERATED by cli/scripts/build-binary.ts during \`bun build --compile\`.\n` +
+    `// Restored to the empty stub after the build finishes — do not commit a\n` +
+    `// non-empty value here.\n` +
+    `export const TREE_SITTER_WASM_BASE64_CHUNKS: readonly string[] = [\n` +
+    chunks.map((c) => `  ${JSON.stringify(c)},`).join('\n') +
+    `\n]\n`
+
+  writeFileSync(stubPath, generated)
+  logAlways(
+    `Embedded tree-sitter.wasm from ${sourceWasm} (${wasmBytes.length} bytes → ${chunks.length} chunks of ~${CHUNK_SIZE} chars).`,
+  )
+
+  // Pull a few sample chunks from the start, middle, and end for the
+  // post-build verification scan. If any one is missing in the compiled
+  // binary, something dropped or transformed the literals.
+  const samples = [
+    chunks[0],
+    chunks[Math.floor(chunks.length / 2)],
+    chunks[chunks.length - 1],
+  ].filter((c): c is string => Boolean(c))
+
+  return { restore, sampleChunks: samples }
 }
 
 /**
- * Sanity-check the compiled binary actually contains web-tree-sitter's
- * tree-sitter.wasm. The pre-init imports it via `with { type: 'file' }`,
- * which should bundle the asset at a bunfs path. If tree-shaking or a
- * future bundler change drops the import, the binary still compiles but
- * tree-sitter init fails at runtime — this scan fails the build before
- * we upload that artifact.
- *
- * Looks for the actual wasm bytes (a unique 64-byte chunk pulled from
- * the source file's interior), not just the wasm magic header — OpenTUI
- * embeds its own tree-sitter language wasms, so a magic-bytes-only scan
- * would false-pass even without our import. A literal bytes match
- * proves *this specific* wasm shipped.
+ * Sanity-check the compiled binary actually contains all the chunked
+ * base64 we just embedded. We pass in a few sample chunks from the
+ * start / middle / end of the array; each must appear in the binary.
+ * If any one is missing, the bundler dropped or inlined-away part of
+ * the literal table, and the runtime decode would produce garbage.
  */
-function verifyTreeSitterWasmEmbedded(outputFile: string): void {
-  const wasmPath = findWebTreeSitterWasm()
-  const wasm = readFileSync(wasmPath)
-  // Take a 64-byte slice from the middle of the file. The header has
-  // generic wasm magic + section markers; the tail can be padding. The
-  // middle is densely packed code/data unique to this specific wasm
-  // module.
-  const needleStart = Math.floor(wasm.length / 2)
-  const needle = wasm.subarray(needleStart, needleStart + 64)
-
+function verifyTreeSitterWasmEmbedded(
+  outputFile: string,
+  sampleChunks: string[],
+): void {
+  if (sampleChunks.length === 0) {
+    throw new Error('verifyTreeSitterWasmEmbedded called with no sample chunks')
+  }
   const binary = readFileSync(outputFile)
-  const idx = binary.indexOf(needle)
-  if (idx === -1) {
-    throw new Error(
-      `web-tree-sitter wasm content not found in ${outputFile}.\n` +
-        `Source wasm: ${wasmPath} (${wasm.length} bytes)\n` +
-        `Searched for 64 bytes from offset ${needleStart} of the source.\n` +
-        `Either the \`with { type: 'file' }\` import in the pre-init was\n` +
-        `tree-shaken out, or bun --compile didn't embed the asset on this\n` +
-        `platform. The runtime tree-sitter init would fail with\n` +
-        `"Internal error: tree-sitter.wasm not found".`,
-    )
+  for (const chunk of sampleChunks) {
+    const needle = Buffer.from(chunk, 'utf8')
+    const idx = binary.indexOf(needle)
+    if (idx === -1) {
+      throw new Error(
+        `Embedded tree-sitter wasm chunk not found in ${outputFile}.\n` +
+          `Missing chunk (first 80 chars): ${chunk.slice(0, 80)}…\n` +
+          `Either the \`tree-sitter-wasm-bytes.ts\` literals were tree-shaken,\n` +
+          `the minifier transformed them away, or the pre-init's import wasn't\n` +
+          `actually consumed. The runtime tree-sitter init would fail with\n` +
+          `"Internal error: tree-sitter.wasm not found".`,
+      )
+    }
   }
   logAlways(
-    `Verified embedded tree-sitter.wasm at offset ${idx} of compiled binary (source: ${wasmPath}).`,
+    `Verified ${sampleChunks.length} embedded base64 chunks in compiled binary.`,
   )
 }
 

cli/src/pre-init/tree-sitter-wasm-bytes.ts

@@ -0,0 +1,14 @@
+// Stub committed for dev mode and tests. The real wasm chunks are written
+// here by `cli/scripts/build-binary.ts` immediately before
+// `bun build --compile`, then restored to an empty array after the build
+// completes. Dev mode and unit tests see the empty stub and fall back to
+// path-based resolution in `packages/code-map/src/init-node.ts` (which
+// works locally because `node_modules/web-tree-sitter/tree-sitter.wasm`
+// exists on the filesystem).
+//
+// Why an array of small chunks rather than one big string: a single
+// 274KB string literal got dropped/transformed by bun's Windows
+// minifier (the binary built clean but ran without the bytes). Many
+// small string literals slip under whatever threshold caused that. See
+// `cli/src/pre-init/tree-sitter-wasm.ts` for the full failure history.
+export const TREE_SITTER_WASM_BASE64_CHUNKS: readonly string[] = []

cli/src/pre-init/tree-sitter-wasm.ts

@@ -4,68 +4,42 @@
 // parser, and its init reads what we publish here on `globalThis` and via
 // the env var.
 //
-// Why `with { type: 'file' }` rather than embedding base64 in TS source:
-// the latter doesn't survive `bun --compile` on Windows. The base64 string
-// gets dropped or transformed somewhere in the bundle/minify pipeline, so
-// the runtime sees an empty stub even though the build script wrote the
-// real bytes. `with { type: 'file' }` is Bun's documented asset-embed
-// path — the file gets placed at a bunfs location the runtime can read.
-
-import { readFileSync } from 'fs'
-
-// Important: this is a *relative* import of a wasm file the build script
-// copies in from `web-tree-sitter/tree-sitter.wasm` immediately before
-// `bun build --compile`. On Windows, bun's `with { type: 'file' }`
-// returned falsy at runtime when this import was a node_modules subpath
-// (`web-tree-sitter/tree-sitter.wasm`) even though the bytes ended up in
-// the binary — OpenTUI works around the same issue by using relative
-// paths from inside its own package, which is what we're mirroring here.
+// History of failed approaches before this one:
 //
-// The `.wasm` lives at `./tree-sitter.wasm` next to this file. It is
-// .gitignored; build-binary.ts copies it in before compile and removes
-// it after, so dev-mode runs see no `.wasm` here and fall back to
-// path-based resolution via init-node.ts (which works locally).
+//  1. `with { type: 'file' }` import of `web-tree-sitter/tree-sitter.wasm`
+//     (node_modules subpath) — bun --compile on Windows embedded the
+//     bytes but bound the import variable to undefined.
+//  2. `with { type: 'file' }` import of a copied-in relative wasm file —
+//     same problem; this turns out to be a bun/Windows bug, not a
+//     subpath-vs-relative thing.
+//  3. Single 274KB base64 string literal in a generated TS module —
+//     bun's Windows minifier dropped/transformed the literal even
+//     though the embed step wrote it.
 //
-// @ts-expect-error - TS has no loader for .wasm; bun's `with { type: 'file' }`
-// returns a string path at compile time.
-import treeSitterWasmPath from './tree-sitter.wasm' with { type: 'file' }
+// What works: many small base64 chunks (each well under any plausible
+// minifier threshold) joined at runtime. The build script writes the
+// chunks; this module decodes them. The committed file ships an empty
+// stub array — dev-mode runs see no chunks and fall through to
+// path-based resolution in init-node.ts (which works locally because
+// `node_modules/web-tree-sitter/tree-sitter.wasm` exists on disk).
 
-let embeddedWasm: Uint8Array | undefined
+import { TREE_SITTER_WASM_BASE64_CHUNKS } from './tree-sitter-wasm-bytes'
 
-if (treeSitterWasmPath) {
-  // Path stays for the locateFile fallback in init-node.ts. That fallback
-  // accepts bunfs-style paths (`/~BUN/root/...`) without checking
-  // fs.existsSync, because fs.existsSync misreports those paths on Windows.
-  // emscripten's wasm loader will fs.readFile them through its own runtime.
-  process.env.CODEBUFF_TREE_SITTER_WASM_PATH = treeSitterWasmPath
-
-  // Also try a synchronous read so we can hand the bytes straight to
-  // Parser.init via wasmBinary — bypassing locateFile entirely is the most
-  // robust path. If readFileSync of the bunfs path throws on this OS (we've
-  // seen this happen on Windows in some configurations), log it loudly so
-  // the smoke check / user reports include the diagnostic, then fall
-  // through to the locateFile flow.
-  try {
-    const buf = readFileSync(treeSitterWasmPath)
-    embeddedWasm = new Uint8Array(buf.buffer, buf.byteOffset, buf.byteLength)
-    ;(
-      globalThis as { __CODEBUFF_TREE_SITTER_WASM_BINARY__?: Uint8Array }
-    ).__CODEBUFF_TREE_SITTER_WASM_BINARY__ = embeddedWasm
-  } catch (err) {
-    console.error(
-      '[tree-sitter pre-init] readFileSync failed for embedded wasm at',
-      treeSitterWasmPath,
-      '—',
-      err instanceof Error ? err.message : String(err),
-    )
-  }
+let embeddedWasm: Uint8Array | undefined
+if (TREE_SITTER_WASM_BASE64_CHUNKS.length > 0) {
+  // Joined string is up to ~275KB but only lives long enough to decode.
+  const buf = Buffer.from(TREE_SITTER_WASM_BASE64_CHUNKS.join(''), 'base64')
+  embeddedWasm = new Uint8Array(buf.buffer, buf.byteOffset, buf.byteLength)
+  // globalThis is the only cross-bundle channel: the SDK pre-built bundle
+  // inlines its own copy of `init-node.ts`, so a module-level variable
+  // here isn't visible to the singleton initialized via the SDK.
+  ;(
+    globalThis as { __CODEBUFF_TREE_SITTER_WASM_BINARY__?: Uint8Array }
+  ).__CODEBUFF_TREE_SITTER_WASM_BINARY__ = embeddedWasm
 }
 
-// `--smoke-tree-sitter` is the deterministic CI gate. We can't handle it
-// here with top-level await — bun --compile on Windows didn't preserve the
-// blocking semantics in our last attempt, so commander still ran and
-// rejected the unknown flag. Instead, the handler lives at the top of
-// main() in cli/src/index.tsx (before parseArgs), where we can synchronously
-// short-circuit before commander parses argv. This module's job is just to
-// publish the wasm bytes / path on globalThis + process.env so that the
-// handler (and the SDK's eager Parser.init) can find them.
+// `--smoke-tree-sitter` is the deterministic CI gate. The handler lives at
+// the top of main() in cli/src/index.tsx (before parseArgs), not here —
+// top-level await in this module didn't actually pause subsequent module
+// evaluation under bun --compile on Windows. See the comment over the
+// handler in index.tsx for the full reasoning.