feat(bundle): bundle the bit cli with esbuild - #10590
Draft
GiladShoham wants to merge 471 commits into
Draft
GiladShoham wants to merge 471 commits into
GiladShoham wants to merge 471 commits into
Conversation
covers the design, the four problems that mattered, verification (40 commands + the full build pipeline), what is installed next to the bundle and why, script vs sea timings, the publishable package layout, and the open questions.
the bundler spawns child processes with other cwds (npm install inside the bundle dir, the bundle introspection for the esm bridges), so a relative out-dir resolved differently per step.
setup_esbuild_bundle compiles the workspace and builds the bundle once, then persists it; e2e_test_esbuild_bundle symlinks its launcher onto PATH and reuses the existing e2e_test_cmd (same file splitting, timings and junit output) with --bit_bin=bit-bundled. Gated to ^bit-bundle.* branches until it is green.
bit install compiles by default, which is why setup_harmony has bbit compile commented out - the attached workspace already has every component's dist/. Building the bundle needs only node and npm, so bvm is not needed either.
it is declared nowhere in the repo and imported by nothing - the reference came from bit-bundle2, which predates the split of legacy into per-concern packages that bundle like any other component. it resolved locally only because a stale copy sat in node_modules, so the bundle built here and failed on CI. the extras list is now filtered to what is actually installed, so a missing optional extra warns instead of failing the build.
23/2876 baseline failures vs 41/2837 bundled; all baseline failures reproduce in the bundle and none are unique to it, so the delta is 18 real regressions. 12 of them are the already-documented UI-bundling and require.resolve surface; the zlib inflate failure and the startup-budget miss are new.
…i-bundler the logic had to leave @teambit/bit before an env for @teambit/bit could use it: an env that imports the component it builds is a cycle. the component now takes the core aspect ids and the packages root as inputs instead of importing manifests and resolving from the running process, so both callers - npm run bundle and the build task - drive the same code. adds teambit.harmony/envs/bit-cli-app-env, extending core-aspect-env with a BundleCliApp task that reads the ids from the capsule's compiled manifests and bundles the capsule rather than the running installation.
…s shims it is a locator that linkCoreAspect dereferences, not a package, and node picks package.json main over index.js for a directory require - so the two roles cannot share one directory. lists the three publishable layouts.
shims move to dist/core-aspects/node_modules/@teambit/<name> and the bundle to dist/core-aspects/bundle. node's upward node_modules walk from the bundle file finds them, so resolution needs no runtime change, and npm strips only the root node_modules so a nested one publishes (verified with npm pack --dry-run). the bundler now also emits the per-aspect locators at dist/<name>/index.js that DependencyLinker.linkCoreAspect dereferences, which is what lets a bundled build drop @teambit/aspect's CoreExporterTask. externals become ordinary dependencies of the distribution's package.json instead of a second package.json inside the bundle dir. config.ts documents the layout and the rejected alternatives. verified: bit install in a workspace links @teambit/workspace to the shim, and --version/status/list/show/compile all pass.
…ession entry point: current state, the immediate next step (env set -> install -> build) with the failure modes to expect from the never-yet-executed build task, the 18 bundle-only e2e failures in priority order, the commands, the CI setup, and the repo gotchas.
…indow both call sites treat an expected, temporary condition as fatal. an install re-injects the workspace packages, deleting every component's dist, and rebuilds them in the compile step that runs last. code in between that reaches a lazily-required module dies on a dist that existed when the process started - aborting the install *before* the compile that would have restored it, which leaves the workspace unable to run bit at all. - syncCoreAspectLinksForEnvs guarded its trailing createLinks but not the body that throws (a lazy require of @teambit/aspect-loader for getCoreAspectName). its own comment already documents the bridge as best-effort. - resolveAspects resolved core-aspect defs with an unguarded Promise.all(map(getAspectDef)), ignoring the throwOnError: false that InstallMain.reloadMovedEnvs explicitly passes.
…the env to @teambit/bit generatePackageJson read workspace.jsonc and package.json from packagesRoot unconditionally. both are repo-root files that do not exist in a capsule, so the build task threw before writing anything. they are only fallbacks - in a capsule every external is a real dependency, so the installed copy answers first - hence degrade to undefined instead of throwing. also assigns bit-cli-app-env to teambit.harmony/bit in .bitmap.
…tall loop the task runs; the capsule premise does not hold - its node_modules carries published @teambit/*@1.0.1097 rather than the freshly compiled workspace components, and is missing 35 core aspects outright. also documents the install/compile deadlock between the release binary and bd, and the recovery sequence out of it.
…indow both call sites treat an expected, temporary condition as fatal. an install re-injects the workspace packages, deleting every component's dist, and rebuilds them in the compile step that runs last. code in between that reaches a lazily-required module dies on a dist that existed when the process started - aborting the install *before* the compile that would have restored it, which leaves the workspace unable to run bit at all. - syncCoreAspectLinksForEnvs guarded its trailing createLinks but not the body that throws (a lazy require of @teambit/aspect-loader for getCoreAspectName). its own comment already documents the bridge as best-effort. - resolveAspects resolved core-aspect defs with an unguarded Promise.all(map(getAspectDef)), ignoring the throwOnError: false that InstallMain.reloadMovedEnvs explicitly passes.
they are not missing from the capsule. capsules share a hoisted root whose node_modules/@teambit has 299 entries, including all 35 the bundler reported as 'not installed'. the bug is that getCoreAspectsInfo path-joins instead of resolving, so it never looks one level up. same bug in teambit-dist-resolver-plugin.
the bundler moved to teambit.harmony/modules/cli-bundler but three references still pointed at @teambit/bit/dist/bundle/ensure-bundle.js, which no longer exists - CI failed with MODULE_NOT_FOUND on 'npm run bundle:ensure'. also fingerprint the cli-bundler dist for the staleness stamp. it only covered the thin arg parser left in @teambit/bit, so edits to the actual bundling logic left the stamp unchanged and ensure-bundle would reuse a stale artefact.
…/<pkg> the bundler located every package by path-joining onto packagesRoot. that holds for this repo and breaks in a capsule, where most dependencies hoist to the shared capsule root - so a real build found 71 of 106 core aspects, 70 of them 'without a main runtime', copied 0 of 4 runtime assets and resolved 3 of 11 external versions. all of it silently. - resolvePackageDir walks the node_modules chain the way node does, and returns the realpath so a pnpm-linked package's own dependencies stay on the resolution path. - findRuntimeAndAspectFiles looks in dist/ as well as the top level; published packages and capsules keep the compiled runtime files there. - specifiers keep the extension under dist/ and drop it at the top level - the two take different branches of the exports map and neither extension-probes. - the dist resolver keys off componentId rather than _bit_local, which a capsule's copies don't carry, and falls back to esbuild for non-workspace components instead of failing the build. - unresolved externals now warn: they are excluded from the bundle, so dropping one silently yields a runtime 'Cannot find module'. the task now reports 106 aspects (9 genuinely UI-only), copies 105 assets and builds a 69MB bundle with 107 shims and 107 locators.
…rect 53 capsules against a bit status of 2 new + 51 modified confirms the rule: new/modified components are built into sibling capsules and linked fresh, unmodified ones install from the registry - where the published package is by definition the current code. the earlier concern that the bundle would ship stale aspects was unfounded.
bit-cli-app-env is itself an env, so teambit.envs/env; cli-bundler is a plain node module rather than an aspect, so node-babel-mocha.
…/@teambit/bit/bin/bit
a user's workspace resolves @teambit/<aspect> to a shim whose body is one dynamic require, so without declarations every import degraded to any - no type checking, no autocomplete. the .d.ts tree is copied verbatim from the package being shimmed rather than regenerated. that preserves type identity: declarations re-export their siblings and other @teambit/* packages, and those references resolve through the sibling shims, so every aspect sees the same Component and Workspace. rolling them up per package would break exactly that. the whole tree is copied, not just index.d.ts, because the re-exports are relative paths into it. the shim's exports map points 'types' at the copied declarations - inheriting the original, which points at .ts sources a shim does not have, would leave everything untyped. capsules always carry declarations; this repo needs 'bit compile --generate-types'. both verified, 107/107 shims with 1722 files. an external workspace type-checks against them under noImplicitAny, and a negative control confirms the types are enforced rather than silently any: ws.path resolves as string and cm.toArray() as [Component, string][], the Component coming from a sibling shim.
…k status docs.mdx explains each module and the traps: resolution is not path-joining, exports maps don't extension-probe, a missing main runtime is legitimate so a resolution bug looks like a normal build. plan gains 9e - the task runs green, what the first runs exposed, freshness confirmed correct, types verified in an external workspace, and what is still open.
the task wrote to <capsule>/app-bundle, so the build produced a prototype dir that something would
later have had to lift into the package. it now builds into the capsule itself, which *is* what gets
published, giving the 9b layout directly: package.json + bin/bit + dist/<aspect>/index.js locators +
dist/core-aspects/{bundle,node_modules}.
inPlace changes three things: never clean (the out dir holds the component's own sources and dist -
cleanOutDir would delete them), merge into the real package.json instead of writing the
@teambit/bit-bundle-externals stand-in, and skip .npmrc, which only exists for the prototype's local
npm install.
the merge prunes the dependency surface to the externals alone - 168 declared dependencies replaced
by 7. leaving them would make a consumer's install re-download the very tree the bundle replaces,
and would resolve a second copy of every core aspect next to the shims, so @teambit/workspace could
resolve to a published package rather than the bundle slice. dev/peer/optional deps go too;
identity fields (name, version, componentId, engines) are untouched.
also excludes dist/core-aspects from the .d.ts copy: in place, @teambit/bit's source dir is the
capsule, whose dist now contains the generated shims, so an unfiltered glob copied all 106 shims'
declarations into the bit shim (1158 files instead of 18).
verified: the capsule's sources and dist survive, and the built package runs --version, init, status
and list from a fresh workspace.
MochaMain.createTester() has no callers anywhere in the repo or in any resolved env - real consumers (node-babel-mocha, node-typescript-mocha) already import MochaTester directly from @teambit/defender.mocha-tester, which has no dependency on this aspect. Being an eager BitMain dependency, it forced mocha's require chain onto every bit invocation for no reason.
Pulls in the mocha core-aspect removal (750f930).
mocha is no longer needed - the @teambit/mocha core aspect that pulled it in unconditionally is gone (merged from remove-core-envs-from-manifest). Rebuild verified: externalsInstalled 11 -> 10, coreAspects 106 -> 105, zero require sites for mocha in bit.app.js. bundle-plan.md records the webpack/mocha externals research and the mocha removal end to end.
…VendorDllPaths Wires resolveUiVendorDllPackages/buildUiVendorDll into BundleUiTask.execute() and adds UiMain.getUiVendorDllPaths() so external consumers (e.g. bit-cloud) can discover the shipped vendor DLL artifact. Fixes found and corrected while wiring against a real build: - resolveUiVendorDllPackages derived package names with the non-core naming convention instead of the core-aspect one; used getCoreAspectPackageName from @teambit/aspect-loader instead (not @teambit/bit, which would balloon teambit.ui-foundation/ui's own dependency graph and break the isolator). - buildUiVendorDll required each package's whole main entry instead of its specific .ui.runtime.js/.preview.runtime.js file, pulling in unrelated non-browser-safe code (e.g. a native binary loader via @teambit/pnpm). - @teambit/ui's barrel re-exports BundleUiTask as a real value, making the new ui-vendor-dll.ts module (and its rspack/webpack loader dependencies) require()-reachable from client-rendered code; externalized these in both rspack.browser.config.ts and the DLL's own compiler config. - buildUiVendorDll's own rspack config lacked the CSS/SCSS loader, JSX/TSX transform, and resolve fallback/alias table the existing pre-bundle already has, so real core-aspect ui.runtime files (which pull in the actual UI component library) failed to bundle; now reuses rspack.common's pieces. Verified via a real `bd build teambit.ui-foundation/ui --tasks BundleUI`: existing pre-bundle unaffected, ui-vendor-dll/ produced with real content (vendor.js ~6.4MB covering 3393 real modules, correctly keyed by exact runtime-file path).
…ted SSR-size regression
rspack keys a dll manifest by each module's path relative to the build's context, and DllReferencePlugin matches by recomputing that path on the consuming side. 2629 of the shipped manifest's 3393 keys ran through pnpm's .pnpm/<name>@<version>_<peer-hash> store directories, whose hash comes from the building install's own dependency graph - no separate project reproduces it, so every entry silently missed and recompiled from source. Re-key the shipped manifest by package name + subpath, and add createUiVendorDllReference() to turn it back into DllReferencePlugin options for whatever install is consuming it. Also expose vendor.css through getUiVendorDllPaths().
Corrects 26's "intercepts transparently" claim, adds D17 and a Task 6 findings entry with the measured before/after from a real separate-install repro, and amends gap 1's mitigation note.
… for cross-rspack-version consumers
…pm-hoisting workaround, remaining coverage gap
…mport gap for real
…kages from the vendor dll
…ender verification
The browser config already externalizes @rspack/core to keep the vendor-dll import chain out of the client bundle; the ssr config had no externals at all, so the same chain (via @teambit/ui's barrel re-exporting BundleUiTask) dragged @rspack/core's ~40 MB native binding into the shipped ssr bundle. Verified safe against a real default build where @rspack/core isn't installed: 16/16 e2e passing, including real SSR render.
It ran unguarded inside e2e_test_esbuild_bundle's sweep and hit the
missing --ui-bundling toolchain ("Cannot find module 'assert/'"),
unlike its ui-start.e2e.ts/ui-ssr.e2e.ts siblings which already skip
unless BIT_E2E_UI_MODE is set.
Fails CI if any part of the bundle grows more than 10% past baseline -
direct response to the SSR bundle's 6 MB -> 53 MB regression. Two
phases (pre/post UI-preview injection) so the cheaper check fails fast
before the 40-way e2e fan-out starts.
Also fixes lint-staged's oxlint glob to skip scripts/, matching
.oxlintrc.json's own ignorePatterns - it was erroring on any staged
scripts/*.mjs file ("no files found to lint") since oxlint treats an
explicitly-passed but ignore-listed path as zero matched files.
…er check Pre-phase baseline was captured on macOS and undercounted externals by ~8.7% vs. real CircleCI (Linux) numbers, eating most of the margin on the first run. Recalibrated from the actual CI job's report. Also adds total-pre/total-post checks measuring the entire out-dir, so anything landing anywhere in the shipped folder trips the guard, not just the named sub-paths.
Move the post-phase check + inject_ui_prebundle out of e2e_test_ui_prebundle into a new check_ui_prebundle_size job - build validation, not a test, so it shouldn't live in the e2e job. Persists the already-injected bundle so e2e_test_ui_prebundle attaches it directly instead of redoing the injection.
It builds the bundle, it doesn't set anything up. Updates the job key, every requires: reference, and current-state docs; historical findings log entries keep the old name since they describe what was true then.
React version compatibility check, process.cwd()-independent resolve root, correct nested node_modules handling, per-package version- collision handling, dependency-aware router-context safety, and no longer leaking the generated entry file's build paths into the shipped artifact - all in ui-vendor-dll.ts. Also: fail (not warn) on an unresolvable harmony runtime dep and vendor a missing browser-dist dependency in generate-shim-packages.ts; fail the size guard on a silently-missing artifact instead of reading it as 0 bytes; wire the size guard into both bvm publish CI jobs; fix a lint-staged glob that silently excluded root-level JS files.
## Proposed Changes The component-peer dependency e2e test gives `comp1` a custom TypeScript env but leaves `comp2` on the empty default env. With root components enabled, `comp1` references `comp2` as a TypeScript project even though the peer has no compiler to generate its tsconfig. Assign the same custom env to `comp2` so both projects have a compiler. This fixes the test fixture while preserving its peer-dependency and hidden-peer assertions; it does not change the external compiler's handling of non-TypeScript dependencies. Validation: `npm run lint` and `git diff --check` pass. Focused e2e runs were attempted under Node 22 with the local compiler reference-pruning patch disabled, but both original and corrected setups hit 18 unrelated local type errors mixing checkout types with an installed Bit version. A clean CI e2e result is still needed.
The previous commit's import { tmpdir } from 'os' broke the real
browser UI/preview pre-bundle build: ui-vendor-dll.ts is reachable
from a browser compilation via the @teambit/ui barrel, and os has no
browser resolve fallback the way fs/path do. Write the scratch entry
file under dirname(outputPath) instead - no new Node builtin needed,
and it sits outside the artifact glob rather than merely being
cleaned up in time.
Also: buildUiVendorDll's new sourceRoot default broke two existing
unit tests under real capsule execution - they used lodash.compact/
lodash.flatten as fixtures, which aren't actual dependencies of
@teambit/ui and only resolved by cwd happenstance. Switched to
p-map-series/chalk, real declared dependencies guaranteed present
wherever the component's own install is.
Also updates bundle-size-baseline.json's shims-dist check for the
expected size growth from vendoring @teambit/base-react.navigation.link.
p-map-series/chalk still failed to resolve via buildUiVendorDll's new sourceRoot default inside bit_pr's real build capsule, even though p-map-series is a genuine dependency of @teambit/ui. Revert to the original lodash.compact/lodash.flatten fixtures, explicitly resolved off process.cwd() (confirmed via CI history that this exact pairing passed before any of this sourceRoot work) - real BundleUiTask usage never passes a sourceRoot, so it's unaffected and still gets the correct default. Also replaces the resolveContextProviderMismatchUnsafePackages negative-case test's use of the live @teambit/preview component (whose dependency graph disagreed between CI and local) with a fully controlled fake package.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bundles the CLI into a single 59 MB CJS file plus 10 externals that cannot be inlined, with a generated shim package per core aspect re-exporting its slice of the bundle.
npm run bundlebuilds it;--seaalso builds a node single executable.1.2 GB / 141k files → 159 MB / ~2.8k files.
bit --help~0.53s warm.The shims emit the same
dist/*.aspect.jsanddist/*.main.runtime.jsfilenames the aspect loader already discovers, so the runtime needed no changes. The one source fix is real and independent of bundling:hook-requirepatchedmodule.constructor.prototype.require, which under any bundler installs an enumerablerequireonObject.prototype.bit startnow works too: it serves the pre-built UI/preview bundles instead of running a bundler at all (shouldServeBundleUi/writePreviewEntryhash-match and serve from the shippedartifacts/, nopublic/written). Verified end to end from a freshbit init+bit createworkspace: UI shell, workspace/scope roots, and the component's own preview all served from the pre-bundle.@rspack/dev-serverand@rspack/coreitself (42 MB — was the single biggest external) are both fully excluded from the default build'spackage.json/node_modules, not just externalized — neither is reachable oncebit startserves the pre-bundle instead of rebuilding.Also adds a "UI vendor DLL" artifact, built alongside the existing UI/preview pre-bundle:
BundleUiTaskruns an additional rspackDllPluginpass producingvendor.js+ a portablevendor-manifest.json, covering React and every core-aspect package that ships browser (.ui.runtime/.preview.runtime) code. A newcreateUiVendorDllReference()lets a separate consuming project turn that manifest into realDllReferencePluginoptions, so it can intercept already-compiled core UI code instead of recompiling it from source — fixing the gap where a bundledbitcan't build a third-party UI root (e.g. an app likecommunity-cloud). Design doc:bundle-plan/26-ui-vendor-dll-design.md; plan + decision log:bundle-plan/27-ui-vendor-dll-plan.md.Also adds a CircleCI guard (
scripts/bundle-size-guard.mjs+scripts/bundle-size-baseline.json) that fails the build if any part of the bundle grows more than 10% past its baseline — a direct response to an SSR bundle regression (6 MB → 53 MB) caught and fixed on this branch by externalizing@rspack/coreinrspack.ssr.config.ts. Runs in two phases so the cheaper check fails fast:pre(no UI/preview build needed) inbuild_esbuild_bundle, right after the bundle is built and before the e2e fan-out starts;postine2e_test_ui_prebundle, right afterinject_ui_prebundle. Every check also has a whole-folder catch-all alongside the twelve named sub-paths.--update-baselinere-measures and rewrites the baseline file, the way to accept an intentional size increase later. Baseline is calibrated against a real CircleCI (Linux) run, not a local machine (native binaries for packages likeesbuild/@swc/corediffer by platform).Current size breakdown (measured on a real CircleCI Linux run):
bit.app.js(the CLI bundle)@teambit/*+ vendored harmony deps), before UI/preview injecteddist/barrels 0.12 MB + browser barrels (browser/, real compiled dist per shim, for third-party bundlers) 7.32 MB + vendored harmony runtime deps/locators/types the restartifacts/), injected after@rspack/coreexternalize fix) + UI vendor DLL (vendor.js+ manifest) 7.38 MB + preview 0.68 MBdist/core-aspects(bundle + all shims, before → after injection)node_modules)The UI pre-bundle shrink from the original ~82.7 MB two-build layout down to the single shared compilation comes from upstream #10628 (SSR fix + minifier) and #10629 (single rspack compilation shared by both UI roots instead of two separate builds), both merged into this branch, plus #10631's
bit startsanity e2e. Note the UI artifact no longer splits into separateui-bundle/workspace/ui-bundle/scopedirectories — it's one sharedui-bundle/public/bit/tree (withssr/andstatic/under it) plus the siblingui-vendor-dll/.esbuild's own
metafile.json(8.9 MB of build-analysis JSON, never read at runtime) is no longer written into the published package — still produced for localnpm run bundleiteration and CI's diagnostic capture.Producing the UI/preview pre-bundle locally needs a real
bit build --tasks BundleUI,PreBundlePreview; it's now cached under a gitignored.bundle-cache/(with a commit-hash + datemeta.json) sonode_moduleswipes don't force re-deriving it every time.Verified from an isolated dir: 40+ commands including
create,status,tag,export,import,watch,server,start, andbuild --unmodified(all 9 tasks, rspack included). UI vendor DLL additionally verified against a real rebuilt CLI bundle and against a genuinely separatepnpm installwith different peer dependencies, confirming realDllReferencePlugininterception (not just a clean build).npm run e2e-test:bundle/:searun the suite against the artifact; CircleCI builds it once inbuild_esbuild_bundleand shares it across the e2e nodes (gated to^bit-bundle.*branches).Full architecture, measurements, externals breakdown, script-vs-SEA analysis, the publishable package layout and open questions are in
bundle-plan.md.Draft: based on
remove-core-envs-from-manifest, so the diff includes that branch. Opened to get CircleCI running.ui-vendor-dll(#10690) has been fast-forward merged into this branch and closed.