Skip to content

development.md

Latest commit

 

History

History
177 lines (141 loc) · 9.46 KB

development.md

File metadata and controls

177 lines (141 loc) · 9.46 KB
Raw

Development

Build and verify

Use the versions pinned in package.json and Cargo.lock, a stable Rust toolchain, and Bun for runtime compatibility tests.

pnpm install --frozen-lockfile
pnpm build
pnpm check
pnpm pack:check

check runs oxfmt, cargo fmt, oxlint, Clippy with warnings denied, strict TypeScript checks, Rust tests, and the same compiled integration tests under Node and Bun. pack:check installs a local tarball offline, checks its file allowlist, and runs Node, Bun, and TypeScript consumer fixtures against the installed package.

Use pnpm format before reviewing changes. The native loader and declarations are generated by napi-rs; regenerate them with pnpm build:native instead of editing or formatting them manually. The repository's AGENTS.md records development conventions.

Implementation

Area Responsibility
src/source.rs Calamine readers and shared, immutable input ownership
src/reader.rs, src/executor.rs Input snapshots and bounded native execution
src/cancellation.rs, src/stream.rs Cancellation and stream execution permits
src/workbook.rs, src/sheet.rs, src/vba.rs Resource ownership and upstream operations
src/cell.rs Cell values, dates, errors, and large integers
lib/reader.ts, lib/read.ts Input validation, sheet selection, and complete reads
lib/workbook.ts, lib/options.ts Handle lifetime, iteration, and read options
lib/native.ts, lib/stream.ts AbortSignal adaptation and temporary-file spooling
lib/types.ts Public API types
test/ Runtime tests and attributed fixtures
scripts/ Packaging checks, benchmarks, fixture and license generation

read accepts ReadInput and ReadOptions, returning ReadResult. Opening a file, Buffer, or stream returns WorkbookHandle, which must be closed. Complete reads and handle reads share SheetResult; batch iteration adds an offset in RowBatch. Keep those paths on the same parser and conversion implementation.

Native execution and ownership

napi-rs owns the shared Tokio runtime. A reader adds one semaphore, with executing concurrency defaulting to the runtime's actual worker count. Extra work waits asynchronously; there is no queue-length quota. An explicit reader limit changes its semaphore without creating another runtime or resizing the shared pool.

AsyncBlockBuilder preserves a synchronous entry phase for byte snapshots, then connects a Rust future to a JS Promise. Byte views are borrowed only during that entry and copied once into Arc<[u8]>. Calamine's format-detection attempts clone an Arc-backed cursor instead of copying the entire input. Paths open on a blocking worker without a preliminary JS file read.

After obtaining a permit, parsing, batch preparation, and explicit cleanup use spawn_blocking. They run outside Tokio's async workers and libuv's shared pool. JS arrays and strings are still constructed on the JS thread in bounded batches. Complete reads collect those batches without a JSON serialization round trip.

A stream obtains a permit before JS pulls input. The adapter writes each chunk to a private temporary file before pulling the next. It transfers the same permit to native parsing after download. Storage SDKs, authentication, and networking stay outside the package.

Cancellation uses an owned watch channel. The JS adapter removes listeners and preserves the caller's abort reason. Queued tasks can exit promptly; a running Calamine call retains its permit until it finishes. Cleanup closes newly returned resources before reporting cancellation. A workbook also owns its active sheet, so closing it releases a paused iterator's range. Cleanup itself cannot be cancelled.

Concurrency bounds executing operations, not all resident inputs or open handles. Queued Buffers retain snapshots, and complete results remain live while the caller retains them. Batch output does not make Calamine's worksheet allocation streaming. See the README for limits and cancellation semantics.

Maintenance scripts

  • pnpm benchmark --input <file> measures complete reads. See performance for SheetJS comparisons and reproduction commands.
  • node scripts/generate-benchmarks.mjs <SheetJS module> <directory> creates synthetic numeric and string workloads outside the repository.
  • node scripts/generate-fixtures.mjs <SheetJS module> regenerates the synthetic test fixtures. Review changes against test/fixtures/README.md; preserve the separately licensed upstream fixtures.
  • pnpm notices regenerates THIRD_PARTY_LICENSES.txt from the locked Rust dependency graph. Review it whenever native dependencies change.

SheetJS is an optional external benchmark/fixture tool, not a package dependency. Do not run performance tests against production files or services. Keep local measurements in docs/performance.json with methodology in docs/performance.md; remove superseded experiment logs instead of accumulating contradictory reports.

Packaging and release

The package uses Node-API 8 and TypeScript ESM output. CommonJS consumption is not part of the current compatibility contract. Source files accompany source and declaration maps so installed-package debugging and editor navigation work. The npm allowlist excludes tests, scripts, agent instructions, and raw benchmark data.

Routine releases

Work on main using Conventional Commits (feat:, fix:, docs:, ci:, and other appropriate types). Release Please maintains a version PR with the title chore: release vX.Y.Z, updating package.json, Cargo.toml, Cargo.lock, and CHANGELOG.md. No separate long-lived release branch is needed. The generated changelog retains Release Please's formatting and is excluded from Oxfmt.

Merge that PR when ready to release. Automation creates a version tag and draft GitHub Release, then starts publish.yml. The workflow:

  1. Verifies the tag and package versions match.
  2. Runs the complete platform CI on the tagged commit.
  3. Packages all eight native targets with the official napi-rs tooling, then checks the root and Linux platform tarballs with Node, Bun, and TypeScript.
  4. Publishes the eight platform packages before the root package using npm OIDC.
  5. Attaches the tarballs and their integrity manifest, then makes the GitHub Release public.

Release Please explicitly dispatches checks and publication with GITHUB_TOKEN; no personal access token is required. npm publication uses id-token: write and the npm GitHub environment, without a stored npm token. Public OIDC releases receive npm provenance automatically.

For a packaging rehearsal, run Actions → Publish → Run workflow, select the version tag, and choose prepare. This builds and tests everything but only uploads the npm-release artifact. It does not publish npm or the GitHub Release.

If publication fails partway through, use Re-run failed jobs. The publisher checks existing versions against the original tarball integrity and skips exact matches. Keep the original artifacts: rebuilding after partial publication can produce different bytes for an immutable npm version. Release artifacts are retained for 14 days and attached permanently when publication completes.

First publication

npm requires a package to exist before configuring its trusted publisher. The first release therefore needs a one-time authenticated publication of the real, tested packages.

Create the initial version tag and draft release, run Publish in prepare mode, then download its npm-release artifact into release/. From a checkout of that tag, with npm logged in to the publishing account:

node scripts/publish-release.mjs release
node scripts/publish-release.mjs release --publish

The first command is a dry run. The second uploads the existing tarballs; it does not build locally and requires no Windows or macOS toolchain. Complete npm's 2FA prompts yourself. This account-authenticated bootstrap does not get the automatic GitHub OIDC provenance of subsequent releases.

Configure a trusted publisher for each of the nine packages, using the names in release/manifest.json:

npm trust github <package-name> \
  --repo Kunduin/calamine-node \
  --file publish.yml \
  --environment npm \
  --allow-publish

In npm's web interface the equivalent fields are owner Kunduin, repository calamine-node, workflow publish.yml, environment npm, with permission to publish. Permission to stage a package alone is insufficient for this workflow.

After all packages are published and trusted publishers configured, attach the original tarballs and manifest to the draft and publish the GitHub Release. Set Settings → Secrets and variables → Actions → Variables → RELEASE_AUTOMATION_ENABLED to true. Subsequent releases use the version PR and GitHub Actions flow above.