perf(seed,import): route bulk writes through the engine's batch insert path#2680
Conversation
…t path Seed loader and data-import wrote records one at a time even though the engine's array-form insert() already does the efficient thing (one driver.bulkCreate round-trip + parent-deduplicated summary recompute). Add a shared bulkWrite/withTransientRetry helper (@objectstack/core) that batches writes, retries transient driver errors with backoff, and degrades to per-row writes when a batch fails for a non-transient reason so one bad row can't drop the rest. Wire it into SeedLoaderService.loadDataset() and import-runner's runImport(), and fix Protocol.createManyData to forward context to engine.insert() like createData already does. Fixes #2678.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
documentation
📓 Docs Drift CheckThis PR changes 6 package(s): 37 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
…) does SqlDriver.bulkCreate() never ran the id/_id normalization create() applies, so a row inserted via the array path without a driver-native id default landed with id: NULL. This bulk path was rarely exercised before #2678 routed real seed/import data through it at scale, surfacing the gap (the showcase invoice-line dogfood test failed: product/invoice references resolved to NULL because the referenced object's bulk-inserted rows had no id to record). bulkCreate() now assigns nanoid(DEFAULT_ID_LENGTH) per row exactly like create() does.
|
Investigated the CI failures from the first push: Real regression (fixed, pushed in 6e3f21d): Unrelated, pre-existing (not fixed here): CodeQL flagged 2 high-severity "polynomial regex" alerts in Continuing to watch CI. Generated by Claude Code |
2 participants
Summary
Fixes #2678.
The seed loader (
defineDatasetreplay) and the data-import runner both wrote records one at a time through the single-record engine path, even though ObjectQL's engine already has an efficient batched branch (driver.bulkCreate+ parent-deduplicated summary recompute) for the array form ofinsert(). Over a remote DB this meant N round-trips + N summary recomputes instead of ~N/batch, and a transient network error on any one row was silently dropped with no retry (the 2026-07-06 HotCRM first-boot incident).bulkWrite/withTransientRetry(packages/core/src/utils/bulk-write.ts): batches rows through a caller-supplied batch-write function, retries a whole-batch transient failure (network blip / timeout) with exponential backoff, and degrades to per-row writes (each itself retried) when a batch fails for a non-transient reason — so one bad row can't drop the other N-1.SeedLoaderService.loadDataset()(packages/metadata-protocol/src/seed-loader.ts) buffers insert-mode records and flushes them in batches of 200 via the engine's array-forminsert(). Datasets with a self-referencing field (e.g.employee.manager_id -> employee) keep the historical strictly-sequential per-record path, since a later record may need an earlier one's freshly-assigned id viainsertedRecords— batching would defer that write past the point it's needed. Update-mode writes are wrapped in the same transient retry.runImport()(packages/rest/src/import-runner.ts) buffers create-resolved rows and flushes them viaprotocol.createManyData()when the protocol supports it (falling back to the original per-rowcreateData()call otherwise). Per-row result ordering is preserved via an index-addressed results array even though creates are now written out of their original per-row sequence.Protocol.createManyData(packages/metadata-protocol/src/protocol.ts) now forwardscontexttoengine.insert(), matchingcreateData— previously it silently dropped tenant/security context, which is presumably why the issue called this endpoint "may be unwired".Test plan
bulkWrite/withTransientRetry/defaultIsTransientError(packages/core/src/utils/bulk-write.test.ts): bulk-path write-count assertion, transient-retry, logical-error per-row degradation, index correlation across batches.runImport()'s batching (packages/rest/src/import-runner-bulk.test.ts): batch write-count, transient retry, logical degrade, no-createManyDatafallback, result ordering with interleaved updates.packages/runtime/src/seed-loader.test.tsandpackages/runtime/src/http-dispatcher.test.tsthat asserted the old N-calls-per-record behavior, now asserting the batched call.pnpm test(via turbo) green across@objectstack/core,@objectstack/metadata-protocol,@objectstack/objectql,@objectstack/rest,@objectstack/runtime— 461+797+216+15+... tests passing, no regressions.pnpm lintclean on all changed files.Notes / follow-ups
findExisting/reference-resolution reads are unchanged (still per-row) — the issue's evidence and acceptance criteria are scoped to the write path.🤖 Generated with Claude Code
Generated by Claude Code