feat: packagist transitive dependent counts

[epipav]

No description provided.

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cursor]

PR Summary

Medium Risk
Bulk derived updates on all Packagist packages and a weekly seq scan of package_dependencies (~1.5B rows) can load production DB; guards limit bad merges, and criticality impact is additive-only per ADR.

Overview
Adds a Packagist transitive dependents pipeline so packages.transitive_dependent_count can be filled from stored Composer edges (deps.dev has no Packagist coverage).

Data & DAL: New packagist_transitive_runs ledger and SQL in transitiveDependents to snapshot direct edges into staging, compute the reverse closure (depth ≥ 2, deps.dev semantics), and keyset-merge into packages with zero-fill and IS DISTINCT FROM churn control. Empty snapshots or empty staging counts fail loudly instead of overwriting good data.

Temporal: computePackagistTransitiveDependents runs prepare → batched merge → finish with continueAsNew; metadata drain chain-starts it on completion. Monday backstop cron starts the same fixed workflow id only when no recent successful run and metadata drain is not running. Manual pnpm trigger-packagist:local transitive uses that workflow id to avoid overlapping drains.

Other: findPendingJobByKind reused by rankPackages for pending ingest job adoption; ADR-0009 and packagist README document the fifth lane; unit/integration tests cover workflows, activities, and ledger lifecycle.

^{Reviewed by Cursor Bugbot for commit 3f436db. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Adds weekly Packagist transitive-dependent computation and integrates it with the packages worker.

Changes:

• Adds PostgreSQL snapshot, closure, merge, and run-ledger DAL operations.
• Adds Temporal workflow/activity orchestration, manual triggering, and metadata-drain chaining.
• Adds migrations, tests, ADR updates, and operational documentation.

Review notes: The PR has five unresolved findings. Its title also lacks the required JIRA key, and the diff exceeds the recommended 1,000-line target.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/tsconfig.json	Enables BigInt test syntax.
services/libs/data-access-layer/src/packages/transitiveDependents.ts	Implements snapshot, closure, and merge queries.
services/libs/data-access-layer/src/packages/transitiveDependents.integration.test.ts	Tests graph and ledger behavior.
services/libs/data-access-layer/src/packages/packagistTransitiveRuns.ts	Adds run-ledger operations.
services/libs/data-access-layer/src/packages/index.ts	Exports new DAL modules.
services/libs/data-access-layer/src/osspckgs/ingestJobs.ts	Adds pending-job lookup.
services/apps/packages_worker/src/workflows/index.ts	Exports the workflow.
services/apps/packages_worker/src/scripts/triggerPackagistSeed.ts	Adds manual transitive trigger.
services/apps/packages_worker/src/packagist/workflows.ts	Orchestrates preparation and merge draining.
services/apps/packages_worker/src/packagist/README.md	Documents the new lane.
services/apps/packages_worker/src/packagist/activities.ts	Implements Temporal activities.
services/apps/packages_worker/src/packagist/__tests__/wiring.test.ts	Verifies worker exports.
services/apps/packages_worker/src/packagist/__tests__/transitiveDependents.test.ts	Tests workflow orchestration.
services/apps/packages_worker/src/criticality/activities.ts	Reuses pending-job lookup.
services/apps/packages_worker/src/activities.ts	Registers new activities.
docs/adr/0009-packagist-worker-design-decisions.md	Records the architecture decision.
backend/src/osspckgs/migrations/V1785740540__packagist_transitive_runs.sql	Creates the run ledger.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

services/libs/data-access-layer/src/osspckgs/ingestJobs.ts:80

• This coerces a PostgreSQL bigserial identifier to a JavaScript number even though the packages-db connection intentionally leaves int8 values as strings. Once an ID exceeds Number.MAX_SAFE_INTEGER, the rounded value can update or query the wrong job. Keep the ID as a string and align createIngestJob, markJobStatus, and their callers with that representation.

  // id is bigserial (pg returns int8 as a string) — convert so the declared type is true.
  return row ? Number(row.id) : null

services/apps/packages_worker/src/packagist/workflows.ts:70

• batch.changed is not retry-stable. If the merge activity commits its UPDATE but Temporal loses the completion, the retry processes the same cursor after the rows already match and returns changed: 0; this accumulator then permanently under-reports changed_rows. Persist per-batch/cumulative change counts atomically with the merge (keyed by run/cursor), or otherwise make the returned count deterministic across activity retries.

      const batch = await acts.mergePackagistTransitiveBatch(cursor, TRANSITIVE_MERGE_BATCH)
      processed += batch.processed
      changed += batch.changed

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (3)

services/libs/data-access-layer/src/packages/transitiveDependents.ts:23

• statement_timeout applies to each statement, not the whole prepare activity. This helper runs CTAS, index creation, and ANALYZE twice sequentially, so a valid attempt can exceed the 45-minute start-to-close timeout while its transaction remains active. Temporal may then retry the full scan concurrently and contend on these global staging tables. Split the phases into separately timed activities or enforce an end-to-end database deadline that terminates the original attempt before Temporal retries it.

    await tx.result(`SET LOCAL statement_timeout = '40min'`)

docs/adr/0009-packagist-worker-design-decisions.md:191

• This edit leaves ADR-0009 outside the repository's mandatory ADR structure: it has ## Decisions rather than ## Decision and lacks ## Alternatives Considered plus the required ## Consequences subsections. Please bring the ADR into compliance with .claude/rules/adr-format.md:18-29 while updating it.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

services/apps/packages_worker/src/packagist/activities.ts:520

• This marks the ledger row failed after every failed activity attempt. Since Temporal retries prepare up to three times and findUnfinishedPackagistTransitiveRun excludes failed rows, the next attempt creates a new row; one workflow run can therefore produce several failed rows before succeeding, defeating the documented retry reuse and “one row per run” lifecycle. Keep the row unfinished for retryable, non-final attempts and mark it failed only for a non-retryable error or after retries are exhausted.

  } catch (err) {
    await failRunInLedger(qx, runId, (err as Error).message)
    throw err

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Suppressed comments (2)

services/libs/data-access-layer/tsconfig.json:9

• Raising the shared DAL target to ES2020 for one test removes the ES2017 syntax check inherited from services/base.tsconfig.json:3, even though this comment says production sources must remain ES2017-compatible. Keep the library target at ES2017 and isolate or rewrite the test's BigInt literal instead of weakening checks for every DAL source file.

  // target/lib raised over base's es2017 for the BigInt literals in the packagist
  // transitive integration test (include pulls *.test.ts into tsc-check). Consumers
  // compile DAL sources under their own (es2017) configs, so src itself must stay
  // free of post-es2017 syntax.
  "compilerOptions": {
    "target": "es2020",
    "lib": ["es2020", "ES2021.String"]

docs/adr/0009-packagist-worker-design-decisions.md:191

• This adds another decision as a ### subsection, but .claude/rules/adr-format.md:18-29 requires edited ADRs to use the mandatory ## Decision, ## Alternatives Considered, and ## Consequences structure. This ADR still lacks those required sections and should be restructured before merging.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

[epipav]

@cursor review

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Suppressed comments (3)

services/libs/data-access-layer/src/osspckgs/ingestJobs.ts:80

• osspckgs_ingest_jobs.id is bigserial, and the packages DB intentionally leaves int8 values as strings (services/libs/database/src/connection.ts:81-85). Converting it with Number loses identity once IDs exceed Number.MAX_SAFE_INTEGER, potentially updating the wrong job. Keep the ID as a string and align the existing create/mark job signatures and callers accordingly.

  return row ? Number(row.id) : null

docs/adr/0009-packagist-worker-design-decisions.md:191

• The repository ADR rule (.claude/rules/adr-format.md:18-28) requires edited numbered ADRs to contain ## Decision, ## Alternatives Considered, and ## Consequences in order. This adds another ### entry under ## Decisions while those mandatory sections remain absent, so the ADR still violates the enforced format. Restructure the living ADR or record this as a conforming separate ADR.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

services/libs/data-access-layer/src/packages/transitiveDependents.integration.test.ts:73

• This cleanup casts bigserial job IDs to int[], so it will fail with integer out of range once the sequence exceeds the 32-bit range. Cast to bigint[] to match the table schema.

        await qx.result(`DELETE FROM osspckgs_ingest_jobs WHERE id = ANY($(jobIds)::int[])`, {

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

services/libs/data-access-layer/src/packages/transitiveDependents.ts:103

• The emptiness guard and zero-fill update run as separate statements. A PostgreSQL crash can truncate this UNLOGGED table after the guard succeeds but before the update reconnects, so the safety check can still allow real counts to be wiped; run both operations in one transaction or statement.

  const guard = await qx.selectOne(
    `SELECT EXISTS (SELECT 1 FROM staging.packagist_transitive_counts) AS populated`,
  )
  if (!guard.populated) {
    throw new EmptyPackagistTransitiveCountsError()

services/libs/data-access-layer/src/osspckgs/ingestJobs.ts:80

• id is bigserial, and this connection leaves int8 values as strings; converting to number can silently round valid IDs above Number.MAX_SAFE_INTEGER, causing later status updates to target the wrong row. Keep the ID as a string and align the ingest-job APIs accordingly.

  // id is bigserial (pg returns int8 as a string) — convert so the declared type is true.
  return row ? Number(row.id) : null

docs/adr/0009-packagist-worker-design-decisions.md:191

• Editing this ADR must preserve the mandatory structure in .claude/rules/adr-format.md:18-28, but ADR-0009 still lacks ## Decision, ## Alternatives Considered, and ## Consequences with the required subsections. Restructure the living decisions into that template as part of this edit.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

services/libs/data-access-layer/src/packages/transitiveDependents.ts:34

• This timeout is per statement, but each builder runs CTAS, index creation, and ANALYZE, and prepare invokes two builders. The total can exceed the 90-minute activity deadline, causing Temporal to retry while prior SQL still runs; split the phases or enforce a total budget below the activity timeout.

    // Temporal timeouts don't kill in-flight SQL — this bounds each statement so a hung
    // CTAS dies well inside the 90-min activity deadline (2 statements + slack).
    await tx.result(`SET LOCAL statement_timeout = '40min'`)

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Suppressed comments (1)

docs/adr/0009-packagist-worker-design-decisions.md:191

• This edit leaves ADR-0009 outside the required ADR structure. .claude/rules/adr-format.md:18-28 requires ## Decision, ## Alternatives Considered, and ## Consequences (with Positive/Negative/Risks) whenever an ADR is edited; this file still has only ## Decisions and no required alternatives/consequences sections. Please restructure the ADR before merging.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Suppressed comments (2)

services/libs/data-access-layer/src/packages/transitiveDependents.ts:103

• The emptiness guard and the zero-filling UPDATE run as separate statements. If PostgreSQL restarts after this check succeeds but before the update starts, crash recovery truncates the UNLOGGED counts table and the second statement can zero-fill a batch—the exact failure this guard is intended to prevent. Run the guard and update inside one qx.tx; then a restart aborts the transaction and the activity retry observes the empty table.

  const guard = await qx.selectOne(
    `SELECT EXISTS (SELECT 1 FROM staging.packagist_transitive_counts) AS populated`,
  )
  if (!guard.populated) {
    throw new EmptyPackagistTransitiveCountsError()

docs/adr/0009-packagist-worker-design-decisions.md:191

• This added decision continues the living ## Decisions structure, but the repository ADR rule requires every edited numbered ADR to contain top-level ## Decision, ## Alternatives Considered, and ## Consequences sections in order (.claude/rules/adr-format.md:18-28). Either bring ADR-0009 into that structure or explicitly exempt the living-ADR format in the rule.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Suppressed comments (2)

services/libs/data-access-layer/src/osspckgs/ingestJobs.ts:80

• Converting this bigserial ID to number can silently lose precision once it exceeds Number.MAX_SAFE_INTEGER, causing later status updates to target the wrong job. The packages DB intentionally leaves PostgreSQL int8 values as strings (services/libs/database/src/connection.ts:81-85); keep job IDs as strings and update the related DAL signatures/callers consistently instead of normalizing with Number.

  // id is bigserial (pg returns int8 as a string) — convert so the declared type is true.
  return row ? Number(row.id) : null

services/apps/packages_worker/src/packagist/workflows.ts:77

• changed is not retry-stable under Temporal's at-least-once activity execution. If the batch UPDATE commits but its activity completion is lost, the retry processes the same batch, reports changed = 0 because IS DISTINCT FROM now filters every row, and this accumulator permanently under-reports changed_rows. Persist each batch's result/idempotency key with the run, or otherwise make the returned count replay-stable before treating it as an exact run total.

      changed += batch.changed

[epipav]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c20e6e3. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Suppressed comments (5)

services/libs/data-access-layer/src/osspckgs/ingestJobs.ts:80

• Converting this bigserial identifier to number can lose precision once it exceeds Number.MAX_SAFE_INTEGER, so later status updates can target the wrong or no job. The packages DB intentionally leaves int8 values as strings (services/libs/database/src/connection.ts:81-85); preserve that representation through this helper and the job-ID APIs.

  return row ? Number(row.id) : null

services/libs/data-access-layer/src/packages/transitiveDependents.ts:34

• A 40-minute limit per statement does not keep this transaction within the 90-minute activity deadline: CTAS, index creation, and ANALYZE can each consume that limit, after which Temporal may time out and start an overlapping retry while SQL continues. Use per-phase limits whose worst-case total stays below 90 minutes, or enforce one overall database-side deadline.

    await tx.result(`SET LOCAL statement_timeout = '40min'`)

services/apps/packages_worker/src/scripts/triggerPackagistSeed.ts:79

• The backstop only checks the fixed packagist-metadata-drain ID, but manual metadata runs receive unique IDs here. The backstop can therefore start the closure during a manual metadata crawl; that crawl's completion then cannot start a fresh closure, leaving counts based on the mid-crawl snapshot. Use the fixed metadata ID for manual metadata runs as well.

  const workflowId =
    target === 'transitive' ? 'packagist-transitive-drain' : `packagist-${target}-manual-${now}`

services/apps/packages_worker/src/packagist/activities.ts:567

• This comment describes terminal failure marking, but it is attached to packagistTransitiveRanRecently, so it documents the wrong function contract.

// Terminal failure marking for the merge phase, called from the workflow's catch so a
// permanently failed drain reads 'failed' instead of sitting in 'merging' forever.

docs/adr/0009-packagist-worker-design-decisions.md:191

• This added decision continues the free-form layout instead of the mandatory ADR structure (## Decision, ## Alternatives Considered, and structured ## Consequences) required by .claude/rules/adr-format.md:16-28. Since this PR edits the ADR, bring it into the current template and include at least one alternative with pros, cons, and why it was rejected.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Suppressed comments (4)

services/libs/data-access-layer/src/packages/transitiveDependents.ts:32

• This timeout is applied per SQL statement, but each of the two rebuilds executes DROP, CTAS, CREATE INDEX, and ANALYZE. Their cumulative runtime can exceed the activity's 90-minute start-to-close timeout; because Temporal does not cancel the in-flight SQL, a retry can overlap destructive staging-table work and repeat the 1.5B-row scan. Bound the aggregate database work below the activity timeout, or raise that timeout to the true worst case and add database-level serialization/cancellation.

    // Temporal timeouts don't kill in-flight SQL; this bounds each statement so a hung
    // CTAS dies well inside the 90-min activity deadline (2 statements + slack).
    await tx.result(`SET LOCAL statement_timeout = '40min'`)

services/apps/packages_worker/src/scripts/triggerPackagistSeed.ts:77

• Manual metadata runs still receive unique workflow IDs, while the backstop only checks the fixed packagist-metadata-drain ID (packagist/activities.ts:577). A Monday backstop can therefore start the closure while a manually triggered metadata crawl is still changing dependency edges. Reuse the fixed metadata-drain ID here as well so the backstop and Temporal's duplicate-start protection cover manual crawls.

  const workflowId =
    target === 'transitive' ? 'packagist-transitive-drain' : `packagist-${target}-manual-${now}`

services/apps/packages_worker/src/packagist/activities.ts:567

• This comment documents terminal failure marking, but it is attached to the recent-success backstop query. It misstates the function's behavior and leaves the actual failure-marking function below undocumented.

// Terminal failure marking for the merge phase, called from the workflow's catch so a
// permanently failed drain reads 'failed' instead of sitting in 'merging' forever.
export async function packagistTransitiveRanRecently(withinDays: number): Promise<boolean> {

docs/adr/0009-packagist-worker-design-decisions.md:193

• This numbered ADR is being edited without the required ## Decision, ## Alternatives Considered, and ## Consequences section sequence. .claude/rules/adr-format.md:18-28 requires those sections for edited files under docs/adr/; please bring the ADR into the prescribed structure while adding this decision.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

A fifth lane, `computePackagistTransitiveDependents`, populates

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Suppressed comments (3)

services/apps/packages_worker/src/packagist/workflows.ts:155

• This guard only checks the fixed packagist-metadata-drain execution, but the manual command added in this PR starts metadata as packagist-metadata-manual-${now}. While that manual crawl is updating dependency edges, the backstop can therefore see “not running” and start the closure against a changing dataset. Coordinate all metadata runs through a shared lock/discoverable identifier before starting the transitive drain.

  if (await acts.packagistMetadataDrainRunning()) return

docs/adr/0009-packagist-worker-design-decisions.md:191

• This new decision does not follow the repository ADR template: .claude/rules/adr-format.md:18-29 requires exact ## Decision, ## Alternatives Considered, and ## Consequences sections, including Positive/Negative/Risks. A ### entry under ## Decisions plus bold inline consequence labels does not satisfy those required headings; restructure the ADR or formally exempt this living-ADR format.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

services/libs/data-access-layer/src/packages/transitiveDependents.integration.test.ts:64

• osspckgs_ingest_jobs.id is bigserial, but this cleanup casts the collected IDs to int[]. Once the sequence exceeds the int4 range, the destructive integration test will fail cleanup and can leave its job fixtures behind. Match the column type with bigint[].

        await qx.result(`DELETE FROM osspckgs_ingest_jobs WHERE id = ANY($(jobIds)::int[])`, {
          jobIds,