feat(cli): bundle pg-delta for database workflows

apps/cli-go/CONTRIBUTING.md

@@ -44,9 +44,48 @@ The Supabase API client is generated from OpenAPI spec. See [our guide](api/READ
 
 ## Testing local pg-delta builds
 
-To exercise unpublished `@supabase/pg-delta` changes inside CLI edge-runtime scripts (`db pull`, `db diff`, `db push`, etc.), publish a local build via Verdaccio in [pg-toolbelt](https://github.com/supabase/pg-toolbelt) and point the CLI at that registry.
-
-### 1. Start Verdaccio (pg-toolbelt)
+Pg-delta and pg-topo run in-process by default and are bundled into the CLI binary.
+Their versions are fixed by `apps/cli/package.json` and the lockfile at CLI build
+time. `PGDELTA_NPM_REGISTRY` and a test project's
+`supabase/.temp/pgdelta-version` do not affect this implementation.
+
+### Default bundled implementation
+
+To exercise unpublished [pg-toolbelt](https://github.com/supabase/pg-toolbelt)
+changes locally:
+
+1. Build the `@supabase/pg-delta` and `@supabase/pg-topo` packages in pg-toolbelt.
+2. Temporarily point both dependencies in `apps/cli/package.json` at those local
+   package directories (for a sibling checkout, use
+   `file:../../../pg-toolbelt/packages/pg-delta` and
+   `file:../../../pg-toolbelt/packages/pg-topo`).
+3. Run `pnpm install` from the CLI repository root, then run the CLI from source
+   with `pnpm --dir apps/cli dev:legacy -- db diff ...`, or rebuild the legacy
+   binary before testing it.
+
+Both packages must be updated together so local source runs and built binaries
+use the same planner/reorder implementation. Restore the package references and
+lockfile after local testing. New-engine SQL need not byte-match the legacy
+renderer; verify that the SQL executes and a subsequent operation converges to
+an empty diff.
+
+When `PGDELTA_DEBUG` is enabled, the bundled engine writes non-reusable debug
+artifacts under `supabase/.temp/pgdelta/v2/debug/<id>/`: `metadata.json` plus
+`source-snapshot.json`, `desired-snapshot.json`, `plan.json`, and
+`diagnostics.json` when those values are available. Legacy catalogs remain at
+the `supabase/.temp/pgdelta/` root and are never consumed by the bundled engine.
+For declarative `generate` and `sync`, `--no-cache` bypasses legacy catalog
+reuse/warming; the bundled engine already extracts live state and has no
+reusable catalog cache.
+
+### Legacy edge-runtime implementation
+
+To exercise unpublished legacy `@supabase/pg-delta` changes inside edge-runtime
+scripts, select `SUPABASE_USE_PG_DELTA_NEXT=false`, publish a local build via
+Verdaccio, and point the CLI at that registry. These instructions describe the
+temporary compatibility implementation only.
+
+#### 1. Start Verdaccio (pg-toolbelt)
 
 ```sh
 cd pg-toolbelt
@@ -55,7 +94,7 @@ bun run verdaccio:start
 
 Verdaccio listens on `http://localhost:4873`. `@supabase/*` packages you publish locally are served from local storage; other `@supabase/*` dependencies (for example `@supabase/pg-topo`) are proxied to npmjs.
 
-### 2. Publish a local pg-delta build
+#### 2. Publish a local pg-delta build
 
 After changing `packages/pg-delta`:
 
@@ -68,7 +107,7 @@ This publishes a fresh `0.0.0-local.<timestamp>` version and restores `package.j
 
 Re-run whenever you change pg-delta source.
 
-### 3. Run the CLI against the local registry
+#### 3. Run the CLI against the local registry
 
 Set `PGDELTA_NPM_REGISTRY` to a URL reachable **from inside the edge-runtime Docker container**:
 
@@ -79,6 +118,8 @@ export PGDELTA_NPM_REGISTRY=http://host.docker.internal:4873
 # Linux (Docker 20.10+)
 export PGDELTA_NPM_REGISTRY=http://host.docker.internal:4873
 # or: export PGDELTA_NPM_REGISTRY=http://172.17.0.1:4873
+
+export SUPABASE_USE_PG_DELTA_NEXT=false
 ```
 
 Then run any pg-delta-backed command, for example:
@@ -89,4 +130,7 @@ supabase db pull --db-url "$DATABASE_URL" --diff-engine pg-delta
 
 When set, the CLI injects a scoped `.npmrc` and forwards `NPM_CONFIG_REGISTRY` into the edge-runtime container (`PgDeltaNpmRegistryOption` in `internal/utils/pgdelta_local.go`).
 
-Unset `PGDELTA_NPM_REGISTRY` to return to the npmjs version pinned in config / `supabase/.temp/pgdelta-version`.
+Unset `PGDELTA_NPM_REGISTRY` to return to the legacy npmjs version pinned in
+config / `supabase/.temp/pgdelta-version`. Unset
+`SUPABASE_USE_PG_DELTA_NEXT` (or set it to `true`) to return to the bundled
+default implementation.

apps/cli-go/cmd/db.go

@@ -1,8 +1,12 @@
 package cmd
 
 import (
+	"bufio"
+	"context"
+	"encoding/json"
 	"errors"
 	"fmt"
+	"io"
 	"os"
 	"path"
 	"path/filepath"
@@ -30,6 +34,78 @@ import (
 	"github.com/supabase/cli/pkg/migration"
 )
 
+type pgDeltaNextShadowEndpoint struct {
+	ContainerID string `json:"containerId"`
+	URL         string `json:"url"`
+}
+
+type pgDeltaNextShadowHandoff struct {
+	Migrations  pgDeltaNextShadowEndpoint `json:"migrations"`
+	Declarative pgDeltaNextShadowEndpoint `json:"declarative"`
+}
+
+// handoffPgDeltaNextShadow transfers cleanup ownership only after the caller
+// has received and acknowledged the complete JSON description. Until then Go
+// removes both containers on every exit path, including cancellation and I/O
+// failure.
+func handoffPgDeltaNextShadow(ctx context.Context, shadow diff.PgDeltaNextShadow, in io.Reader, out io.Writer, remove func(string)) error {
+	transferred := false
+	defer func() {
+		if transferred {
+			return
+		}
+		remove(shadow.Migrations.Container)
+		remove(shadow.Declarative.Container)
+	}()
+
+	payload := pgDeltaNextShadowHandoff{
+		Migrations: pgDeltaNextShadowEndpoint{
+			ContainerID: shadow.Migrations.Container,
+			URL:         utils.ToPostgresURLWithoutPassword(shadow.Migrations.Config),
+		},
+		Declarative: pgDeltaNextShadowEndpoint{
+			ContainerID: shadow.Declarative.Container,
+			URL:         utils.ToPostgresURLWithoutPassword(shadow.Declarative.Config),
+		},
+	}
+	if err := json.NewEncoder(out).Encode(payload); err != nil {
+		return fmt.Errorf("failed to encode pg-delta shadow handoff: %w", err)
+	}
+	if flusher, ok := out.(interface{ Flush() error }); ok {
+		if err := flusher.Flush(); err != nil {
+			return fmt.Errorf("failed to flush pg-delta shadow handoff: %w", err)
+		}
+	}
+
+	type readResult struct {
+		line string
+		err  error
+	}
+	result := make(chan readResult, 1)
+	go func() {
+		line, err := bufio.NewReader(in).ReadString('\n')
+		result <- readResult{line: line, err: err}
+	}()
+
+	select {
+	case <-ctx.Done():
+		return ctx.Err()
+	case read := <-result:
+		if err := ctx.Err(); err != nil {
+			return err
+		}
+		if read.err != nil {
+			return fmt.Errorf("failed to read pg-delta shadow handoff acknowledgment: %w", read.err)
+		}
+		if read.line != "ack\n" {
+			return fmt.Errorf("unexpected pg-delta shadow handoff acknowledgment %q", read.line)
+		}
+	}
+
+	transferred = true
+	return nil
+}
+
 var (
 	dbCmd = &cobra.Command{
 		GroupID: groupLocalDev,
@@ -200,20 +276,17 @@ var (
 		},
 	}
 
-	shadowMode        string
-	shadowTargetLocal bool
-	shadowUsePgDelta  bool
-	shadowSchema      []string
-	shadowProjectRef  string
+	shadowMode       string
+	shadowSchema     []string
+	shadowProjectRef string
 
 	// dbShadowCmd is a hidden seam used by the native-TypeScript db diff/pull
-	// commands to provision the throwaway shadow database that the diff "source"
-	// runs against, then leave it running so the TS caller can run the differ
-	// (migra or pg-delta) itself and remove the container afterwards. It prints
-	// three newline-separated lines to stdout: the container id, the source
-	// Postgres URL, and an optional target-override URL (empty unless the
-	// local-target declarative branch redirects the diff target to a second
-	// shadow database). The URLs are emitted WITHOUT the password
+	// commands to provision throwaway shadow databases, then leave them running
+	// so the TS caller can run the differ itself and remove the containers
+	// afterwards. Legacy modes print two newline-separated lines. pgdelta-next
+	// emits a JSON object describing its two isolated clusters, then retains
+	// cleanup ownership until the caller acknowledges receipt. URLs are emitted
+	// WITHOUT the password
 	// (ToPostgresURLWithoutPassword) so we never log a credential to stdout
 	// (CWE-312); the TS caller re-injects the local Postgres password it already
 	// resolves from config.toml, which is the same value the shadow uses. Shadow
@@ -246,13 +319,20 @@ var (
 			if err := flags.LoadConfig(fsys); err != nil {
 				return err
 			}
+			if shadowMode == "pgdelta-next" {
+				nextShadow, err := diff.PreparePgDeltaNextShadow(cmd.Context(), fsys)
+				if err != nil {
+					return err
+				}
+				return handoffPgDeltaNextShadow(cmd.Context(), nextShadow, os.Stdin, os.Stdout, utils.DockerRemove)
+			}
 			var src diff.ShadowSource
 			var err error
 			switch shadowMode {
 			case "declarative":
 				src, err = diff.PrepareRawShadow(cmd.Context())
 			case "diff", "":
-				src, err = diff.PrepareShadowSource(cmd.Context(), shadowSchema, shadowTargetLocal, shadowUsePgDelta, fsys)
+				src, err = diff.PrepareShadowSource(cmd.Context(), fsys)
 			default:
 				return fmt.Errorf("unknown shadow mode: %s", shadowMode)
 			}
@@ -261,11 +341,6 @@ var (
 			}
 			fmt.Println(src.Container)
 			fmt.Println(utils.ToPostgresURLWithoutPassword(src.Source))
-			if src.TargetOverride != nil {
-				fmt.Println(utils.ToPostgresURLWithoutPassword(*src.TargetOverride))
-			} else {
-				fmt.Println("")
-			}
 			return nil
 		},
 	}
@@ -614,9 +689,7 @@ func init() {
 	dbCmd.AddCommand(dbPullCmd)
 	// Build hidden shadow-provisioning seam command
 	shadowFlags := dbShadowCmd.Flags()
-	shadowFlags.StringVar(&shadowMode, "mode", "diff", "Shadow mode: diff (baseline + migrations) or declarative (bare shadow).")
-	shadowFlags.BoolVar(&shadowTargetLocal, "target-local", false, "Whether the diff target is the local database (enables the declarative-schema branch).")
-	shadowFlags.BoolVar(&shadowUsePgDelta, "use-pg-delta", false, "Whether pg-delta is the active diff engine (selects the declarative-apply path).")
+	shadowFlags.StringVar(&shadowMode, "mode", "diff", "Shadow mode: diff (baseline + migrations), declarative (bare shadow), or pgdelta-next (migrated + empty scratch).")
 	shadowFlags.StringSliceVarP(&shadowSchema, "schema", "s", []string{}, "Comma separated list of schema to include.")
 	shadowFlags.StringVar(&shadowProjectRef, "project-ref", "", "Linked project ref, so the shadow merges the matching [remotes.<ref>] config override.")
 	dbCmd.AddCommand(dbShadowCmd)

apps/cli-go/cmd/db_shadow_test.go

@@ -0,0 +1,112 @@
+package cmd
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"io"
+	"strings"
+	"testing"
+
+	"github.com/jackc/pgconn"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"github.com/supabase/cli/internal/db/diff"
+)
+
+func TestHandoffPgDeltaNextShadowTransfersOwnershipAfterAck(t *testing.T) {
+	shadow := testPgDeltaNextShadow()
+	var output bytes.Buffer
+	var removed []string
+
+	err := handoffPgDeltaNextShadow(context.Background(), shadow, strings.NewReader("ack\n"), &output, func(container string) {
+		removed = append(removed, container)
+	})
+
+	require.NoError(t, err)
+	assert.Equal(t, "{\"migrations\":{\"containerId\":\"migrations-container\",\"url\":\"postgresql://postgres@migrations-host:6543/postgres?connect_timeout=10\"},\"declarative\":{\"containerId\":\"declarative-container\",\"url\":\"postgresql://postgres@declarative-host:7654/postgres?connect_timeout=10\"}}\n", output.String())
+	assert.Empty(t, removed)
+}
+
+func TestHandoffPgDeltaNextShadowRetainsOwnershipOnHandshakeFailure(t *testing.T) {
+	tests := []struct {
+		name    string
+		input   string
+		wantErr string
+	}{
+		{name: "EOF", wantErr: "failed to read"},
+		{name: "ack without newline", input: "ack", wantErr: "failed to read"},
+		{name: "bad acknowledgment", input: "nope\n", wantErr: "unexpected"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			var removed []string
+			err := handoffPgDeltaNextShadow(context.Background(), testPgDeltaNextShadow(), strings.NewReader(tt.input), io.Discard, func(container string) {
+				removed = append(removed, container)
+			})
+
+			assert.ErrorContains(t, err, tt.wantErr)
+			assert.Equal(t, []string{"migrations-container", "declarative-container"}, removed)
+		})
+	}
+}
+
+func TestHandoffPgDeltaNextShadowCleansBothOnEncodingFailure(t *testing.T) {
+	var removed []string
+	err := handoffPgDeltaNextShadow(context.Background(), testPgDeltaNextShadow(), strings.NewReader("ack\n"), failingWriter{}, func(container string) {
+		removed = append(removed, container)
+	})
+
+	assert.ErrorContains(t, err, "failed to encode")
+	assert.Equal(t, []string{"migrations-container", "declarative-container"}, removed)
+}
+
+func TestHandoffPgDeltaNextShadowCleansBothOnCancellation(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	reader, writer := io.Pipe()
+	cancel()
+	t.Cleanup(func() {
+		_ = reader.Close()
+		_ = writer.Close()
+	})
+	var removed []string
+
+	err := handoffPgDeltaNextShadow(ctx, testPgDeltaNextShadow(), reader, io.Discard, func(container string) {
+		removed = append(removed, container)
+	})
+
+	assert.ErrorIs(t, err, context.Canceled)
+	assert.Equal(t, []string{"migrations-container", "declarative-container"}, removed)
+}
+
+func testPgDeltaNextShadow() diff.PgDeltaNextShadow {
+	return diff.PgDeltaNextShadow{
+		Migrations: diff.PgDeltaNextShadowDatabase{
+			Container: "migrations-container",
+			Config: pgconn.Config{
+				Host:     "migrations-host",
+				Port:     6543,
+				User:     "postgres",
+				Password: "must-not-be-emitted",
+				Database: "postgres",
+			},
+		},
+		Declarative: diff.PgDeltaNextShadowDatabase{
+			Container: "declarative-container",
+			Config: pgconn.Config{
+				Host:     "declarative-host",
+				Port:     7654,
+				User:     "postgres",
+				Password: "must-not-be-emitted",
+				Database: "postgres",
+			},
+		},
+	}
+}
+
+type failingWriter struct{}
+
+func (failingWriter) Write([]byte) (int, error) {
+	return 0, errors.New("write failed")
+}

apps/cli-go/docs/supabase/db/diff.md

@@ -6,12 +6,18 @@ Requires the local development stack to be running when diffing against the loca
 
 Runs [djrobstep/migra](https://github.com/djrobstep/migra) in a container to compare schema differences between the target database and a shadow database. The shadow database is created by applying migrations in local `supabase/migrations` directory in a separate container. Output is written to stdout by default. For convenience, you can also save the schema diff as a new migration file by passing in `-f` flag.
 
+Normal diff mode always compares that migrations shadow with the selected live database. Declarative files under `supabase/database/` and `[db.migrations].schema_paths` do not replace the target. Use `supabase db schema declarative sync` to compare the complete declarative desired state.
+
 By default, all schemas in the target database are diffed. Use the `--schema public,extensions` flag to restrict diffing to a subset of schemas.
 
 Projects created by a recent `supabase init` default to the pg-delta diff engine (`[experimental.pgdelta] enabled = true` in `config.toml`). Existing projects are unaffected and keep using migra unless they opt in. To fall back to the legacy migra engine, set `enabled = false` under `[experimental.pgdelta]`, or pass `--use-migra` for a single run.
 
+The pg-delta engine runs in-process by default and is bundled into the CLI together with pg-topo at build time. Set `SUPABASE_USE_PG_DELTA_NEXT=false` to temporarily select the legacy edge-runtime implementation. `PGDELTA_NPM_REGISTRY`, `supabase/.temp/pgdelta-version`, and legacy catalogs under `supabase/.temp/pgdelta/` affect only that opt-out; there is no automatic fallback.
+
 With the pg-delta engine the diff SQL is formatted by default with the same settings the declarative export uses (uppercase keywords, wrapped at a max width of 180, indented and column-aligned); execution-aware transaction boundaries are preserved as per-unit header comments in the output. Configure overrides with `[experimental.pgdelta] format_options`, or set `format_options = "null"` to emit raw, unformatted statements.
 
+The bundled and legacy renderers can produce different SQL bytes or file segmentation. The compatibility contract is executable SQL and convergence: after applying the result, a subsequent diff should be empty. With `PGDELTA_DEBUG=1`, bundled-engine snapshots, plans, and diagnostics are stored under `supabase/.temp/pgdelta/v2/debug/<id>/`; those files are diagnostic artifacts, not reusable catalogs.
+
 While the diff command is able to capture most schema changes, there are cases where it is known to fail. Currently, this could happen if you schema contains:
 
 - Changes to publication