Branch resolution: declare new tables into a draft schema beside the pipeline's own

[dimitri-yatsenko]

A schema name resolves per session to the draft and the real schema

A session may be bound to a branch. When it is, dj.Schema('proj_ephys') resolves to two
physical schemas: proj_ephys, which holds the pipeline's declared tables, and
br_{branch_id}_proj_ephys, which holds what this branch adds. A table that already exists
in proj_ephys binds there and is never re-declared; a table that exists in neither is
declared into the branch schema. Off a branch, proj_ephys resolves to proj_ephys and
nothing below applies.

The Python file is byte-identical on the branch and on main after the merge — no second
schema object, no decorator to flip, no runner to delete:

# ephys_analysis.py
schema = dj.Schema('proj_ephys')

@schema
class ArtifactScore(dj.Computed):
    definition = """ -> Recording
                     -> ArtifactParams
                     ---
                     score : float """

The platform owns the lifecycle — checkout, teardown, re-declaration, the branch login and its
grants. The library owns naming, binding, and declaration.

database.branch, and the name derived from it

DatabaseSettings gains branch: str | None, default None, alias DJ_BRANCH, with an
ENV_VAR_MAPPING entry. Connection._config is per-connection (connection.py:188), so a
branch binds to a connection and one process can hold a branch session and a main session at
once.

_Schema.activate() keeps both names: self.database is what the caller typed, and
self.branch_database is br_{branch_id}_{schema} when a branch is set, None otherwise.
Connection.register() registers both, so Dependencies.load() builds one graph over both
schemas. The branch schema is created on first declaration into it, not at activate(), so
importing a module the branch never extends leaves no empty schema behind.

A branch may introduce a schema, not only extend one. Where the base schema does not exist,
creating the draft creates the base too, empty. A draft exists to be merged, and a draft whose
base can never exist can never be promoted — so the base is part of what the branch proposes.
It also keeps exists, list_tables(), make_classes and drop() answering truthfully, since
all four read the logical schema. The base stays empty until the merge: with nothing declared in
it, every table the branch declares lands in the draft under the binding rule.

The branch is read once and frozen. Activating a second schema under a different branch raises,
naming the branch the connection already holds. The refusal lands at that activation rather than
where the setting changed, because the config object holds no reference back to the connections
reading it. Two branches at once means a connection each.

The library validates only that branch yields a legal identifier within the 64-character
budget. The digit-led six-character branch_id and the br_ reservation are the platform's
rules, enforced where branches are created.

Where each table binds

_Schema._decorate_table() assigns table_class.database. On a branch it resolves first:

table exists in	binds to	declared?
the logical schema	the logical schema	no — already declared
the branch schema only	the branch schema	no — already declared
neither	the branch schema	yes

Without this rule, importing the pipeline module on a branch re-creates Subject, Session
and Recording empty in the branch schema from the branch's own definitions, every new table
hangs off empty parents, and populate does nothing.

The lookup is one query per schema at activation, listing the logical schema's table names and
caching them on the _Schema object. Off a branch it does not run.

A part binds where its master binds. A new part under an existing master therefore resolves to
the logical schema, where this release does not declare; that raises a named error rather than
an access error. A new master declared with its own parts is a different case and works —
master and parts land in the draft together, so the name-prefix invariant that ties them holds
inside it.

A branch records its references in DataJoint, not in the database

A table declared into a branch schema records every reference in DataJoint and emits no
FOREIGN KEY clause — not only the references that cross into production, all of them.
declare.py:298-301 fills fk_attribute_map before :315 emits the constraint, so
attribute inheritance, types, comments, lineage, and the
supporting index are unchanged and only the constraint is dropped. Joins and restrictions match
on lineage and are unaffected. Acyclicity is still checked, over a graph that carries the
symbolic edges. Primary keys are not foreign keys and stay real.

The edge is recorded in a hidden ~edge table in the branch schema, with the columns
Dependencies.load() already consumes: constraint_name, referencing_table,
referenced_table, column_name, referenced_column_name. On a branch, load() unions those
rows into the same fks dict the catalog query fills, and everything downstream — parents(),
key_source, Diagram, the cascade walk — works off one graph with no further change.
autopopulate.py is untouched.

A production session never reads ~edge and never discovers a branch schema:
find_downstream_schemas_sql reads the catalog, and a branch leaves no catalog row. That is
what keeps a pipeline's deletes off the drafts, and it behaves the same on MySQL and
PostgreSQL.

Deferred until the minimum scope ships

Draft and production tables as one diagram. Registering both physical schemas puts a
branch's tables in the diagram, and Diagram(schema) selects them by matching either physical
name. Drawing them as one logical cluster rather than two is presentation, and the minimum
scope does not need it.

Per-branch drop inside DataJoint. A caller tears a branch down by pattern with its own
credential, which needs nothing from the library. A schema.drop() that knows which tables
belong to a branch is a convenience on top.

Per-branch object cleanup. GarbageCollector is bound to a set of schemas at construction
(gc.py), so collecting a branch's objects is a matter of which schemas it is given. A
collector that walks only main treats a live branch's objects as orphans, so a run over
production must either exclude branch schemas or include every active one. Until the collector
is branch-aware, run it over production only.

Decisions, stated so the design spec need not re-derive them

A symbolic reference is a property of where a table lands, not a keyword in its definition.
-> Recording means the same thing on the branch and on main; only the SQL differs. A
definition-level marker would put a branch artifact in the customer's file, which is the one
thing this design exists to avoid.

delete cascades over symbolic edges. A branch's own references have no database
constraint to catch a missed cascade, so skipping them would silently orphan rows inside the
branch. Production is protected by never loading the branch schema, not by the cascade skipping
edges. The cost: ON DELETE RESTRICT no longer backstops a missed walk, so a production delete
of a row a draft depends on succeeds and leaves that draft row referencing nothing. A branch
holds a bounded sample rebuilt at every checkout, so this is accepted.

key_source reads symbolic edges without changing. The ~edge rows carry the child-side
and parent-side column names, from which load() derives attr_map, aliased, primary and
multi exactly as it does for catalog rows. If autopopulate.py needs an edit, the marker is
in the wrong place.

The ~edge read is gated on branch is not None. Off a branch, the union costs no query
and executes no new code. The consequence, stated rather than discovered later: this release
does not give DataJoint a general unenforced foreign key. ~edge is branch machinery, and an
off-branch session cannot see a branch's links — which is the required behavior.

Binding to an existing table raises when the shape differs, where shape means the primary
key in order and the set of attribute names. Under the binding rule, a branch declaring a
table whose name already exists binds to the existing one instead of creating a draft. The
grant catches the write, but the caller believes it created a table it adopted.

The comparison deliberately stops short of the whole definition. Recovering a definition from a
declared table means reconstructing it, and that reconstruction is lossy in known ways — a
functional index does not survive it, and bigint unsigned comes back as int64 — so a
comparison built on it would refuse tables that are in fact identical, which is a worse failure
than the one the rule prevents. Primary key and attribute names come from the live heading,
which is authoritative, on a path that has already loaded it. A changed attribute type on a
colliding name therefore goes undetected here; a stored definition hash is the durable fix and
the second PR's hidden-table machinery makes it cheap.

Lineage on a draft table records the physical schema. _populate_lineage copies the
parent's lineage for foreign-key attributes, so a draft table referencing Recording inherits
proj_ephys.recording.session_id and stays correct across promotion. Only an attribute
originating in a draft table records br_…, and that row dies with the schema that holds it.
Recording the logical name instead would claim an origin in a schema where the attribute does
not exist.

Creating the branch schema stays Schema.activate()'s job, on first declaration. On MySQL
the branch login holds CREATE under its pattern. On PostgreSQL CREATE SCHEMA cannot be
scoped by name, so the platform creates branch schemas and the library's attempt fails; the
error names the branch schema and says the platform provisions it. Creating a base schema needs
CREATE on the base name, which a credential scoped to the br_ prefix does not hold — the
same error shape applies, naming the base schema.

A reader that imported nothing still sees the branch. make_classes and dj.VirtualModule
list both physical schemas on a branch, so a draft's tables become classes alongside the
pipeline's; list_tables() matches both for the same reason; and schema['X'] resolves like
declaration rather than pointing at a logical table that may not exist. This is what recording
the reference where the catalog can be read back is for — it only pays off if a reader that
imported nothing can also find the tables those references connect. Where one name is reachable
as a single class from both schemas, the load refuses rather than picking one.

schema.drop() refuses on a branch. It drops the schema it is bound to, which is the
logical one — so on a branch the thing a caller can see would stop being the thing that gets
dropped, and the call would take the pipeline's schema out from under a live draft. Dropping a
branch's schemas is the caller's operation, not this one's.

What stays the same with no branch set

This is the line the feature does not cross. With database.branch unset:

• Name derivation is the identity, and activate() registers one schema.
• The declaration-time lookup does not run: no extra round trip per table on import.
• Dependencies.load() issues the same queries it issues today and reads no ~edge table.
• compile_foreign_key emits the same constraint and the same index.
• _populate_lineage, object paths, key_source, delete and drop are untouched.

What ships first, and what it does not yet unblock

First PR — the setting, the derived name, and the binding rule. These ship together or not
at all: a branch that derives a name without the binding rule shadows the whole pipeline into
the branch schema. Landing this alone is enough to exercise declaration in CI
against a throwaway server, and not enough to deploy, because a draft still declares a real
cross-schema constraint.

Second PR — symbolic references and ~edge. After this, a branch declares without touching
production's constraint graph and populate computes from the recorded edges. The minimum
scope is complete at this point.

The three deferred items follow in any order.

What a caller can test from outside the library

Each of these is observable without reading the library's source.

1. With DJ_BRANCH unset, the existing test suite passes unchanged, and a query count over a
   module import and over Dependencies.load() matches the count before this change.
2. With DJ_BRANCH=a1b2c3, dj.Schema('proj_ephys') declares a new table into
   br_a1b2c3_proj_ephys and leaves proj_ephys byte-for-byte unchanged in the catalog.
3. Importing a pipeline module whose tables all exist declares nothing: br_a1b2c3_proj_ephys
   holds no table until a genuinely new one is declared.
4. A new table under a branch declaring -> Recording produces no row in
   information_schema.key_column_usage for either schema, and one ~edge row per foreign-key
   column.
5. On that branch, NewTable.key_source equals the join of its parents and populate()
   computes the same keys it would with a real constraint.
6. A session with no branch set, connected as a production login, sees no branch table in
   Diagram, in parents()/children() of any production table, or in a delete cascade
   preview — while the branch schema exists and holds rows.
7. Recording.delete() from a production session succeeds with a draft referencing it, and
   deletes nothing in the branch schema.
8. On a branch, a delete inside the branch cascades from a draft parent to a draft child.
9. A branch declaring a class whose name matches an existing table raises when the primary key
   or the attribute-name set differs, and binds without declaring when both match. A table
   carrying a unique index or a bigint unsigned-backed type binds cleanly rather than raising.
10. Declaring a new part table under an existing master raises an error naming the master's
    schema. A new master declared with its own parts places master and parts in the draft.
11. dj.Schema('proj_new') on a branch, where proj_new exists nowhere, creates both
    proj_new (empty) and br_a1b2c3_proj_new, and every table declares into the draft.
12. On a branch, dj.VirtualModule and list_tables() show the pipeline's tables and the
    draft's together; off the same branch they show only the pipeline's.
13. schema.drop() raises on a branch and both physical schemas survive.
14. Both behaviors hold on MySQL 8.4 and PostgreSQL 16.

Where this meets open work

key_source stays a read of the parent join (#1523). A symbolic edge enters key_source
through parents(primary=True, …), which is exactly the surface #1523 keeps as internal
machinery while moving user narrowing to key_source_restriction. Nothing here touches the
override path, and nothing in #1523 changes where the edge comes from.

The cascade decision sits next to #1496. Both are about what the reverse-topological delete
walk does with an edge it can see. #1496 is a correctness bug on real constraints; this issue
adds edges the database does not enforce, so a fix for #1496 has to hold for symbolic edges
too.

The deferred collector work is #1478 and #1445. GarbageCollector is bound to its schemas
at construction, so a run over production while a branch is live reads that branch's objects as
orphans. Whichever of those lands first should take the branch schemas as part of its schema
set rather than inventing a second mechanism.

~edge follows the ~lineage precedent (#1454) and the hidden-attribute precedent (#1547).
A per-schema hidden table, written at declaration, read back from the catalog rather than held
in the declaring process. #1454 is the cautionary half: a marker table that can go stale needs
an idempotent rewrite on re-declaration, which a branch gets for free because every checkout
drops and re-declares.

The edge keying from #1492 is what makes the union safe. Edges are keyed by the child-side
attribute names, derived from the schema rather than from a database constraint name, so rows
read from ~edge merge into the same graph as catalog rows without a synthetic alias node.

Keep the supporting index; #1512 is why it is separate. PostgreSQL never indexes the
referencing columns and offers no server setting for it, so declare.py emits the index
itself. Dropping the constraint on a branch must not drop the index — they were already
decided independently.

The query-count criterion exists because of #1493. That issue was a per-key dependency
reload costing 59 information_schema queries. Any new per-import or per-load round trip here
would land in the same place, which is why criterion 1 counts queries rather than asserting
intent.